Add back frequency.txt to OTP2

t2gran commented 3 years ago

OTP2 do not support the GTFS frequency.txt.

See issue #3243 for discussion and test-cases.

There is a few things that we need to account for:

RT-updates on these kind of trips
How is frequency trips handled on the API
Import into the OTP internal domain model
exact_times=1 Support Schedule-based trips first.

The exact_times=0 is a bit problematic, and we would need to agree on how the system should work to implement it. As a first take on this I suggest we implement exact_times=0 first, which can be done with just expanding the stop times for each trip.

Version of OTP used (exact commit hash or JAR name)

dev-2.x

Data sets in use (links to GTFS and OSM PBF files)

See #3243

Router config and graph build config JSON

Default

linleybird commented 3 years ago

Frequency type trips are vital in transport systems which operate in developing/emerging countries. Including this file will make OPT 2 able to better represent the majority of the world - making the platform far more inclusive.

This would be a big win for my workflow!

t2gran commented 3 years ago

@linleybird I agree. I think the exact_times=0 is simple to implement. Help with the implementation is appreciated.

Mchristos commented 3 years ago

Hi @t2gran - to clarify, I assume you mean exact_times=1 would be simple to implement (and is not currently supported). Is that correct? As I understand it, exact_times=0 would be problematic as there is no simple way to expand out the stop_times.

Mchristos commented 3 years ago

To make OTP2 work with my GTFS dataset, since it doesn't support frequencies.txt, I have written an algorithm to expand out all the frequencies as defined in frequencies.txt into the stop_times.txt file (with a new set of trips created for each frequency trip, stored in trips.txt).

The agency I am working with has around 24,000 frequency trips, which once expanded gives me a stop_times file around 4GB in size. I needed a rather hefty machine to perform the OTP graph build, but I managed to build the graph and serve OTP.

My problem is, the response times for journey requests are very slow. The results I get running load tests are below:

min=222.18ms med=975.64ms p(90)=3.35s p(95)=3.58s max=4.39s

I'm hoping for better response times. Using OTP v1 on the same dataset (no manual expansion there since it support frequencies), I get around med=350ms - so I was expecting OTP v2 to do much better.

Are there any suggestions - is it possible that I am doing something wrong in the frequencies expansion? If OTP2 were to support frequencies.txt, would it not simply expand out the frequencies in any case? Presumably it would be a bit smarter, given the faster response times of OTP1.

Help would be greatly appreciated, and if you had the time to have a conversation about it, that could be really helpful and interesting @t2gran. Perhaps some pointers about how you would go about implementing it in the codebase.

t2gran commented 3 years ago

@Macroz I am happy to assist you on this. I can meet with you tomorrow if you want between 10:00-15:00 CEST on for example google meet. Send me an invite to t2gran on gmail.com.

I looked at it a little now, and it looks almost to simple - it never is, but here is an outline:

1) In GenerateTripPatternsOperation@line 137 we can add an extra loop creating new TripTimes using the copy-constructor. 2) Set the TripTime.timeShift.

Then test. Of cause we must also update documentation and remove obsolete code.

Notes!

What do we do with the exact_times=0, I think maybe we can get ok results adding a board/alight slack using a a function of the headway.

barbeau commented 3 years ago

What do we do with the exact_times=0, I think maybe we can get ok results adding a board/alight slack using a a function of the headway.

IIRC, OTP1 assumes a pessimistic approach to wait time at a transit stop for true frequency-based (exact_times=0) trips. In other words, if the frequency of the vehicle is 15 minutes, it assumes the traveler will need to wait the full 15 minutes before boarding the vehicle. Will OTP2 take the same approach?

(Note that if real-time info is available, you could calculate real-time estimated wait times instead of the pessimistic worst-case scenario)

t2gran commented 3 years ago

Let use a configurable factor for "additional" alight or board slack with default value 1.0. This would give us the pessimistic worst-case scenario, then if someone would like to use the average probably "wait-time" the factor can be set to 0.5.

Where should the "wait-time/slack" be added, before or after the trip? Maybe it does not matter, the times in the itinerary should be shown with the uncertainty.

t2gran commented 3 years ago

The PR above add back basic support for frequency.txt. However it ignore the value of the exact_times and treat all trips as exact scheduled trips. The API Leg.isNonExactFrequency is set. I had a look at the code to see how easy it would be to support a very simple "wait-time" approach to implement true frequency based trips.

I think we can do it 2 ways:

The simplest is to shift the arrival times by a factor of the headway. Example: headway 10 min, stopTime 12:00 => departure-time is 12:00, but the arrival-time at the same stop would be 11:50.
Another way, witch is more complicated to implement is to add board- and/or alight-slack. This is more work, because it touches the performance optimized code and we have to make sure we can do it without touching the Trip and Frequency instance in memory.

barbeau commented 3 years ago

Where should the "wait-time/slack" be added, before or after the trip? Maybe it does not matter, the times in the itinerary should be shown with the uncertainty.

Hmm, I'm not sure it matters, as long as it's taken into consideration for transfers? For example, if you were transferring from a frequency-based (type 0) trip to a schedule-based trip (type 1 or normal), your transfer options may differ depending on the time you would alight at the last stop on the frequency-based trip plus the slack. To me though it seems more logical and easier to reason about if it's added before the trip, because conceptually you're saying "add padding to this trip to because we don't know exactly when it's going to arrive to pick you up."

Agreed that clients should always be labeling these somehow as "arriving every 10 minutes" so the end-user doesn't assume that it's a scheduled time when they are first boarding.

The simplest is to shift the arrival times by a factor of the headway. Example: headway 10 min, stopTime 12:00 => departure-time is 12:00, but the arrival-time at the same stop would be 11:50.

I think that would work 🤔

t2gran commented 3 years ago

I think all of these variation would work, I just tested adjusting the arrival-time. I do not like this because it looks very strange when the arrival is BEFORE the departure when debugging, and the arrival time must be reset to the "correct" value after the routing request.

So the clean way to do it is to add slack. If the slack is before, it is easier to reason about and explain, but it might also tell the traveler to arrive at the stop too late - if he/she is lacy like me he/she will just look at the time for the departure. But, from a server point of view I don´t think it matters. Either way, the UI must communicate this and can shift the trip - if wanted.

mvanlaar commented 3 years ago

Is there any progress on this issue? Like @Mchristos is also have a lot of frequency based trips in my data sets. Is it in the plan to merge it in 2.1 when it will be released?

civiliaInc commented 2 years ago

Hello, what is the status of the frequency with OTP2?

leonardehrenfried commented 2 years ago

There is currently a PR in review: #3916 It's likely that it will be merged soon.

opentripplanner / OpenTripPlanner