ODT : incorrect behavior ?

christopheblin commented 2 years ago

I try the following commands and I expect that gtfs2ntfs would create a comments.txt and a comment_links.txt for ODT as indicated in the docs

To summarize in a human friendly way what I did

launch latest version (0.51.0) of transit_model through docker
execute gtfs2ntfs
list files of the output directory to check that commetns.txt and comment_links.txt are missing
display the content of the -c option (you can see that both odt:true and odt_comment are set)
display an extract of the content of stop_times.txt (header + at least one trip where pickup_type and drop_off_type are set to 2)

Am I doing something wrong or is there a bug ?

Dont hesitate to ask more details if needed

Some details

https://github.com/hove-io/transit_model/blob/master/documentation/gtfs_to_ntfs_specs.md#prepending-data

Two parameters can be specified in the configuration of the converter in order to determine if on demand transport (ODT) data should be considered when reading the input GTFS (in particular, when reading the stop_times.txt file):

a boolean parameter odt, by default set to false, indicating if the GTFS should be considered as containing ODT information

a string odt_comment setting the message associated to an ODT comment.

https://github.com/hove-io/transit_model/blob/master/documentation/gtfs_to_ntfs_specs.md#reading-stop_timestxt

NTFS file	NTFS field	Constraint	Value/Note
comments.txt	comment_type	Optional	on_demand_transport
comments.txt	comment_name	Required	The message set for the parameter odt_comment.

Commands

docker run --rm -it -v $PWD:/data --entrypoint /bin/sh navitia/transit_model:v0.51.0
# gtfs2ntfs -i /data/gtfs -c /data/navitia.json -p vss -o /data/ntfs
2022-06-29T07:31:20.033793Z  INFO gtfs2ntfs: Launching gtfs2ntfs...
2022-06-29T07:31:20.033837Z  INFO transit_model::configuration: Reading dataset and contributor from "/data/navitia.json"
2022-06-29T07:31:20.038736Z  INFO transit_model::parser: Reading calendar.txt
2022-06-29T07:31:20.041757Z  INFO transit_model::parser: Reading calendar_dates.txt
2022-06-29T07:31:20.044944Z  INFO transit_model::parser: Reading agency.txt
2022-06-29T07:31:20.046066Z  INFO transit_model::gtfs::read: Reading stops.txt
2022-06-29T07:31:20.048915Z  INFO transit_model::parser: Reading stops.txt
2022-06-29T07:31:20.054125Z  INFO transit_model::parser: Skipping transfers.txt
2022-06-29T07:31:20.056582Z  INFO transit_model::parser: Reading shapes.txt
2022-06-29T07:31:20.126731Z  INFO transit_model::gtfs::read: Reading routes.txt
2022-06-29T07:31:20.128869Z  INFO transit_model::parser: Reading routes.txt
2022-06-29T07:31:20.135860Z  INFO transit_model::parser: Reading trips.txt
2022-06-29T07:31:20.151524Z  INFO transit_model::parser: Reading stop_times.txt
2022-06-29T07:31:20.276189Z  INFO transit_model::parser: Skipping frequencies.txt
2022-06-29T07:31:20.277361Z  INFO transit_model::parser: Skipping pathways.txt
2022-06-29T07:31:20.277966Z  INFO transit_model::parser: Skipping levels.txt
2022-06-29T07:31:20.303937Z  INFO transit_model::transfers: Generating transfers...
2022-06-29T07:31:20.303988Z  INFO transit_model::transfers: Adding missing transfers from stop points.
2022-06-29T07:31:20.367829Z  INFO transit_model::ntfs: Writing NTFS to "/data/ntfs"
2022-06-29T07:31:20.367850Z  INFO transit_model::ntfs::write: Writing feed_infos.txt
2022-06-29T07:31:20.373247Z  INFO transit_model::utils: Writing contributors.txt
2022-06-29T07:31:20.376253Z  INFO transit_model::utils: Writing datasets.txt
2022-06-29T07:31:20.379926Z  INFO transit_model::utils: Writing networks.txt
2022-06-29T07:31:20.385608Z  INFO transit_model::utils: Writing commercial_modes.txt
2022-06-29T07:31:20.390496Z  INFO transit_model::utils: Writing companies.txt
2022-06-29T07:31:20.396827Z  INFO transit_model::utils: Writing lines.txt
2022-06-29T07:31:20.399354Z  INFO transit_model::utils: Writing physical_modes.txt
2022-06-29T07:31:20.401930Z  INFO transit_model::utils: Writing routes.txt
2022-06-29T07:31:20.404441Z  INFO transit_model::utils: Writing geometries.txt
2022-06-29T07:31:20.460760Z  INFO transit_model::utils: Writing transfers.txt
2022-06-29T07:31:20.466583Z  INFO transit_model::ntfs::write: Writing trips.txt and stop_times.txt
2022-06-29T07:31:20.631095Z  INFO transit_model::calendars: Writing calendar_dates.txt
2022-06-29T07:31:20.635350Z  INFO transit_model::calendars: Writing calendar.txt
2022-06-29T07:31:20.639392Z  INFO transit_model::ntfs::write: Writing stops.txt
2022-06-29T07:31:20.677037Z  INFO transit_model::ntfs::write: Writing object_codes.txt
# ls /data/ntfs
calendar.txt          companies.txt     feed_infos.txt  networks.txt        routes.txt      transfers.txt
calendar_dates.txt    contributors.txt  geometries.txt  object_codes.txt    stop_times.txt  trips.txt
commercial_modes.txt  datasets.txt      lines.txt       physical_modes.txt  stops.txt
# cat /data/navitia.json
{
  "contributor": {
    "contributor_id": "test",
    "contributor_name": "Test",
    "contributor_license": "ODbL",
    "contributor_website": "https://www.test.io/"
  },
  "dataset": {
    "dataset_id": "test",
    "dataset_desc": "Donnees GTFS",
    "dataset_system": ""
  },
  "odt": true,
  "odt_comment": "Ceci est un test"
}# head -n 1 /data/gtfs/stop_times.txt          
trip_id,stop_id,arrival_time,departure_time,min_arrival_time,start_pickup_dropoff_window,max_departure_time,end_pickup_dropoff_window,stop_sequence,stop_headsign,pickup_type,drop_off_type,continuous_pickup,continuous_drop_off,start_service_area_radius,end_service_area_radius,departure_buffer,mean_duration_factor,mean_duration_offset,safe_duration_factor,safe_duration_offset
# grep /data/gtfs/stop_times.txt -e '^32,'
32,277,07:00:00,07:00:00,,,,,1,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,303,07:00:00,07:00:00,,,,,2,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,531,07:00:00,07:00:00,,,,,3,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,698,07:00:00,07:00:00,,,,,4,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,259,07:00:00,07:00:00,,,,,5,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,289,07:00:00,07:00:00,,,,,6,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,268,07:00:00,07:00:00,,,,,7,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,279,07:00:00,07:00:00,,,,,8,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,411,07:00:00,07:00:00,,,,,9,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,291,07:00:00,07:00:00,,,,,10,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,285,07:00:00,07:00:00,,,,,11,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,275,07:00:00,07:00:00,,,,,12,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,273,07:00:00,07:00:00,,,,,13,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,671,07:00:00,07:00:00,,,,,14,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,316,07:00:00,07:00:00,,,,,15,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,578,07:00:00,07:00:00,,,,,16,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,368,07:00:00,07:00:00,,,,,17,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,539,07:00:00,07:00:00,,,,,18,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,527,07:00:00,07:00:00,,,,,19,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,427,07:00:00,07:00:00,,,,,20,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,581,07:00:00,07:00:00,,,,,21,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,631,07:00:00,07:00:00,,,,,22,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,424,07:00:00,07:00:00,,,,,23,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,469,07:00:00,07:00:00,,,,,24,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
32,448,19:00:00,19:00:00,,,,,25,,2,2,1,1,-999.0,-999.0,0,-999.0,-999.0,-999.0,-999.0
#

woshilapin commented 2 years ago

I believe the documentation might be a bit misleading.

Two parameters can be specified in the configuration of the converter

I see you put odt and odt_comment in the navitia.json. I believe you can remove them. Try the following instead.

docker exec --rm -it -v $PWD:/data --entrypoint /bin/sh navitia/transit_model:v0.51.0 \
    gtfs2ntfs \
        --input /data/gtfs \
        --config /data/navitia.json \
        --prefix vss \
        --output /data/ntfs \
        --odt \
        --odt-comment "Ceci est un test"

Tell me if that works better. If this was the problem, we should probably fix a little bit the documentation.

christopheblin commented 2 years ago

@woshilapin yes it works (at least for the comments.txt+comment_links.txt on stop_time), thank you !

From there, what should I expect to behave differently in navitia ?

At the moment (without --odt), when I request a journey on this trip

if I set datetime to 7am, it finds the trip (but then, I travel at the "speed of light" because all stop_times have the same timestamps 😄 )
if I set datetime to 8am (or anything far enough), it does not find the trip

Please note that setting all the timestamps to the same value is from the doc at https://github.com/hove-io/ntfs-specification/blob/master/ntfs_fr.md#ligne-arr%C3%AAt-%C3%A0-arr%C3%AAt-sans-horaires

woshilapin commented 2 years ago

On the calculus part, I'm not familiar enough with ODT and how it's handle on Navitia. I'll try to see if I can find someone that can help on that.

christopheblin commented 2 years ago

@woshilapin ok thank you, I've made a test on my side with --odt and I still have the exact same behavior on the journey requests -> either I did something wrong or the --odt has no impact on the trip planner, which is a bit disappointing to me ...

prhod commented 2 years ago

@christopheblin If i'm reading your GTFS right, each stop_time is available to drop_off (drop_off_type=2 => Must phone agency to arrange drop off). If you want to avoid a speed of light travel, you may try to change 07:00:00 stop_times with a drop_off_type=1 (No drop off available.)

christopheblin commented 2 years ago

@prhod thanks for the suggestion : it means that the TAD trip will not appear anymore in any journey request, but it's fine for my use case

@woshilapin I think that the doc could be improved for --odt as you said (i.e just showing the CLI is fine I think). Thanks again for the support.

prhod commented 2 years ago

@christopheblin with my suggestion, with a A > B > C > D trip, you will be able to have a journey from A to D, B to D and C to D, but not from A to B. I thought it was the intended behaviour looking at your GTFS. Maybe it was not ;)

woshilapin commented 2 years ago

@christopheblin I created #855. Tell me if that looks better, and don't hesitate to tell if this can be even better. An outsider is usually better on that kind of things :wink:

christopheblin commented 2 years ago

@prhod ah I see what you mean, thanks for the clarification !

however, the current GTFS data are A -> B -> C -> .... -> X and all stop_times have 7am except X which is 7PM (but some travelers want to do B -> P for ex)

The correct way here I think would be to change the timestamps to match reality at 7 am and then to declare a frequency of 10 or 15 minutes with pickup_type=drop_off_type=2

Thank you again for the support :+1:

prhod commented 2 years ago

I'm closing the issue.Feel free to reopen if necessary

hove-io / transit_model

ODT : incorrect behavior ? #854