search

trafiklab / gtfs-examples

Examples on how to use GTFS(-RT) data for different purposes
6 stars 1 forks source link

No data updates after 24h #5

Open hallonstedt opened 2 years ago

hallonstedt commented 2 years ago

I have gotten TimeTableApi.py up and running after applying pull-request #4 but now another issue has appeared.

The received data doesn't update when the process has been left running for an extended period of time. I do get a response when accessing the end-point but it is outdated and does not take the current time into account so I receive the same departure times regardless when I access the server.

If I restart the process I immediately get updated information. My guess is that some data is only loaded on process start and then never refreshed. When I restart the process, the following message appears which may be a hint; 

2022-11-09 10:01:09,298 - root - INFO - GTFS archive is older than 1 day
2022-11-09 10:01:09,298 - root - INFO - Updating GTFS archive

Is this something you could take a look at. I am, of course, happy to helt testing but I am not a programmer so I don't think I will sort this on my own.

Truth be told, I just want to get real-time data from my home bus-stop and was hoping an existing API would do the trick but I do not get real-time data from ResRobot 2.1 using, for example, this query: https://api.resrobot.se/v2.1/departureBoard?id=740000944&duration=20&passlist=1&format=xml&accessId=[my-key] According to the documentation, Skånetrafiken provides real-time values like rTime/rDate but they do not appear in the response XML. Is there any other API and/or query I could use?

Bertware commented 2 years ago

According to the documentation, Skånetrafiken provides real-time values like rTime/rDate but they do not appear in the response XML. Is there any other API and/or query I could use?

This is an error in the documentation. Skånes realtime data used to be included, but Resrobot is currently quite limited in how realtime data can be integrated. We are looking at possible solutions to include more realtime data in Resrobot, but there is not time estimate on when this could be done.

The reason the application stops working after a while is because it only loads the gtfs file at startup, so after 24h it starts complaining. I'll see if I can add a background job to update the GTFS file once a day, that way it could keep running.

hallonstedt commented 2 years ago

Thank you, that would be ideal. For now, I'll write an ugly batch-script that kills and re-starts the process nightly.

hallonstedt commented 2 years ago

It seems my problem statement above was incorrect.

The problem with the application is that upon restart, it loads 2 hours worth of departures from the selected bus stop and after that it never updates that window. This means that if I reload the page during the 2 hour window, I will get updated information about the buses that are in motion but it will not add to the list. After two hours, all buses are at their destination and the page remains static until I restart the process and get a new 2 hour window.

This means that in order to provide value, I now have to restart the TimeTableApi.py application every hour and adjust the outcome to reflect current time + 1 hour. It works, but it is not pretty!

There may be an issue with the gtfs files not being updated as well but we can only know once above behavior is resolved.

hallonstedt commented 2 months ago

After a year, I have finally found myself with a bit of time on my hands and have managed to identify a few simple fixes that I think will add value. I added them to PR #11 for you to review.

hallonstedt commented 2 months ago

The only remaining issue after above PR is the filtering and selection of stop times when the time window spans across midnight. Currently, no departures are provided past midnight before the current time also is past midnight.

I have solved this issue by adding each of the functions involved in GtfsTimeTable.py to a Jupyter notebook and creating loads of verbose logging. Unfortunately, the end code is more or less rewritten so I am struggling to make meaningful code-commits.

I therefore decided to exclude this update in the PR above (#11) and will submit it for your review once the above PR is merged (I haven't managed to figure out how to submit a PR to the baseline of a pending PR)

Additionally, now that I got everything working perfectly, I got a all excited and added a bit of extra functionality such as a config-file with lots of configuration options, improved logging and updated documentation. This will follow if above PR's are merged. Brace for impact 😀