search

noi-techpark / odh-mentor-otp

5 stars 8 forks source link

Stops.txt: describe stops relationships #43

Open zabuTNT opened 3 years ago

zabuTNT commented 3 years ago

I suggest in situations like Meran station to use location_type / parent_station and platform_code, in order to describe better the node, and relationships between boarding areas, platforms and station. (it's similar to Netex stop description)

Otherwise you have a lot of stops with the same name and it's not reccomended.

2021-03-12-084346_596x479_scrot

This is not a journey planner issue, it's only a suggestion to produce high quality GTFS for other users.

rcavaliere commented 3 years ago

Fully agree. They support all this but probably it's currently not exported in the GTFS

rcavaliere commented 3 years ago

@zabuTNT feedback by Mentz: this is possible, there is something to change in the configuration of the export in DIVA. I have asked STA to test this and to provide us a different GTFS export which we can test

rcavaliere commented 3 years ago

STA will do this at the beginning of August (current reference person is on holiday)

rcavaliere commented 3 years ago

@zabuTNT @stefanocudini STA has made the upgrade of the GTFS export with the details of the stops relationships. It's available on ftp://ftp.sta.bz.it/gtfs/ as a new file (google_transit_parental_stops.zip). What is the name of the GTFS export we are expecting so to automatically download the GTFS and rebuild the OTP graph? I would then rename this GTFS file so that then everything is tested with the automatic chain you have set up

stefanocudini commented 3 years ago

the gtfs filename from which to generate the graph is set with this environment variable: GTFS_FILE Currently I believe that in your infrastructure it has this value: "latestGTFS.zip"

stefanocudini commented 3 years ago

ops... sorry for misunderstanding! the source gtfs is this variable: GTFS_URL that currently is "ftp://ftp.sta.bz.it/gtfs/google_transit_shp.zip" so to download the new gtfs it should be enough to set the value in your environment GTFS_URL=ftp://ftp.sta.bz.it/gtfs/google_transit_parental_stops.zip

rcavaliere commented 3 years ago

@stefanocudini @zabuTNT @bertolla STA has renamed the file so that we don't have to change nothing. I therefore assume that now everything should be be computed automatically this night, correct? Let's tomorrow what we will see on the application....

stefanocudini commented 3 years ago

Yes good

eventually you can monitor the logs of the otp container.. which produces messages in case it verify changes of the gtfs file and the rebuilding of the graph

rcavaliere commented 3 years ago

@stefanocudini @zabuTNT it seems that it has worked as expected, can you also check? If ok, we can close this issue and #7

stefanocudini commented 3 years ago

about this we would need to be able to see the logs of the otp container and data directory contents /opt/odh-mentor-otp/ maybe you could send via email.

in the meantime we are developing the part that processes the data I add the parent relations of the stops, this requires some changes that I will send you in the next PR

bertolla commented 3 years ago

Here the log and the contents of dir: otp.log

.
├── all.osm
├── build-config.json
├── latestGTFS.zip
├── openmove
│   └── Graph.obj
├── osm.url
├── router-config.json
├── srtm_39_03.tif
└── srtm_39_03.zip
stefanocudini commented 3 years ago

@bertolla we need an ls -lR to understand when latestGTFS.zip file and Graph.obj has been modified also ls -lR /etc/periodic please

stefanocudini commented 3 years ago

new PR https://github.com/noi-techpark/odh-mentor-otp/pull/84 create a new log file, more verbose, located at: /data/gtfs_download.log to monitoring check and auto update of gtfs data from ftp server.

stefanocudini commented 3 years ago

@rcavaliere to test the data updated after the build with the new gtfs structure having parent station you have to go here with the browser: https://otp.opendatahub.testingmachine.eu/otp/routers/default/index/stops?detail=true and verify that some of the stops contain the attribute: parentStation

here a sample from my local OTP instance:

// http://localhost:8080/otp/routers/default/index/stops?detail=true
[
  {
    "id": "1:it:22021:1068:0:8275",
    "name": "Verdignes",
    "lat": 46.658341595077,
    "lon": 11.5751370114614,
    "locationType": 0,
    "parentStation": "it:22021:1068P",
    "wheelchairBoarding": 0,
    "vehicleType": -999,
    "vehicleTypeSet": false
  },
  {
    "id": "1:it:22021:501:0:9279",
    "name": "Gruber",
    "lat": 46.5433129358046,
    "lon": 11.3526081334373,
    "locationType": 0,
    "parentStation": "it:22021:501P",
    "wheelchairBoarding": 0,
    "vehicleType": -999,
    "vehicleTypeSet": false
  },
rcavaliere commented 3 years ago

@stefanocudini I have checked this but it seems to me that no stop has parentStation attribute. Can you confirm this?

stefanocudini commented 3 years ago

Sorry but to understand if the problem is on the code side I should have access to the logs of the deployed containers.

rcavaliere commented 3 years ago

@RudiThoeni now that the build was successfully completed, can you share the requested log file (/data/gtfs_download.log) to this thread?

RudiThoeni commented 3 years ago

gtfs_download.log

stefanocudini commented 3 years ago

@RudiThoeni I think this environment variable: GTFS_URL_UPDATEHOOK in the container has not been set, otherwise it would have been printed in the log file, this must be set inside the otp container

[10/30/21_04:00:00] Download new gtfs and checksum...
-rw-r--r--    1 root     root      88634121 Oct 30 04:00 /tmp/gtfs_LzK3ug7pKW14jgRlopSmD8q5b3LIbEpX.zip
[10/30/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[10/30/21_04:00:00] run rebuild hook
-rw-r--r--    1 root     root      88634121 Oct 30 04:00 /tmp/gtfs_WhIxfPXg1Mw574Ng4Si8SthmIyndZ67W.zip
[10/30/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[10/30/21_04:00:00] gtfs not changed!

look here the script that generate the log file: https://github.com/noi-techpark/odh-mentor-otp/blob/development/gtfs-import/gtfs-download.sh#L41

the last time the gtfs data was automatically downloaded it happened on October 30th, but hook url being empty it was not executed and the graph has not been regenerated by the build container.

now that I look closer it might be safer not to print the hook url, I can log the http code response, but this does not change the problem.

stefanocudini commented 3 years ago

here: https://github.com/noi-techpark/odh-mentor-otp/blob/master/infrastructure/Jenkinsfile-Prod-Execute.groovy#L56 I think something is missing credentials() or other things

RudiThoeni commented 3 years ago

@stefanocudini Thank you, i will have a look at it tommorrow, maybe you can also have a look on the logs (https://github.com/noi-techpark/odh-mentor-otp/issues/90#issuecomment-954713045) we posted here https://github.com/noi-techpark/odh-mentor-otp/issues/90#issuecomment-954717273 and referenced in this comment?

stefanocudini commented 3 years ago

Now I have improved the generated log, so it is more secure and you can understand if the web hook is working> https://github.com/noi-techpark/odh-mentor-otp/pull/99/commits/7ffdf861861411ff2d70e53e530eb8965cfc2085

all in the last PR

RudiThoeni commented 3 years ago

Hi @stefanocudini as you pointed out correctly, the variable GTFS_URL_UPDATEHOOK was not set in the container now i added it and merged your PR and deployed on testing + production, will post the log next week then we see if the update is done..... however starting the otp container still gives some errors, i will post it in issue https://github.com/noi-techpark/odh-mentor-otp/issues/90

stefanocudini commented 3 years ago

fine! we're working about #90 today, i will send you a PR in the next few days for that. I think the problem is in the proxy in front of the gbfs service, i will let you know soon

RudiThoeni commented 3 years ago

Hi the log now gives

[11/05/21_04:00:00] Download new gtfs and checksum...
-rw-r--r--    1 root     root      88634121 Nov  5 04:00 /tmp/gtfs_7Nqcnlf1ypsogLG1Lix05WOHigggI4qD.zip
-rw-r--r--    1 root     root      88634121 Nov  5 04:00 /tmp/gtfs_x6oR3UavQcyWB7Jzag8oLNMgQzwgFGsH.zip
[11/05/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[11/05/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[11/05/21_04:00:00] gtfs not changed!
[11/05/21_04:00:00] run rebuild hook...
hook http response: 000
[11/06/21_04:00:00] Download new gtfs and checksum...

[11/06/21_04:00:00] Download new gtfs and checksum...
-rw-r--r--    1 root     root      88634121 Nov  6 04:00 /tmp/gtfs_TVTX0ckN4Yzl0yXFcUPog9Rj09sfH8xx.zip
[11/06/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[11/06/21_04:00:00] gtfs not changed!
-rw-r--r--    1 root     root      88634121 Nov  6 04:00 /tmp/gtfs_LQRR4YuWSKtSEltSe2ByGYIABQdcvsBY.zip
[11/06/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[11/06/21_04:00:00] gtfs not changed!
[11/07/21_04:00:00] Download new gtfs and checksum...

[11/07/21_04:00:00] Download new gtfs and checksum...
-rw-r--r--    1 root     root      88634121 Nov  7 04:00 /tmp/gtfs_TpMjt7Vjd5eb3kKiRTBNRmhzO2S2Sc8z.zip
-rw-r--r--    1 root     root      88634121 Nov  7 04:00 /tmp/gtfs_CkAVRAzZrorrmJNViXQgZ59Apst943ZA.zip
[11/07/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[11/07/21_04:00:00] gtfs not changed!
[11/07/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[11/07/21_04:00:00] gtfs not changed!

[11/08/21_04:00:00] Download new gtfs and checksum...
[11/08/21_04:00:00] Download new gtfs and checksum...
-rw-r--r--    1 root     root      88634121 Nov  8 04:00 /tmp/gtfs_JVlfLhmZx3vTUikWPvQlrLb93V5CDFAU.zip
-rw-r--r--    1 root     root      88634121 Nov  8 04:00 /tmp/gtfs_ohDO1aKn4afTnoO2prtxq3Ydzhx15V30.zip
[11/08/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[11/08/21_04:00:00] gtfs not changed!
[11/08/21_04:00:00] new checksum e7d4d89e1dc6342c8518fc4434efd9645c1fb191fca422796cab1f974426f068
[11/08/21_04:00:00] gtfs not changed!

is this the right behavior?

[11/05/21_04:00:00] run rebuild hook...
hook http response: 000
stefanocudini commented 3 years ago

no I think there are network problems between the otp container and the webhook service, 000 is a common code to use when no HTTP code was received a network error.

I advise you to verify that from inside the container you can exit on the hostname specified in the webhook. what our code does is a simple curl: https://github.com/noi-techpark/odh-mentor-otp/blob/development/gtfs-import/gtfs-download.sh#L46 we don't know url of pass via GTFS_URL_UPDATEHOOK, maybe it contains error in url format?

RudiThoeni commented 3 years ago

ok thx you are right i have seen that on production there is a timeout and on development there is a 403 by calling the updatehook url....... i have to see how this all works together