search

availabs / MTA_Subway_GTFS-Realtime_to_SIRI_Converter

1 stars 0 forks source link

npm run start fails #1

Open dancesWithCycles opened 2 years ago

dancesWithCycles commented 2 years ago

Hi folks, Thank you so much for providing this repository.

I cloned this repo, fired the instruction npm run start and got the following response.

npm run start

> MTA_Subway_SIRI_Server@0.0.0 start
> node siri_server.js

Could not construct the GTFS Feed Handler.
The server is listening on port 16181
{"info":"ConfigsService initializing the various configurations","timestamp":1648805586625.1804}
{"debug":"server.json configuration file successfully read."}
{"debug":"server.json configuration file successfully parsed."}
{"debug":"The logging.json was successfully parsed."}
{"__isValid":true,"defaultPortNumber":{"info":"defaultPortNumber looks okay."},"feedConfigurationFile":{"info":"mta_subway.json was found on the server."},"authenticator":{"info":"User API key authentication module found."},"__timestamp":1648805586626.7434}
{"__isValid":true,"unsupportedLoggingOptions":{"warn":"The following logging options are not supported:\n\tdaysToKeepLogsBeforeDeleting."},"__timestamp":1648805586626.141}
{"debug":"The active feed is set to mta_subway.","timestamp":1648805586626.45}
{"debug":"mta_subway configuration file read from disk.","timestamp":1648805586626.483}
{"debug":"Successfuly parsed the configuration file for mta_subway.","timestamp":1648805586626.512}
{"__isValid":true,"feedURL":{"info":"GTFS feedURL is a valid URL."},"tripKeyMutator":{"info":"GTFS tripKeyMutator appears valid."},"__timestamp":1648805586627.048}
{"__isValid":true,"feedURL":{"info":"The feedURL supplied for GTFS-Realtime configuration looks valid."},"readInterval":{"info":"GTFS-Realtime configuration field 'readInterval' looks valid."},"retryInterval":{"info":"GTFS-Realtime configuration field 'retryInterval' looks valid."},"maxNumRetries":{"info":"GTFS-Realtime configuration field 'maxNumRetries' looks valid."},"protofile":{"info":"The specified .profo file exists."},"__timestamp":1648805586628.6384}
{"__isValid":true,"fieldMutators_OriginRef":{"info":"The OriginRef fieldMutator is valid."},"fieldMutators_DestinationRef":{"info":"The DestinationRef fieldMutator is valid."},"fieldMutators_StopPointRef":{"info":"The StopPointRef fieldMutator is valid."},"__timestamp":1648805586628.945}
(node:10775) Warning: Accessing non-existent property 'resetConverterStatus' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
{"debug":"ConverterService start called.","timestamp":1648805586629.1516}
{"debug":"ConverterService starting the GTFS_FeedHandlerService.","timestamp":1648805586629.2637}
{"error":"ConverterStream startup process encountered the following error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/indexedScheduleData.json' GTFS-Realtime to Siri conversion will not be available until the configuration problems are fixed.","debug":"Error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/indexedScheduleData.json'\n    at Object.openSync (node:fs:585:3)\n    at Object.readFileSync (node:fs:453:35)\n    at Object.readFileSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/jsonfile/index.js:32:26)\n    at FeedHandler.getIndexedScheduleDataSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:237:44)\n    at FeedHandler.getLatestGTFSIndicesSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:228:58)\n    at new FeedHandler (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:70:59)\n    at Object.start (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/GTFS_FeedHandlerService.js:27:23)\n    at start (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/ConverterService.js:72:33)\n    at Object.<anonymous> (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/ConverterService.js:304:1)\n    at Module._compile (node:internal/modules/cjs/loader:1099:14)","timestamp":1648805586631.829}
(node:10775) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency

I am especially tripping over this snippet.

{"error":"ConverterStream startup process encountered the following error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/indexedScheduleData.json' GTFS-Realtime to Siri conversion will not be available until the configuration problems are fixed.","debug":"Error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/indexedScheduleData.json'\n at Object.openSync (node:fs:585:3)\n at Object.readFileSync (node:fs:453:35)\n at Object.readFileSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/jsonfile/index.js:32:26)\n at FeedHandler.getIndexedScheduleDataSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:237:44)\n at FeedHandler.getLatestGTFSIndicesSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:228:58)\n at new FeedHandler (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:70:59)\n at Object.start (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/GTFS_FeedHandlerService.js:27:23)\n at start (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/ConverterService.js:72:33)\n at Object. (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/ConverterService.js:304:1)\n at Module._compile (node:internal/modules/cjs/loader:1099:14)","timestamp":1648805586631.829}

I appreciate any hint in the right direction that helps me solve or bypass this issue.

Cheers!

PaulTomchik commented 2 years ago

Sorry for the poor documentation.

You can generate those files in the MTA_Subway_SIRI_Server repo with the following command:

./bin/updateGTFSData.js url
dancesWithCycles commented 2 years ago 
./bin/updateGTFSData.js url

Hi @PaulTomchik , Thank you so much for the prompt reply. Is the url parameter you mentioned above the address ot the GTFS-RT feed?

Cheers!

dancesWithCycles commented 2 years ago 
./bin/updateGTFSData.js

Hi @PaulTomchik , I am doing something wrong but I am so blind that I do not see the point. I called the following CLI instruction in the root folder of the MTA_Subway_SIRI_Server repo.

./bin/updateGTFSData.js http://data.ndovloket.nl/flixbus/flixbus-eu.zip

This instruction results in the following complain.

USAGE: An unrecognized GTFS feed source was given.
    This script expects a single command-line argument that specifies where to retrieve
    the GTFS feed data. This argument should be either "file" or "url".
    If no argument is given, the default of "url" is used.
    If "file" is specified, the GTFS feed data must be in a zip file archive
    at the path given in the GTFS config's feedDataZipFilePath property.
    If "url" is specified, the GTFS feed data will be retrieved from the url given
    in GTFS config's feedURL property, where, again, a zip archive is the required format.

I thought I am using the script in the expected way. You can use the provided URL in Firefox browser to download the GTFS feed as ZIP archive.

I am appreciating any thoughts about the point I am missing.

Cheers!

dancesWithCycles commented 2 years ago 
./bin/updateGTFSData.js url

Hi @PaulTomchik , When I use a config file instead of the CLI parameter argument to supply the GTFS-RT feed address the ./bin/updateGTFSData.js url script is executing and comes back with the following reply.

./bin/updateGTFSData.js
INFO: Because the feed source ("file"|"url") was not specified, the default of "url" will be used.
    The GTFS data will be downloaded from http://data.ndovloket.nl/flixbus/flixbus-eu.zip
[Error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/tmp/shapes.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/tmp/shapes.txt'
}
getStopIDToCoords: 1.514s
getTripID2ShapeID: 2.447s
getTripID2StopIDs: 9.118s
GTFS update complete.

Is this really an error? I mean, is a GTFS feed required to contain a shapes.txt file? Or is this reply rather a warning or an info and the service is running no matter if a shapes.txt file is part of the GTFS feed?

dancesWithCycles commented 2 years ago

Hi @PaulTomchik , When I call npm start, the service comes back with the following reply.

npm start

> MTA_Subway_SIRI_Server@0.0.0 start
> node siri_server.js

Could not construct the GTFS Feed Handler.
The server is listening on port 16181
{"info":"ConfigsService initializing the various configurations","timestamp":1649944260692.807}
{"debug":"server.json configuration file successfully read."}
{"debug":"server.json configuration file successfully parsed."}
{"debug":"The logging.json was successfully parsed."}
{"__isValid":true,"defaultPortNumber":{"info":"defaultPortNumber looks okay."},"feedConfigurationFile":{"info":"mta_subway.json was found on the server."},"authenticator":{"info":"User API key authentication module found."},"__timestamp":1649944260693.695}
{"__isValid":true,"unsupportedLoggingOptions":{"warn":"The following logging options are not supported:\n\tdaysToKeepLogsBeforeDeleting."},"__timestamp":1649944260694.3035}
{"debug":"The active feed is set to mta_subway.","timestamp":1649944260694.7878}
{"debug":"mta_subway configuration file read from disk.","timestamp":1649944260694.841}
{"debug":"Successfuly parsed the configuration file for mta_subway.","timestamp":1649944260694.8906}
{"__isValid":true,"feedURL":{"info":"GTFS feedURL is a valid URL."},"tripKeyMutator":{"info":"GTFS tripKeyMutator appears valid."},"__timestamp":1649944260695.6218}
{"__isValid":true,"feedURL":{"info":"The feedURL supplied for GTFS-Realtime configuration looks valid."},"readInterval":{"info":"GTFS-Realtime configuration field 'readInterval' looks valid."},"retryInterval":{"info":"GTFS-Realtime configuration field 'retryInterval' looks valid."},"maxNumRetries":{"info":"GTFS-Realtime configuration field 'maxNumRetries' looks valid."},"protofile":{"info":"The specified .profo file exists."},"__timestamp":1649944260696.4578}
{"__isValid":true,"fieldMutators_OriginRef":{"info":"The OriginRef fieldMutator is valid."},"fieldMutators_DestinationRef":{"info":"The DestinationRef fieldMutator is valid."},"fieldMutators_StopPointRef":{"info":"The StopPointRef fieldMutator is valid."},"__timestamp":1649944260696.943}
(node:18636) Warning: Accessing non-existent property 'resetConverterStatus' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
{"debug":"ConverterService start called.","timestamp":1649944260698.7988}
{"debug":"ConverterService starting the GTFS_FeedHandlerService.","timestamp":1649944260698.9792}
{"error":"ConverterStream startup process encountered the following error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/indexedSpatialData.json' GTFS-Realtime to Siri conversion will not be available until the configuration problems are fixed.","debug":"Error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/indexedSpatialData.json'\n    at Object.openSync (node:fs:585:3)\n    at Object.readFileSync (node:fs:453:35)\n    at Object.readFileSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/jsonfile/index.js:32:26)\n    at FeedHandler.getIndexedSpatialDataSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:271:44)\n    at FeedHandler.getLatestGTFSIndicesSync (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:229:57)\n    at new FeedHandler (/home/begerad/git/github/MTA_Subway_SIRI_Server/node_modules/GTFS_Toolkit/lib/FeedHandler.js:70:59)\n    at Object.start (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/GTFS_FeedHandlerService.js:27:23)\n    at start (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/ConverterService.js:72:33)\n    at Object.<anonymous> (/home/begerad/git/github/MTA_Subway_SIRI_Server/src/services/ConverterService.js:304:1)\n    at Module._compile (node:internal/modules/cjs/loader:1099:14)","timestamp":1649944260735.0164}
(node:18636) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency

Do you know what is meant with Could not construct the GTFS Feed Handler.?

Appreciate!

PaulTomchik commented 2 years ago

Hi @dancesWithCycles

I apologize that it is not in the documentation but the GTFS Feed ZIP archive must include a shapes.txt file.

It appears that flixbus-eu.zip archive provided at the URL mentioned above does not.

$ ls -l 
total 5808
-rw-r--r-- 1 paul paul     719 Jan  7  2021 agency.txt
-rw-r--r-- 1 paul paul    7849 Jan  7  2021 calendar_dates.txt
-rw-r--r-- 1 paul paul   11138 Jan  7  2021 calendar.txt
-rw-r--r-- 1 paul paul     146 Jan  7  2021 feed_info.txt
-rw-rw-r-- 1 paul paul  481081 Apr 15 11:24 flixbus-eu.zip
-rw-r--r-- 1 paul paul   37401 Jan  7  2021 routes.txt
-rw-r--r-- 1 paul paul  212126 Jan  7  2021 stops.txt
-rw-r--r-- 1 paul paul 4674809 Jan  7  2021 stop_times.txt
-rw-r--r-- 1 paul paul   15152 Jan  7  2021 transfers.txt
-rw-r--r-- 1 paul paul   13765 Jan  7  2021 translations.txt
-rw-r--r-- 1 paul paul  466989 Jan  7  2021 trips.txt
dancesWithCycles commented 2 years ago

I apologize that it is not in the documentation but the GTFS Feed ZIP archive must include a shapes.txt file.

Hi @PaulTomchik , No worries! Do you know the reason, why a shape.txt is required? I hope the reason is not the CEN SIRI Vehicle Monitoring Service. That would mean I can not convert GTFS-RT in SIRI VM for Flixbus with the current GTFS feed.

Anyhow, can you explain how/why the two repos MTA_Subway_GTFS-Realtime_to_SIRI_Converter and MTA_Subway_SIRI_Server depend on each other?

I appreciate your help and cooperation very much. Do not hesitate to reach out to me when I can do something for you in return.

Cheers!

dancesWithCycles commented 2 years ago

Hi @PaulTomchik , BTW, Do you know, if MTA is still running any code from about six years ago to provide SIRI services? Cheers!

PaulTomchik commented 2 years ago

Hi @dancesWithCycles ,

Sorry, but the short answer is that the shapes.txt file is currently a hard dependency. In its current form, the project would not be able to perform conversions for GTFS feeds that lack that file. It is in fact largely due to the Vehicle monitoring service, however there are likely many other places the code would break if route shapes are not available because the central data structures assume their existence. I looked for simple ways to remove the dependency but had no success. When I get a few moments I will experiment more with removing that dependency, but it may require a significant amount of work.

The _MTA_Subway_SIRIServer is a wrapper around the _MTA_Subway_GTFS-Realtime_to_SIRIConverter that provides the HTTP API (see code). In general, the design of these projects is such that specialized repos/modules could be created and composed to address specific requirements of a given transit agency--the GTFS specification is rather flexible/loose. For example, the MTA_Subway_SIRI_Server depends on repos specific to the MTA feeds.

PaulTomchik commented 2 years ago

@dancesWithCycles

Have you seen the MTA Bus Time SIRI Intro?

PaulTomchik commented 2 years ago

@dancesWithCycles

I have attached a ZIP archive with what ./bin/updateGTFSData.js would produce for the flixbus-eu.zip if it did not throw an error.

You can try removing the existing ./data/ directory from your MTA_Subway_SIRI_Server clone and extracting this ZIP in its place.

data.zip

dancesWithCycles commented 2 years ago

@dancesWithCycles

Have you seen the MTA Bus Time SIRI Intro?

Hi @PaulTomchik , Thanks for the reminder. Indeed, I know this web page.

If you are curious to read about all of the features of SIRI, the SIRI Handbook makes for a bit of light reading.

It is a pity, that the link for the SIRI Handbook is not working anymore. Do you know how or where to get it?

Cheers!

dancesWithCycles commented 2 years ago

@dancesWithCycles

I have attached a ZIP archive with what ./bin/updateGTFSData.js would produce for the flixbus-eu.zip if it did not throw an error.

You can try removing the existing ./data/ directory from your MTA_Subway_SIRI_Server clone and extracting this ZIP in its place.

data.zip

Hi @PaulTomchik , Today I was exited to follow your suggestion. I did the following.

$ ./bin/updateGTFSData.js url
[Error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/tmp/shapes.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/tmp/shapes.txt'
}
getStopIDToCoords: 1.382s
getTripID2ShapeID: 2.240s
getTripID2StopIDs: 8.381s
GTFS update complete.
$ ./bin/updateGTFSData.js file
/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/tmp/gtfs.zip
[Error: ENOENT: no such file or directory, open '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/tmp/shapes.txt'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/home/begerad/git/github/MTA_Subway_SIRI_Server/data/GTFS/tmp/shapes.txt'
}
getStopIDToCoords: 1.399s
getTripID2ShapeID: 2.256s
getTripID2StopIDs: 8.775s
GTFS update complete.

Is this feedback/output the expected answer for a successful GTFS feed data update. Is it what I should expect?

BTW, in the first place my plan was to set up MTA_Subway_GTFS-Realtime_to_SIRI_Converter and MTA_Subway_SIRI_Server for the Flixbus GTFS and GTFS Realtime feed. Anyhow, if this is a challenge for the long run, my first step could be to switch to a feed that includes shape files. What do you think?