search

planarnetwork / transxchange2gtfs

tool to convert transxchange data into a GTFS feed
30 stars 17 forks source link

No operating profile fails conversion #45

Closed tafflin closed 3 years ago

tafflin commented 3 years ago

@linusnorton
I hope this is the last problem with DFT .xml data. I am still not sure how the previous version we were using did not fail to convert the bulk DFT archive. In the file in the attachment there is no operating profile in one of the VehicleJourneys. Is it possible to skip the trip conversion in such cases? Also is it possible that trancxchange2gtfs converter would skip files which have errors and still generate a resulting GTFS without such trips/routes? Currently converting a bulk archive with ~21000 files takes around 15-20 min I cannot get a GTFS file to validate it.

Processing [swe_30-22-_-y10-1.zip](https://github.com/planarnetwork/transxchange2gtfs/files/7254874/swe_30-22-_-y10-1.zip)
(node:16772) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'RegularDayType' of undefined
    at TransXChangeJourneyStream.getCalendar (/usr/lib/node_modules/transxchange2gtfs/dist/src/transxchange/TransXChangeJourneyStream.js:61:39)
    at TransXChangeJourneyStream._transform (/usr/lib/node_modules/transxchange2gtfs/dist/src/transxchange/TransXChangeJourneyStream.js:44:39)
    at TransXChangeJourneyStream.Transform._read (internal/streams/transform.js:205:10)
    at TransXChangeJourneyStream.Transform._write (internal/streams/transform.js:193:12)
    at writeOrBuffer (internal/streams/writable.js:358:12)
    at TransXChangeJourneyStream.Writable.write (internal/streams/writable.js:303:10)
    at TransXChangeStream.ondata (internal/streams/readable.js:726:22)
    at TransXChangeStream.emit (events.js:412:35)
    at addChunk (internal/streams/readable.js:290:12)
    at readableAddChunk (internal/streams/readable.js:265:9)
tafflin commented 3 years ago

I would have reported all the errors in files I had problems converting but with every archive conversion I only get 1 type of error for 1-2 files. Each time there is a new error. We are using the zip from this link https://data.bus-data.dft.gov.uk/timetable/download/bulk_archive (unzipping all files in the archive in one folder including subfolders and overwriting the files with the same name)

linusnorton commented 3 years ago

Hey, no worries. I've noticed there is an issue where zip files inside a zip file are not processed. Let me fix that and then you can just run it against that one zip file and I'll see how the error handling works.

linusnorton commented 3 years ago

@tafflin I've processed that file and made sure that individual errors are skipped. I've also fixed the underlying issue, although the solution was to just skip over that vehicle because it had no operating profile. Version 1.9.0 should be available now

tafflin commented 3 years ago

@linusnorton thanks, will check soon. If some of the files are skipped or corrupt, is it logged during the conversion?

linusnorton commented 3 years ago

Missing OperatingProfiles are not logged but any other unexpected processing failures are logged

tafflin commented 3 years ago

The archive was processed. Looks good so far. Thanks for quick updates