search

ibi-group / datatools-server

Server for IBI's GTFS data management platform.
MIT License
48 stars 49 forks source link

MobilityData Validator CSV Parsing Issue with large GTFS files #524

Open philip-cline opened 1 year ago

philip-cline commented 1 year ago

Observed behavior

MobilityData Validation Result can run into CSV parsing issues when validating the MBTA's GTFS feed. Running the same feed with the MobilityData Validator locally does not produce the same result.

Expected behavior

Validator should not produce the parsing issue.

Steps to reproduce the problem

Validate MBTA's latest GTFS feed. Observe the following errors: image

Any special notes on configuration used

None

Version of datatools-server and datatools-ui if applicable (exact commit hash or branch name)

Latest validator branch, dev server.

miles-grant-ibigroup commented 1 year ago

I think we can probably get around this by increasing the timeout between report generation time and report collection time but should we maybe consider making the time dynamic based on file size? What do you think @philip-cline

philip-cline commented 1 year ago

I think we should make it dynamic based on file size personally. Wouldn't want to wait the time it takes to validate MBTA for the results to come back for a tiny feed like Golden Gate

miles-grant-ibigroup commented 1 year ago

Please let me know how #525 works for you!