search

planarnetwork / dtd2mysql

MySQL / MariaDB import for DTD feeds (fares, timetable and routeing)
30 stars 10 forks source link

Document GTFS feed date cutoff #17

Closed mk-fg closed 7 years ago

mk-fg commented 7 years ago

As bde9423 was unlikely to be unintentional, guess it'd be best to document that limitation, especially because some exported data might end up being incorrect due to skipped stp=O/N/C records.

linusnorton commented 7 years ago

Yes, there was a change in the last two data files that vastly increased the amount of data after three months. For some odd reason it had an exponential effect on the processing time (which I should probably look into). As a work around I added that filter, which is the same logic other journey planners use.

mk-fg commented 7 years ago

Hmm, interesting, ttis625 / 639 / 646 (latest from atoc) seem to be pretty much same in size.

Though I've had node's GC hit memory limits myself when importing one of these (don't remember which), and suspect that it might potentially be a cause for a massive slowdown, if node tries to invoke it more-and-more when approaching some pre-calculated point based on compiled-in defaults or the amount of RAM. Resolved the issue myself by using ts-node --max_old_space_size=8000 --max_new_space_size=8000, but wanted to mention here in case it might be an easy explaination for a massive slowdown even if algo itself seem to be linear.