search

trynmaps / metrics-mvp

Prototype of public transit data visualization system
https://muni.opentransit.city
MIT License
30 stars 34 forks source link

Store cache files as CSV instead of JSON #609

Closed hathix closed 4 years ago

hathix commented 4 years ago

On #580, Jesse left this feedback:

Good call on storing cache files as CSV to simplify appending chunks. It seems like we don't need to store the JSON format anymore: trynapi.get_state could rename each temporary CSV file to a permanent cache path when it's done. CachedState.get_for_route could return a pandas DataFrame similar to the DataFrame constructed from JSON by eclipses.produce_buses. the route_state parameter in eclipses.py could be a DataFrame instead of a dict.

This probably would simplify the code as well as improving performance. Do you want to work on this now or create an issue?

Eddy said:

I've updated the PR message with some performance comparisons. Good point that using CSV's would simplify things in eclipses and improve performance there - I'll make an issue for it. I'll let you merge this as to ensure it works properly on the Kubernetes/GCloud environment.

Originally posted by @EddyIonescu in https://github.com/trynmaps/metrics-mvp/pull/580#issuecomment-591233480