search

observatoire-mobilite / odm-website

prototype of ODM's website
https://observatoire-mobilite.github.io/odm-website
0 stars 0 forks source link

Allow for a more convenient, tabular input format on PassengerServiceGrid #95

Open gilgeorges opened 7 months ago

gilgeorges commented 7 months ago

Right now I have this convoluted object format, which has some nice speed advantages as compared to a tabular layout. But it is somewhat inconvenient for exports. Let's change that.

gilgeorges commented 7 months ago

So the new format can be generated straight by calling write_json on a polars.DataFrame.

gilgeorges commented 7 months ago

The code I added in the dev/ndjson branch accepts the new format but seems significantly slower when clicking a new bus line or stop. Also the automatic to another year a newly selected line or stop does not have any data for it is now broken.

Note: I rejected the original idea of using ndjson, because it seems more appropriate with streaming applicaitons, and that I'd have to do artificially in my case so there is no real advantage over conventional JSON.

gilgeorges commented 7 months ago

The broken getFirstValidYear is a bit of a conundrum: looking up data in my new large flat JSON file is significantly slower, it seems. Also getFirstValidYear lives in the store, but everything else is done in memos around the store. So looking up what years are available in the store either requires re-determining which rows are actually available for a given line or stop, which is slow, or significantly refacotring te code and moving bits to the store,