Python CompassApp run_from_file method supports CSV, newline JSON with batching

NREL / routee-compass

The RouteE-Compass energy-aware routing engine

https://nrel.github.io/routee-compass/

BSD 3-Clause "New" or "Revised" License

9 stars 4 forks source link

Open robfitzgerald opened 4 months ago

robfitzgerald commented 4 months ago

in order to run very large batches of queries, we want a method on a CompassApp which

takes a filename and chunksize argument (with reasonable default)
by testing the filename suffix, chooses pandas from_csv or from_json with lines=True
chunks the input, passes it to CompassApp.run

question: is it possible to do this without 1) converting each dataframe to a python List[Dict], 2) stringifying the List?

robfitzgerald commented 1 month ago

The CLI has been updated with this feature and the logic of writes should have the correct append output behavior to support batching from Python.

It would seem like this should just be a simple wrapper function in Python that creates the matches, such as the pandas chunking API.

nreinicke commented 2 weeks ago

This is directly related to #209