search

RSGInc / bca4abm

Benefit Cost Analysis for Travel Demand Models
http://rsginc.github.io/bca4abm/
Other
7 stars 5 forks source link

load_data_processor to import data into hdf5 store #57

Closed toliwaga closed 6 years ago

toliwaga commented 8 years ago

It might be worthwhile to have a load_data_processor that reads the csv data files into an hdf5 store and then the individual processors could read their input from the store (a la activitysim.)

This might be a lot faster, as the load process wold only need to be run if the input data changed, which might be convenient while the model is being initially built, tweaked, and re-run with the same data, but revised specs and settings.

Also, it would make it more flexible because different versions of the load_data_processor could read data from different sources, including reading the data from, say, a MSSQL database.

mabcal commented 8 years ago

Could the loader automatically store the “last modified” time stamp from the CSV file when it loads it into the store, and then only load it again if the csv file name and/or the time stamp has changed?

bstabler commented 6 years ago

this issue is no longer relevant now that we're using the asim pipeline