search

LBNL-UCB-STI / beam

The Framework for Modeling Behavior, Energy, Autonomy, and Mobility in Transportation Systems
https://transportation.lbl.gov/beam
Other
147 stars 57 forks source link

ActivitySim skims: using OMX format #3232

Closed nikolayilyin closed 1 year ago

nikolayilyin commented 3 years ago

ActivitySim team is using OMX format for skims, this format is pretty fast for reading and mode effective for storage comparing to csv file.

There are libraries for Python and Java to work with OMX. In this python notebook there are examples of how to work with OMX as well as code to produce OMX skims out of BEAM csv skims.

We need to be able to produce our activitySim skims in OMX format, either by creating them in OMX or converting them from csv.gz into OMX.

Probably a simple scala util to convert csv.gz skims into OMX would be better

dimaopen commented 2 years ago

Within this task, the project libraries for Python and Java was ported to gradle so that it can be easily used as a dependency. The repository is here https://github.com/LBNL-UCB-STI/omx-java. A sample activitysim skim file in OMX format was generated and provided to the Urbansim team for review.

dimaopen commented 2 years ago

@zneedell The Urbansim team has not reviewed the new skim format. Should we close this issue?

zneedell commented 2 years ago

@dimaopen Since we opened this I've spent enough time working with the omx skims in pilates where I think I should be able to confirm that they're working as expected. I'll test it out this week and hopefully get it merged soon -- thanks for re-surfacing this

dimaopen commented 2 years ago

@zneedell The branch that I used to create the OMX skims is https://github.com/LBNL-UCB-STI/beam/tree/do/%233232-activitysim-skims-omx-format. Please let me know what to do with this task.

dimaopen commented 1 year ago

This was implemented with this PR https://github.com/LBNL-UCB-STI/beam/pull/3736.