wsp-sag / network_wrangler

A Python Library for Managing Travel Model Network Scenarios
https://wsp-sag.github.io/network_wrangler/
Apache License 2.0
13 stars 4 forks source link

🚀 [FEAT] Serialize scenario object (to fix) [BUG] Cannot pickle scenario object #376

Closed i-am-sijia closed 1 month ago

i-am-sijia commented 1 month ago

as a user I'd like to be able to save scenarios and be able to load them with a record of all their applied projects

Background

MetCouncil would just pickle the scenario, but for some reason some thing (specifically lambda functions) are "unpicklable". It's not entirely clear why lambda functions would need to be pickled, but suspect it is b/c the validation requirements are within the network object instances.

Resolution Options

Pickle vs YAML vs networks-only

  1. Store scenarios w/out networks as pickles myscen.pickle() myscen = load_pickle()
  2. Store scenarios as a human-readable toml/yaml, mapping to stored networks and project card filenames myscen.write() myscen = load_scenario()
  3. Don't worry about scenarios, just add a variable to networks themselves that store the applied project names net.projects and add a YAML file to network output specifying applied projects

Where store project data

  1. Writing out projectcards to files in subfolder
  2. Add project card data to network YAML
  3. Just rely on project names

Describe the bug

Rachel likes to save scenario objects out as pickle files via pickle.dump(version_00_scenario, open(working_scenario_filename, 'wb')), and load them via pickle.load(open(working_scenario_filename, 'rb')). This allows her restart from a previous scenario pickle file and not need to reapply previous project cards. The benefit of having the scenario saved is also that it is easy to check what project cards have been applied via scenario.applied_projects.

With the new Network Wrangler, we currently cannot use pickling. When running pickle.dump(), we are getting:

---------------------------------------------------------------------------
PicklingError                             Traceback (most recent call last)
Cell In[29], [line 2](vscode-notebook-cell:?execution_count=29&line=2)
      [1](vscode-notebook-cell:?execution_count=29&line=1) working_scenario_filename = os.path.join(net_dir, 'v00', 'standard_networks', '00_scenario.pickle')
----> [2](vscode-notebook-cell:?execution_count=29&line=2) pickle.dump(version_00_scenario, open(working_scenario_filename, 'wb'))

PicklingError: Can't pickle <function RoadwayNetwork.<lambda> at 0x00000268C5E04A60>: attribute lookup RoadwayNetwork.<lambda> on network_wrangler.roadway.network failed

@e-lo What would be an alternative to the scenario pickle?

To Reproduce

Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Failing tests

Triggering line of code

Thoughts on resolution

Full stack trace

Environment

Operating system: Context (conda, jupyter, etc): Environment (e.g. output from conda list):

e-lo commented 1 month ago

MTC / @lmz and MetCouncil / @RachelWikenMC indicated a preference for:

e-lo commented 1 month ago

Fixed in #379 and merged to V1