Storing PyVIN results in HOBBES database

[msdogan]

@jrmerz we would like to store PyVIN results in HOBBES database. Currently, only HEC-PRM results can be stored HOBBES. Link flow and storage time-series can be found in PyVIN repo https://github.com/msdogan/pyvin/tree/master/examples/statewide_82-year_results. I am not sure what other things we store in HOBBES. I guess only flow and storage, right? Results are all in .csv format. We also have raw results in .json but due to its size, it is hard to move around that file.

@jdherman please let us know if you like to store other results such as dual values.

[jrmerz]

It should be very easy to update the repo from these CSV files. Just let me know what fields. Flow and Storage?

[msdogan]

I think flow and storage would be good for now

[jrmerz]

@msdogan I have added ability to update repo from pyvin results (5e9fd124228484ef8695014dd994ccf491ae1ad4). This command is rough right now, currently it takes a csv file and a property and overwrites the existing data. The csv file is assumed to contain date and kaf. Like I said, rough.

I have also:

• branched the S09105
• pushed the PyVIN flows and storage to master

[msdogan]

@jrmerz that looks great! If want to create a run using PyVIN results as boundary conditions, would the following be the command? I am guessing you haven't posted changes to npm

AD3+msdogan@CWS-Amargosa  /z/calvin-network-tools (master)
$ node ./bin/cli.js matrix --format=csv --start=2002-2 --stop=2002-4 --to=network -O=nodes --ts=. --max-ub=1000000000000 --verbose SR_SHA D5

Also, is there a way to store PyVIN results from command line in case I would like to do that myself in the future? Thanks a lot!

[jrmerz]

@msdogan

Sorry, not sure what you are asking on your first question. To use the new PyVIN flows, simply pull master on calvin-network-data, then re-run matrix command.

As for updating from CSV file. I have pushed new command to calvin-network-tools@3.0.2 update flow:

cnf apply-changes csv --file path/to/flow.csv  --property flow

update storage:

cnf apply-changes csv --file path/to/storage.csv  --property storage

And remember if something bad happens and the data repo get messed up, no worries! Just run:

cd calvin-network-data
git reset HEAD --hard
git clean -d -f

This will rest to the version of the data on github as well as remove any untracked files (in case new files were added).

Let me know if you have any questions.

[msdogan]

thanks! I have exported the matrix, and I will try to upload PyVIN results back into HOBBES.

[msdogan]

@jrmerz I wanted to try uploading results back to database but I am getting an error.

AD3+msdogan@CWS-Amargosa  ~/Desktop/GitHub
$ cnf apply-changes csv --file Z:\pyvin\examples\statewide_82-year_results\flow.csv --data C:\Users\msdogan\Desktop\GitHub\calvin-network-data\data --property flow
Invalid file: undefined

I have flow.csv in Z:\pyvin\examples\statewide_82-year_results\flow.csv and copied the whole calvin-network-data to C:\Users\msdogan\Desktop\GitHub\calvin-network-data\data not to mess with original repo. But it says invalid file Do you know what would be the reason for this?

Also tried following but still the same

AD3+msdogan@CWS-Amargosa  ~/Desktop/GitHub
$ cnf apply-changes csv --file Z:\pyvin\examples\statewide_82-year_results\flow.csv --property flow
Invalid file: undefined

[jrmerz]

My guess is windows. Will get my windows env up and running when I have a sec

[jrmerz]

@msdogan ok, couple things. First, passing paths from the command line windows (specially from a bash like terminal) you should put the path in quotes.

cnf apply-changes csv --file "Z:\pyvin\examples\statewide_82-year_results\flow.csv" --property flow

Second there was some badness in the crawl in windows as well as error reporting. I have pushed fixes for those at calvin-network-tools@3.0.4

Give a pull and let me know how things look.

[msdogan]

@jrmerz where do I need to run this? in calvin-network-data folder or this is a global command. I updated my calvin-network-tools repo and installed 3.0.4 but still getting an error below.

AD3+msdogan@CWS-Amargosa  /z/calvin-network-tools (master)
$ npm install -g calvin-network-tools
npm WARN deprecated node-uuid@1.4.7: use uuid module instead
Z:\nodejs\cnf -> Z:\nodejs\node_modules\calvin-network-tools\bin\cli.js
Z:\nodejs
`-- calvin-network-tools@3.0.4
  +-- concat-stream@1.6.0
  | `-- readable-stream@2.2.3
  +-- hobbes-network-format@2.0.9
  +-- rimraf@2.6.0
  `-- xlsx@0.8.5

AD3+msdogan@CWS-Amargosa  /z/calvin-network-data (master)
$ cnf apply-changes csv --file "Z:\pyvin\examples\statewide_82-year_results\flow.csv" --property flow
Z:\nodejs\node_modules\calvin-network-tools\node_modules\hobbes-network-format\lib\crawler\toGeojson.js:70
    var origin = nodePathLookup[originPath].data;
                                           ^

TypeError: Cannot read property 'data' of undefined
    at Z:\nodejs\node_modules\calvin-network-tools\node_modules\hobbes-network-format\lib\crawler\toGeojson.js:70:44
    at Array.forEach (native)
    at module.exports.networkFiles.data.region.forEach.networkFiles.data.region.forEach.networkFiles.data.node.forEach.networkFiles.data.link.forEach.networkFiles.data.node.forEach.networkFiles.data.region.forEach (Z:\nodejs\node_modules\calvin-network-tools\node_modules\hobbes-network-format\lib\crawler\toGeojson.js:58:26)
    at Z:\nodejs\node_modules\calvin-network-tools\node_modules\hobbes-network-format\lib\crawler\index.js:19:7
    at onResp (Z:\nodejs\node_modules\calvin-network-tools\node_modules\hobbes-network-format\lib\git.js:40:11)
    at Z:\nodejs\node_modules\calvin-network-tools\node_modules\hobbes-network-format\lib\git.js:51:9
    at ChildProcess.exithandler (child_process.js:204:7)
    at emitTwo (events.js:100:13)
    at ChildProcess.emit (events.js:185:7)
    at maybeClose (internal/child_process.js:827:16)

[jrmerz]

What version of windows are you on again?

[msdogan]

This is Windows 7

[jrmerz]

k, try again @ calvin-network-tools@3.0.5

[msdogan]

@jrmerz I got following. Does it mean it updated the database?

AD3+msdogan@CWS-Amargosa  /z/calvin-network-tools (master)
$ npm install -g calvin-network-tools                                           npm WARN deprecated node-uuid@1.4.7: use uuid module instead
Z:\nodejs\cnf -> Z:\nodejs\node_modules\calvin-network-tools\bin\cli.js
Z:\nodejs
`-- calvin-network-tools@3.0.5
  +-- rimraf@2.6.1
  `-- xlsx@0.8.6
    `-- cfb@0.11.0

AD3+msdogan@CWS-Amargosa  /z/calvin-network-tools (master)
$ cnf apply-changes csv --file "Z:\pyvin\examples\statewide_82-year_results\flow.csv" --property flow
Unknown parameter names: ags_sd-sink,ags_vent-sink,ags_bard-sink,ags_owens-sink,c122-sink,c131-sink,c150-sink,c155-sink,c20-sink,c301-sink,c315-sink,c32-sink,c35-sink,c49-sink,c5-sink,c52-sink,c53-sink,c54-sink,c56-sink,c58-sink,c62-sink,c64-sink,c688-sink,c689-sink,c76-sink,c83-sink,c97-sink,cn2-sink,d31-sink,d37-sink,d509-sink,d517-sink,d605-sink,d613-sink,d643-sink,d647-sink,d653b-sink,d66-sink,d664-sink,d699-sink,d703-sink,d73-sink,d743-sink,d744a-sink,d745-sink,d749-sink,d751-sink,d804-sink,d848-sink,d849-sink,d85-sink,d853-sink,d861-sink,d870-sink,d872-sink,d885-sink,d886-sink,d887-sink,d897-sink,d94-sink,desal-sink,eres_cc-sink,eres_ebmud-sink,eres_ewmwd-sink,eres_sf-sink,eres_cmwd-sink,eres_sdwd-sink,ext_baker-sink,ext_blythe-sink,ext_clwa-sink,ext_sb_slo-sink,ext_stock-sink,gw_im-sink,inflow-c106,inflow-c116,inflow-c146,inflow-c148,inflow-c154,inflow-c161,inflow-c2,inflow-c23,inflow-c27,inflow-c28,inflow-c29,inflow-c304,inflow-c308,inflow-c32,inflow-c35,inflow-c37,inflow-c38,inflow-c40,inflow-c5,inflow-c51,inflow-c52,inflow-c54,inflow-c57,inflow-c65,inflow-c7,inflow-c77,inflow-c86,inflow-c87,inflow-c89,inflow-cn2,inflow-d17,inflow-d31,inflow-d37,inflow-d43a,inflow-d509,inflow-d606,inflow-d612,inflow-d616,inflow-d64,inflow-d642,inflow-d646,inflow-d653b,inflow-d66,inflow-d663,inflow-d691,inflow-d692,inflow-d73,inflow-d74,inflow-d75,inflow-d76b,inflow-d77,inflow-d85,inflow-d889,inflow-d9,inflow-d94,inflow-d98,inflow-desal,inflow-gw_01,inflow-gw_02,inflow-gw_03,inflow-gw_04,inflow-gw_05,inflow-gw_06,inflow-gw_07,inflow-gw_08,inflow-gw_09,inflow-gw_10,inflow-gw_11,inflow-gw_12,inflow-gw_13,inflow-gw_14,inflow-gw_15,inflow-gw_16,inflow-gw_17,inflow-gw_18,inflow-gw_19,inflow-gw_20,inflow-gw_21,inflow-gw_av,inflow-gw_ch,inflow-gw_ew,inflow-gw_im,inflow-gw_mj,inflow-gw_mwd,inflow-gw_ow,inflow-gw_sbv,inflow-gw_sc,inflow-gw_sd,inflow-gw_vc,inflow-hxcmwd,inflow-hxewmwd,inflow-hxsd,inflow-n1,inflow-n13,inflow-n14,inflow-n201,inflow-n202,inflow-n7,inflow-n8,inflow-sr_ber,inflow-sr_blb,inflow-sr_buc,inflow-sr_bul,inflow-sr_cfw,inflow-sr_cle,inflow-sr_clk_inv,inflow-sr_cr3,inflow-sr_crw,inflow-sr_dnp,inflow-sr_fol,inflow-sr_gnt,inflow-sr_hid,inflow-sr_hth,inflow-sr_isb,inflow-sr_ll_enr,inflow-sr_mcr,inflow-sr_mil,inflow-sr_nhg,inflow-sr_nml,inflow-sr_par,inflow-sr_pnf,inflow-sr_rll_cmb,inflow-sr_scagg,inflow-sr_scc,inflow-sr_sha,inflow-sr_trm,inflow-sr_tul,inflow-sr_whi,inflow-wtp205,inflow-wtp503,mexico-sink,n13-sink,n14-sink,n201-sink,n202-sink,req_delta-sink,sr_cr1-sink,sr_ol-sink,surp_delta-sink,wwp205-sink,wwp208-sink,wwp209-sink,wwp301-sink,wwp304-sink,wwp406-sink,wwp501-sink,wwp502-sink,wwp503-sink,wwp504-sink,wwp505-sink,wwp506-sink,wwp507-sink,wwp508-sink,wwp509-sink,wwp511-sink

[msdogan]

seems to work on Windows, too. Closing the issue.