Clean up and document data conversion

[hanase]

The README instructions for data conversion is out of date. Needs a description of creating data for estimation. In addition,

• we should have a copy of the latest estimation cache (@pcaballero - where is it?) as well as simulation cache in one place for the conversion and document where that is. An older version of these caches is on modelsrv8 in d:/opusgit/urbansim_data/data/psrc_parcel/base_year_2014_inputs/urbansim2_cache and urbansim2_cache_estimation. The reason why we have copies is that there is an extra table (zoning_heights) that is not available in opus caches that we use. The table is needed for simulation as well as estimation (of REPM). It also allows us to make urbansim2-specific changes to the data.

• a ready to use converted hdf5 files, for both, estimation and simulation could be available somewhere. Document where it is. A previous version of a simulation file is available on modelsrv3 (/d/udst/psrc_urbansim/data/psrc_base_year_2014.h5), since udst is setup there.

• update the setup on modelsrv3 for people to use.

• the file estimation_cache_to_hdf5.py could be renamed as it converts also non-estimation data.

Sorry @stefancoe, these could be four separate issues. Do you think you could work on some of these? Thanks!

[stefancoe]

Sure thing @hanase. I'll be able to work on this next week.

[hanase]

@stefancoe - I created the simulation and estimation caches in \\MODELSRV8\d$\opusgit\urbansim_data\data\psrc_parcel\base_year_2014_inputs, called urbansim2_cache and urbansim2_estimation_cache.

I converted them into h5 using your conversion script (which needs to be renamed and modified as suggested in #132 and #123). These hdf5 files are available on \\MODELSRV3\d$\udst\psrc_urbansim\data as psrc_base_year_2014.h5 and psrc_estimation_2014.h5.

Please update the README files with this info.

[hanase]

Also please add the requirement of installing pylogit into the README.

[stefancoe]

Hana, I have consolidated the two conversion scripts

[stefancoe]

And updated the readme.

[hanase]

@stefancoe - the conversion script throws an error:

$ python cache_to_hdf5.py --help Traceback (most recent call last): File "cache_to_hdf5.py", line 248, in sys.exit(main()) File "cache_to_hdf5.py", line 239, in main if len(args) == 2: TypeError: object of type 'NoneType' has no len()

Also, the conversion README is still referencing the estimation_cache_to_hdf5.py. Can we make the three years a default, so that user does not need to put it into the command line?

[hanase]

@stefancoe - Could you also please add the pylogit reference to README? Thanks!

[stefancoe]

@hanase - looks like you fixed the pylogit reference in the README. Thanks! I tested the data conversion script and it works for both regular and estimation cache. I fixed a typo in the data conversion README. It looks like modelsrv3 has recent data sets. When we merge newdeveloper into dev, we can have someone test the code on modelsrv3 and makes sure everything is working.