A love to letter to dependencies: describing an environment and set of commits in which spandex runs

[tbuckl]

The TableLoader class is sensitive to the version of Anaconda and SQLAlchemy that one is using.

For example, this commit https://github.com/synthicity/spandex/commit/5356cf9e10e2ef36d176bb6ce413be89c3874bee was necessary to get these scripts to work: https://github.com/synthicity/bayarea_urbansim/blob/master/data_regeneration/run.py#L35-L38

However, that same commit breaks https://github.com/synthicity/bayarea_urbansim/blob/master/data_regeneration/run.py#L35-L38 scripts on Anaconda 2.1.0 on this machine https://github.com/MetropolitanTransportationCommission/bayarea_urbansim_setup/tree/vagrant-ubuntu14-bloomberg

Perhaps this would be a good argument for CI? Or very up-front documentation of current requirements?

[tbuckl]

Well, I actually made a mistake and was not using urbansim v1.3 when this error occurred. But now I've installed Anaconda 2.2 and I'm not looking back! :)

[tbuckl]

Re-Opening because the commit https://github.com/synthicity/spandex/commit/5356cf9e10e2ef36d176bb6ce413be89c3874bee does not work on this machine with Anaconda 2.2 https://github.com/MetropolitanTransportationCommission/bayarea_urbansim_setup/tree/vagrant-ubuntu14-bloomberg

[tbuckl]

this is the error i'm getting when running https://github.com/synthicity/bayarea_urbansim/blob/master/data_regeneration/counties/ala.py

Running model 'export_ala'
Traceback (most recent call last):
  File "counties/ala.py", line 183, in <module>
    sim.run(['export_ala'])
  File "/vm_project_dir/urbansim/urbansim/sim/simulation.py", line 1460, in run
    model()
  File "/vm_project_dir/urbansim/urbansim/sim/simulation.py", line 671, in __call__
    expressions=self._argspec.defaults)
  File "/vm_project_dir/urbansim/urbansim/sim/simulation.py", line 818, in _collect_variables
    variables[label] = thing()
  File "/vm_project_dir/urbansim/urbansim/sim/simulation.py", line 410, in __call__
    return self._call_func()
  File "/vm_project_dir/urbansim/urbansim/sim/simulation.py", line 395, in _call_func
    frame = self._func(**kwargs)
  File "counties/ala.py", line 63, in parcels_out
    index = pd.Series(parcels_in.index).dropna().unique()
  File "/vm_project_dir/urbansim/urbansim/sim/simulation.py", line 177, in index
    return self.local.index
  File "/vm_project_dir/spandex/spandex/io.py", line 483, in index
    index_col=self._index_col).index
  File "/vm_project_dir/spandex/spandex/io.py", line 741, in db_to_df
    q = db_to_query(query)
  File "/vm_project_dir/spandex/spandex/io.py", line 672, in db_to_query
    return sess.query(*orm)
  File "build/bdist.linux-x86_64/egg/sqlalchemy/sql/operators.py", line 316, in __iter__
NotImplementedError: Class <class 'sqlalchemy.orm.attributes.InstrumentedAttribute'> is not iterable

[tbuckl]

ala.py completes without error in an environment with a pythong package list like this:

https://gist.github.com/buckleytom/34fdcd9b8f38fdef7c61

at this commit in bayarea_urbansim: https://github.com/synthicity/bayarea_urbansim/commit/beb14f0026cb0b4a8a3a27f235e5d4314d694aee

and at this commit in spandex: beb14f0026cb0b4a8a3a27f235e5d4314d694aee (which has apparently been removed from the repo)

last available commit: https://github.com/synthicity/spandex/commit/6845f3f8996da5d7441e866590bd89055e5550d1

and running on this machine: https://github.com/MetropolitanTransportationCommission/bayarea_urbansim_setup/tree/version/deblasio

however, it throws the following error at cnc.py:

Running model 'export_cnc'
Traceback (most recent call last):
  File "counties/cnc.py", line 194, in <module>
    sim.run(['export_cnc'])
  File "d:\vms\deblasio\urbansim\urbansim\sim\simulation.py", line 1460, in run
    model()
  File "d:\vms\deblasio\urbansim\urbansim\sim\simulation.py", line 672, in __call__
    return self._func(**kwargs)
  File "counties/cnc.py", line 192, in export_cnc
    df_to_db(df, 'attributes_cnc', schema=staging)
  File "d:\urbansim_windows\spandex\spandex\io.py", line 785, in df_to_db
    df.columns.values.tolist()))
psycopg2.DataError: invalid input syntax for type double precision: "single"
CONTEXT:  COPY attributes_cnc, line 2, column res_type: "single"

[tbuckl]

i just noticed that there's a travis config for spandex. d'oh! starting again from there.

[jiffyclub]

We used to have some CI running for Spandex during a trial period, but because the repo is private we can't use the free travis-ci.org. Haven't yet figured out what our CI options are at Autodesk.

[tbuckl]

good to know. thanks @jiffyclub!

[tbuckl]

here's where i'll be working the travis config into a vagrant/shell setup: https://github.com/MetropolitanTransportationCommission/bayarea_urbansim_setup/tree/vagrant-ubuntu14-giuliani

[janowicz]

When on sqlalchemy v0.8.0, I can recreate the ala.py error posted above on both Windows 7 and Ubuntu 14.04 machines. See the latest commit (sqlalchemy to v0.9.9). When on sqlalchemy 0.9.9, the bayarea_urbansim data regeneration scripts successfully run on both my test machines (including ala.py). Let me know if that helps with ala.py in your environment too. Sorry about specifying v0.8.0 of sqlalchemy- that was a mistake! (though did seem to work for us at the time)

I wrote up the specific steps I just took to run the bayarea data regeneration scripts on a Windows machine (all steps except postgres tuning): https://github.com/synthicity/bayarea_urbansim/wiki/Windows-installation-instructions

Also, have an on-going data regeneration test run using the latest urbansim code (not v1.3 which I had previously been running the scripts on)- so far so good, will let you know how that goes.

[janowicz]

at my commit to spandex, which is pointed to in this commit in your environment, spandex passes all its tests: https://github.com/MetropolitanTransportationCommission/bayarea_urbansim_setup/commit/1d74473cfa0eeed4c53d1d3e789c89c285c50f23

[daradib]

For future reference, it might be worth mentioning that there is a tox.ini file. The goal was to test in a clean environment and against multiple versions of dependencies, without being dependent on Travis CI. It uses virtualenv/pip, not conda though. virtualenv/pip might be a little tricky with binary library dependencies, especially if you're not using a Linux package manager.

The tox.ini file was configured to test against Python 2.7 and 3.4, but it could be used to test against multiple versions of SQLAlchemy.