Unable to load data

[BenjiJack]

@theideasmith I am trying to run some of your iPython notebooks in but am having some trouble loading the data.

>>> import data_config as dc
***/neuronal-analysis/data/wbdata/
>>> wormData = dc.kato.data()
>>> len(wormData)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object of type 'NoneType' has no len()

While trying to figure out why no data was being loaded, I noticed in the source code that the data() function requires the data to be cached, so instead I ran:

>>> dc.kato.retrieve()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "biodatamanager.py", line 88, in retrieve
    self.__cache = self.__reader(self.filepath)
  File "***/anaconda/envs/py27/lib/python2.7/site-packages/pandas/io/pickle.py", line 60, in read_pickle
    return try_read(path)
  File "***/anaconda/envs/py27/lib/python2.7/site-packages/pandas/io/pickle.py", line 57, in try_read
    return pc.load(fh, encoding=encoding, compat=True)
  File "***/anaconda/envs/py27/lib/python2.7/site-packages/pandas/compat/pickle_compat.py", line 118, in load
    return up.load()
  File "***/anaconda/envs/py27/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
KeyError: 'v'

Not sure if I'm going about this the wrong way, if there is a dependency problem, or something else. How should I be loading the data?

[lukeczapla]

@BenjiJack Benji, the answer is that the Pickle file was corrupted in archiving and uploading to GIT. Rewrite as follows:

from data_config import kato_matlab t = kato_matlab.data() t.to_pickle('../data/katodata.pickle')

Then you can do what you were doing (using retrieve()) and it should work fine.

[BenjiJack]

@lukeczapla Thanks, Luke.

I got it working with

import data_config as dc
wormData = dc.kato_matlab.retrieve()

dc.kato_matlab.retrieve() and dc.kato_matlab.data() work consistently, even without following your instructions to re-pickle the data. However, for some reason dc.kato.retrieve() and dc.kato.data() are still giving me trouble. I'll stick with kato_matlab for now.

[theideasmith]

The problem is that the pickle file is stored in git large file storage so you'll need to install git-lfs and then run git-lfs pull. Read this for more on git-led usage. https://github.com/github/git-lfs/issues/325.

We'll need to figure out an elegant way of doing this, perhaps moving off git-lfs. Using it maybe was a premature decision. But the above advice should be a temporary workaround

Sent from my iPhone

On Apr 16, 2016, at 2:16 PM, Benjamin Jack notifications@github.com wrote:

@lukeczapla Thanks, Luke.

I got it working with

import data_config as dc wormData = dc.kato_matlab.retrieve() dc.kato_matlab.retrieve() and dc.kato_matlab.data() work consistently, even without following your instructions to re-pickle the data. However, for some reason dc.kato.retrieve() and dc.kato.data() are still giving me trouble. I'll stick with kato_matlab for now.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[BenjiJack]

Thanks @theideasmith. Sounds good.

[slarson]

This looks like it got solved, so closing it. If not, please reopen!