OphysBehaviorSession dataset does not load attribute stimulus_templates

[DowntonCrabby]

Describe the bug the OphysBehaviorSession dataset attribute stimulus_templates fails to load due to a FileNotFoundError for warped_grating_0.png

To Reproduce

from allensdk.brain_observatory.behavior.behavior_ophys_session import BehaviorOphysSession
ophys_experiment_id = 994278291 # this could be any experiment id
dataset = BehaviorOphysSession.from_lims(ophys_experiment_id)
dataset.stimulus_templates

Expected behavior the stimulus_templates dictionary to load

Actual Behavior

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-5-bad8a943bb54> in <module>
----> 1 dataset.stimulus_templates

c:\users\kater\ai_python_packages\allensdk\allensdk\core\lazy_property\lazy_property_mixin.py in __getattribute__(self, name)
     11 
     12         lazy_class = super(LazyPropertyMixin, self).__getattribute__('LazyProperty')
---> 13         curr_attr = super(LazyPropertyMixin, self).__getattribute__(name)
     14         if isinstance(curr_attr, lazy_class):
     15             return curr_attr.__get__(curr_attr)

c:\users\kater\ai_python_packages\allensdk\allensdk\brain_observatory\behavior\behavior_session.py in stimulus_templates(self)
    324             columns contains image arrays, and the df.name is the image set.
    325         """
--> 326         return self._stimulus_templates.to_dataframe()
    327 
    328     @stimulus_templates.setter

c:\users\kater\ai_python_packages\allensdk\allensdk\core\lazy_property\lazy_property_mixin.py in __getattribute__(self, name)
     13         curr_attr = super(LazyPropertyMixin, self).__getattribute__(name)
     14         if isinstance(curr_attr, lazy_class):
---> 15             return curr_attr.__get__(curr_attr)
     16         else:
     17             return super(LazyPropertyMixin, self).__getattribute__(name)

c:\users\kater\ai_python_packages\allensdk\allensdk\core\lazy_property\lazy_property.py in __get__(self, obj, objtype)
     18 
     19         if self.value is None:
---> 20             self.value = self.calculate()
     21         return self.value
     22 

c:\users\kater\ai_python_packages\allensdk\allensdk\core\lazy_property\lazy_property.py in calculate(self)
     28 
     29     def calculate(self):
---> 30         result = self.api_method(*self.args, **self.kwargs)
     31         for wrapper in self.wrappers:
     32             result = wrapper(result)

c:\users\kater\ai_python_packages\allensdk\allensdk\brain_observatory\behavior\session_apis\data_transforms\behavior_data_transforms.py in get_stimulus_templates(self)
    252             "gratings_0.0": {
    253                 "warped": np.asarray(imageio.imread(
--> 254                     "/allen/programs/braintv/production/visualbehavior"
    255                     "/prod5/project_VisualBehavior/warped_grating_0.png")),
    256                 "unwarped": np.asarray(imageio.imread(

~\Anaconda\envs\SDK_VBA_37\lib\site-packages\imageio\core\functions.py in imread(uri, format, **kwargs)
    263 
    264     # Get reader and read first
--> 265     reader = read(uri, format, "i", **kwargs)
    266     with reader:
    267         return reader.get_data(0)

~\Anaconda\envs\SDK_VBA_37\lib\site-packages\imageio\core\functions.py in get_reader(uri, format, mode, **kwargs)
    170 
    171     # Create request object
--> 172     request = Request(uri, "r" + mode, **kwargs)
    173 
    174     # Get format

~\Anaconda\envs\SDK_VBA_37\lib\site-packages\imageio\core\request.py in __init__(self, uri, mode, **kwargs)
    122 
    123         # Parse what was given
--> 124         self._parse_uri(uri)
    125 
    126         # Set extension

~\Anaconda\envs\SDK_VBA_37\lib\site-packages\imageio\core\request.py in _parse_uri(self, uri)
    258                 # Reading: check that the file exists (but is allowed a dir)
    259                 if not os.path.exists(fn):
--> 260                     raise FileNotFoundError("No such file: '%s'" % fn)
    261             else:
    262                 # Writing: check that the directory to write to does exist

FileNotFoundError: No such file: 'C:\allen\programs\braintv\production\visualbehavior\prod5\project_VisualBehavior\warped_grating_0.png'

Environment (please complete the following information):

• OS & version: Windows 10
• Python version: 3.7
• AllenSDK version: 2.8.0

[wbwakeman]

Hmmm...but this does work

>>> from allensdk.brain_observatory.behavior.behavior_ophys_session import BehaviorOphysSession
>>> file = '/allen/programs/braintv/production/visualbehavior/prod4/specimen_962177507/ophys_session_994137503/ophys_experiment_994278291/behavior_ophys_imaging_plane_994278291.nwb'
>>> dataset = BehaviorOphysSession.from_nwb_path(file)
>>> dataset.stimulus_templates
                                                     unwarped                                             warped
image_name
im065       [[nan, nan, nan, nan, nan, nan, nan, nan, nan,...  [[198, 197, 201, 207, 207, 202, 198, 201, 207,...
im077       [[nan, nan, nan, nan, nan, nan, nan, nan, nan,...  [[129, 135, 140, 142, 140, 135, 129, 124, 117,...
...

In fact, even the from_lims() works for me too:

>>> ophys_experiment_id = 994278291
>>> dataset2 = BehaviorOphysSession.from_lims(ophys_experiment_id)
>>> dataset2.stimulus_templates
                                                     unwarped                                             warped
image_name
im065       [[nan, nan, nan, nan, nan, nan, nan, nan, nan,...  [[198, 197, 201, 207, 207, 202, 198, 201, 207,...
im077       [[nan, nan, nan, nan, nan, nan, nan, nan, nan,...  [[129, 135, 140, 142, 140, 135, 129, 124, 117,...
im066       [[nan, nan, nan, nan, nan, nan, nan, nan, nan,...  [[83, 85, 82, 78, 78, 81, 84, 84, 82, 82, 84, ...
...

[DowntonCrabby]

I've just checked this morning and I'm still getting this error.

I am on the master branch of the SDK v2.8.0

[danielsf]

@DowntonCrabby Can you independently verify whether or not you can see that file from whatever machine you are using? When I log onto informatics' synapse server, I can see that file. Thanks.

[DowntonCrabby]

@danielsf well right now it looks like the function is looking to find the warped_grating_0.png on my local computer instead of the network, could that be the issue? Perhaps its a linux vs windows filepath issue?

[danielsf]

That is almost certainly what the problem is. Now to figure out how to automatically learn where /allen/ is mounted on an arbitrary Windows machine....

[danielsf]

Do you mind running

from allensdk.internal.core.lims_utilities import safe_system_path
naive_path = '/allen/programs/braintv/production/visualbehavior/prod5/project_VisualBehavior/warped_grating_0.png'
print(safe_system_path(naive_path))

and seeing if the result is the actual path to that file?

(There must be a way to cast linux /allen/ paths to Windows /allen/ paths; we just don't seem to be using it here)

[DowntonCrabby]

when running the codeblock you just sent I get the following filepath: \allen\programs\braintv\production\visualbehavior\prod5\project_VisualBehavior\warped_grating_0.png

for doing this with future filepaths would it work to just do filepath.replace('/allen', '//allen') ? I believe that works with both windows and linux filepaths

update there are actually 2 slashes before allen. For whatever reason github does not display both of them.

[danielsf]

Is that the filepath that gets you to warped_grating_0.png over the network?

I ask because that safe_system_path() method is how we cast the path to the pickle file into whatever form is needed for the user's machine. I assume that, if your machine could not find the pickle files, we would have noticed by now.

https://github.com/AllenInstitute/AllenSDK/blob/master/allensdk/brain_observatory/behavior/session_apis/data_io/behavior_ophys_lims_api.py#L102-L117

[danielsf]

@DowntonCrabby I have a possible fix. Do you mind installing branch ticket/1964/dev, installing and re-running? I'd test it myself, but, as you have probably inferred, I do not have a Windows machine.

[DowntonCrabby]

Yes I'll check, but please also see the update above. There are actually two slashes before allen to access that image on the network via the windows filepath. It looks like github wont display the second one.

[DowntonCrabby]

@danielsf
Running it via the ticket/1964/dev branch worked!

[danielsf]

Phew.

Thanks. I'll open a PR so that this will be in 2.9.0 (or is this blocking you such that we need to think about releasing a 2.9.1)?

[danielsf]

@DowntonCrabby Small change in the fix due to code review. Do you mind running the test again. I did some forced pushes on the branch, so you won't be able to just pull it. Let me know if I'm speaking gibberish.

[DowntonCrabby]

I immediately get an error on import

[danielsf]

Yeah.... that is because I force pushed to the repository, meaning your history and GitHub's history are out of sync. Sorry. We software types like our code to be aesthetically pleasing.

Do this:

git commit -a -m ''
git checkout master
git branch -D ticket/1964/dev
git pull
git checkout ticket/1964/dev

effectively, you are deleting your local copy of the branch and then pulling the branch back down from GitHub. Should fix the problem (that <<<<<<<<< HEAD line you are getting a complaint about is something git stuck in there when it could not reconcile your history and mine).

[DowntonCrabby]

gotcha. Okay I followed those instructions and it looks like I'm back to the original error:

[danielsf]

Dan just warned me that pathlib (the library I used for the new fix) "doesn't work well with //allen".

Good to know.

I'll go back to the fix we know works and call that good.

Thanks for testing this.