Support Minimalist Cloudpaths

[schlegelp]

Hi Will.

I'm trying to access the Buhmann 2019 FAFB synapse data with cloud volume (see this neuroglancer instance).

CloudVolume doesn't seem to like its path:

>>> vol = CloudVolume('precomputed://gs://neuroglancer-20191211_fafbv14_buhmann2019_li20190805')
UnsupportedProtocolError: 
Cloud Path must conform to FORMAT://PROTOCOL://BUCKET/PATH
Examples: 
  precomputed://gs://test_bucket/em
  gs://test_bucket/em
  graphene://https://example.com/image/em

Supported Formats: None (precomputed), graphene, precomputed, boss
Supported Protocols: gs, file, s3, matrix, http, https

Cloud Path Recieved: precomputed://gs://neuroglancer-20191211_fafbv14_buhmann2019_li20190805

Alternative error:

>>> vol = CloudVolume('precomputed://gs://neuroglancer-20191211_fafbv14_buhmann2019_li20190805', use_https=True)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/metadata.py in to_mip(self, mip)
    554       else:  # mip specified by index into downsampling hierarchy
--> 555         return self.available_mips[mip]
    556     except Exception:

~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/metadata.py in available_mips(self)
    480     """Returns a list of mip levels that are defined."""
--> 481     return range(len(self.info['scales']))
    482 

KeyError: 'scales'

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-64-c62c70a48aed> in <module>
----> 1 vol = CloudVolume('precomputed://gs://neuroglancer-20191211_fafbv14_buhmann2019_li20190805', use_https=True)

~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/cloudvolume.py in __new__(cls, cloudpath, mip, bounded, autocrop, fill_missing, cache, compress_cache, cdn_cache, progress, info, provenance, compress, compress_level, non_aligned_writes, parallel, delete_black_uploads, background_color, green_threads, use_https, max_redirects, mesh_dir, skel_dir, agglomerate)
    202     path = strict_extract(cloudpath)
    203     if path.format in REGISTERED_PLUGINS:
--> 204       return REGISTERED_PLUGINS[path.format](**kwargs)
    205     else:
    206       raise UnsupportedFormatError(

~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/__init__.py in create_precomputed(cloudpath, mip, bounded, autocrop, fill_missing, cache, compress_cache, cdn_cache, progress, info, provenance, compress, compress_level, non_aligned_writes, parallel, delete_black_uploads, background_color, green_threads, use_https, max_redirects, mesh_dir, skel_dir, **kwargs)
     90       meta, cache, config,
     91       image, mesh, skeleton,
---> 92       mip
     93     )
     94 

~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/frontends/precomputed.py in __init__(self, meta, cache, config, image, mesh, skeleton, mip)
     54     # needs to be set after info is defined since
     55     # its setter is based off of scales
---> 56     self.mip = mip
     57     self.pid = os.getpid()
     58 

~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/frontends/precomputed.py in mip(self, mip)
    267   @mip.setter
    268   def mip(self, mip):
--> 269     self.config.mip = self.meta.to_mip(mip)
    270 
    271   @property

~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/metadata.py in to_mip(self, mip)
    560         opening_text = "MIP {}".format(str(mip))
    561 
--> 562       scales = [ ",".join(map(str, scale)) for scale in self.available_resolutions ]
    563       scales = [ "<{}>".format(scale) for scale in scales ]
    564       scales = ", ".join(scales)

~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/metadata.py in available_resolutions(self)
    484   def available_resolutions(self):
    485     """Returns a list of defined resolutions."""
--> 486     return (s["resolution"] for s in self.scales)
    487 
    488   @property

~/.pyenv/versions/3.7.5/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/metadata.py in scales(self)
    424   @property
    425   def scales(self):
--> 426     return self.info['scales']
    427 
    428   @scales.setter

KeyError: 'scales'

As always: greatly appreciate your help with this! :)

Best, Philipp

[schlegelp]

Also: apologies if this is an already known issue or if it's an error on my end. Tried with cloud volume 2.0.1

[william-silversmith]

Ack, sorry I missed this. The notice got lost in my inbox. It looks like at some point neuroglancer stopped enforcing the PROTOCOL://BUCKET/DATASET/LAYER scheme. We'll have to adapt CV to support these kinds of paths.

[william-silversmith]

Working on a first iteration here: https://github.com/seung-lab/cloud-files/pull/23

We'll need to then introduce support in CloudVolume.

[william-silversmith]

The CloudFiles work is done. The CloudVolume work is still in progress.