search

seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
38 stars 8 forks source link

crc check raises error when the issue is permissions #21

Closed fcollman closed 3 years ago

fcollman commented 4 years ago

I received this error in the lastest version of cloud-volume. When i specified use_https=True, it went away. I suspect that the crc checking is failing when a permissions failure is at fault. The error message is not helpful as it behaves now.


Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/app/autoretry.py", line 35, in run
    return task._orig_run(*args, **kwargs)
  File "/app/materializationengine/workflows/live_materialization.py", line 321, in get_cloudvolume_supervoxel_ids
    cv = cloudvolume.CloudVolume(segmentation_source, mip=0)
  File "/usr/local/lib/python3.7/site-packages/cloudvolume/cloudvolume.py", line 204, in __new__
    return REGISTERED_PLUGINS[path.format](**kwargs)
  File "/usr/local/lib/python3.7/site-packages/cloudvolume/datasource/graphene/__init__.py", line 69, in create_graphene
    mesh = GrapheneMeshSource(meta, cache, config)
  File "/usr/local/lib/python3.7/site-packages/cloudvolume/datasource/graphene/mesh/__init__.py", line 12, in __new__
    mesh_meta = GrapheneMeshMetadata(meta, cache)
  File "/usr/local/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/mesh/metadata.py", line 17, in __init__
    self.info = self.fetch_info()
  File "/usr/local/lib/python3.7/site-packages/cloudvolume/datasource/precomputed/mesh/metadata.py", line 71, in fetch_info
    info = self.cache.download_json(self.meta.join(self.mesh_path, 'info'))
  File "/usr/local/lib/python3.7/site-packages/cloudvolume/cacheservice.py", line 354, in download_json
    res = self.download( [ path ], compress=compress, progress=False )
  File "/usr/local/lib/python3.7/site-packages/cloudvolume/cacheservice.py", line 477, in download
    raise frag['error']
  File "/usr/local/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 174, in download
    check_md5(path, content, server_hash)
  File "/usr/local/lib/python3.7/site-packages/cloudfiles/cloudfiles.py", line 145, in check_md5
    path, server_hash, computed_md5
cloudfiles.exceptions.MD5IntegrityError: graphene_meshes_minnie3_v1/info failed its md5 check. server md5: Y computed md5: Y+GFY7Q7Ph2Pwy6cnb6dVQ==
william-silversmith commented 4 years ago

Hi Forrest,

I ran into this problem myself. The issue is that I didn't parse the md5 correctly in google-cloud-python when I submitted my patch.

https://github.com/googleapis/python-storage/pull/258

You can try switching to an earlier version of CV until the patch comes out. I might release a version of CV that allows you to disable md5 checks depending on how long this will take to fix.

fcollman commented 4 years ago

ah thanks!

william-silversmith commented 3 years ago

The md5 logic was reactivated a few days ago in the last release. I'm going to close this, but please reopen if you run into problems!