search

seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
38 stars 8 forks source link

Working in read-only environments #26

Closed perlman closed 3 years ago

perlman commented 3 years ago

Cloud-files fails to start when run in a read-only environment:

File "./app/datasource.py", line 3, in <module>
    from cloudvolume import CloudVolume
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudvolume/__init__.py", line 49, in <module>
    from .cloudvolume import CloudVolume, register_plugin
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudvolume/cloudvolume.py", line 7, in <module>
    from .exceptions import UnsupportedFormatError
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudvolume/exceptions.py", line 1, in <module>
    from cloudfiles.exceptions import CompressionError, DecompressionError
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudfiles/__init__.py", line 12, in <module>
    from .cloudfiles import CloudFiles
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudfiles/cloudfiles.py", line 25, in <module>
    from .interfaces import (
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudfiles/interfaces.py", line 18, in <module>
    from .connectionpools import S3ConnectionPool, GCloudBucketPool, MemoryPool, MEMORY_DATA
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudfiles/connectionpools.py", line 12, in <module>
    from .secrets import google_credentials, aws_credentials
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudfiles/secrets.py", line 13, in <module>
    CLOUD_FILES_DIR = mkdir(os.path.join(HOME, '.cloudfiles', 'secrets'))
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudfiles/lib.py", line 51, in mkdir
    os.makedirs(path)
  File "/usr/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/www/.cloudfiles'

The culprit appear to be a hard coded use of directories in the home directory here.

I am using cloud-volume to access a local volume, so authentication & secrets should not be required.

Apologies if I'm missing some obvious work around. One thought I have would be to allow something like $CLOUD_FILES_HOME to be used an override, allowing me to set the path to a temp directory.

william-silversmith commented 3 years ago

Fixed in version 1.14.1 !

william-silversmith commented 3 years ago

If you need the $CLOUD_FILES_HOME feature, let's open another issue.

perlman commented 3 years ago

Thanks William!

perlman commented 3 years ago

The problem still exists with cloud-volume:

File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudvolume/connectionpools.py", line 10, in <module>
    from .secrets import google_credentials, aws_credentials
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudvolume/secrets.py", line 12, in <module>
    CLOUD_VOLUME_DIR = mkdir(os.path.join(HOME, '.cloudvolume'))
  File "/opt/services/transform_service_dev/env/lib/python3.8/site-packages/cloudvolume/lib.py", line 135, in mkdir
    os.makedirs(path)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/www/.cloudvolume'

Is this a leftover from before cloud-files?

william-silversmith commented 3 years ago

Yep, sorry about that.

william-silversmith commented 3 years ago

It's not completely obsolete because CV needs to check for boss and chunked graph tokens that aren't applicable to CloudFiles.

william-silversmith commented 3 years ago

CV 3.1.1 is out and should fix at least that problem.