search

flatironinstitute / kachery-cloud

Apache License 2.0
20 stars 4 forks source link

Errors with backend uploading of linked files #8

Closed lfrank closed 2 years ago

lfrank commented 2 years ago

Using the backend script at [https://github.com/LorenFrankLab/spyglass/blob/nwb_kachery/src/spyglass/sharing_backend/sharing_backend_kachery.py]

produces the output below following a task request from a remote system:

Task handler: kachery_store_shared_file.1 (action)
Listening for tasks on project lqqrbobsev
/home/loren/anaconda3/envs/spyglass/lib/python3.8/site-packages/distributed/node.py:177: UserWarning: Port 8787 is already in use.
Perhaps you already have a cluster running?
Hosting the HTTP server on port 43417 instead
  warnings.warn(
Task: kachery_store_shared_file.1
Task: kachery_store_shared_file.1
Task: kachery_store_shared_file.1
Task started: kachery_store_shared_file.1
Task started: kachery_store_shared_file.1
Task started: kachery_store_shared_file.1
Task error: kachery_store_shared_file.1
storing /stelmo/nwb/analysis/CH6120211203_OA2YVMLCC9.nwb in cloud
storing /stelmo/nwb/analysis/CH6120211203_OA2YVMLCC9.nwb in cloud
Task finished: kachery_store_shared_file.1
Task finished: kachery_store_shared_file.1

The remote task request code is below:

    project_id = os.environ['KACHERY_CLOUD_PROJECT'] if 'KACHERY_CLOUD_PROJECT' in os.environ else None
    task_client = TaskClient(project_id=project_id)
    print('Requesting upload task')
    task_client.request_task(
        task_type='action',
        task_name='kachery_store_shared_file.1',
        task_input={
            'uri': uri,
        }

On the remote system, I get 

take message interrupted: 'error'
Exception in thread SubscribeMessageWorker:
Traceback (most recent call last):
  File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/workers.py", line 41, in run
    self._take_message()
  File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/pubnub.py", line 365, in _take_message
    self._process_incoming_payload(msg)
  File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/workers.py", line 177, in _process_incoming_payload
    self._listener_manager.announce_message(pn_message_result)
  File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/managers.py", line 198, in announce_message
    callback.message(self._pubnub, message)
  File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/kachery_cloud/TaskBackend/PubsubListener.py", line 99, in message
    cb(channel=message.channel, message=message.message)
  File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/kachery_cloud/TaskBackend/TaskClient.py", line 49, in handle_message
    error = message['error']
KeyError: 'error'
magland commented 2 years ago

@lfrank There was a typo error in reporting the error, which makes this difficult to troubleshoot. I fixed the error reporting code, so please try again...

but you'll need to install kachery-cloud 0.1.26 from source, since that's not pushed to pypi yet.

pip install --upgrade git+https://github.com/flatironinstitute/kachery-cloud
lfrank commented 2 years ago

Still having trouble with the error message on the client. I’ve checked and it is using 0.1.26

Exception: None

take message interrupted: 'errorMessage' Exception in thread SubscribeMessageWorker: Traceback (most recent call last): File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/workers.py", line 41, in run self._take_message() File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/pubnub.py", line 365, in _take_message self._process_incoming_payload(msg) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/workers.py", line 177, in _process_incoming_payload self._listener_manager.announce_message(pn_message_result) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/managers.py", line 198, in announce_message callback.message(self._pubnub, message) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/kachery_cloud/TaskBackend/PubsubListener.py", line 99, in message cb(channel=message.channel, message=message.message) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/kachery_cloud/TaskBackend/TaskClient.py", line 49, in handle_message error = message['errorMessage'] KeyError: 'errorMessage'

-- Loren M. Frank

Professor Howard Hughes Medical Institute Kavli Institute for Fundamental Neuroscience Departments of Physiology and Psychiatry University of California, San Francisco Office Phone: 415-502-7357

On Jul 7, 2022, at 6:16 AM, Jeremy Magland @.**@.>> wrote:

This Message Is From an External Sender This message came from outside your organization.

@lfrankhttps://urldefense.com/v3/__https://github.com/lfrank__;!!LQC6Cpwp!vBQOdvdYQYhRvKdHAZhr_pi5djJ4n9H3JQaGY3eUeJ-fgaeVtBIZHXDbP4QELsUKOWA1I9JkYnsu4YvyncVFif_igQ$ There was a typo error in reporting the error, which makes this difficult to troubleshoot. I fixed the error reporting code, so please try again...

but you'll need to install kachery-cloud 0.1.26 from source, since that's not pushed to pypi yet.

pip install --upgrade git+https://github.com/flatironinstitute/kachery-cloud

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/flatironinstitute/kachery-cloud/issues/8*issuecomment-1177598263__;Iw!!LQC6Cpwp!vBQOdvdYQYhRvKdHAZhr_pi5djJ4n9H3JQaGY3eUeJ-fgaeVtBIZHXDbP4QELsUKOWA1I9JkYnsu4YvyncXeIcC-pw$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABV4PSJBWXBPYHBGK7R5MMLVS3KDXANCNFSM522VD3LA__;!!LQC6Cpwp!vBQOdvdYQYhRvKdHAZhr_pi5djJ4n9H3JQaGY3eUeJ-fgaeVtBIZHXDbP4QELsUKOWA1I9JkYnsu4YvyncVkbt4R2w$. You are receiving this because you were mentioned.

lfrank commented 2 years ago

Also, just FYI, when I try to download a file for which an upload was previously requested I get

NWB file CH6120211203_OA2YVMLCC9.nwb does not exist locally; checking kachery attempting to download uri sha1://89d8ba74d54d3be6f0e24a182e4f8e21d4e25524 Traceback (most recent call last): File "/Users/loren/Src/NWB/notebooks/sharing_kachery_test.py", line 66, in (CuratedSpikeSorting() & {'nwb_filename': 'CH6120211203.nwb', 'sort_group_id': 1}).fetch_nwb() File "/Users/loren/Src/NWB/spyglass/src/spyglass/spikesorting/spikesorting_curation.py", line 863, in fetch_nwb return fetch_nwb(self, (AnalysisNwbfile, 'analysis_file_abs_path'), *attrs, **kwargs) File "/Users/loren/Src/NWB/spyglass/src/spyglass/common/dj_helper_fn.py", line 82, in fetch_nwb get_nwb_file(file) File "/Users/loren/Src/NWB/spyglass/src/spyglass/common/nwb_helper_fn.py", line 41, in get_nwb_file if not AnalysisNwbfileKachery.download_file(nwb_file_path) and not NwbfileKachery.download_file(nwb_file_path): File "/Users/loren/Src/NWB/spyglass/src/spyglass/sharing/sharing_kachery.py", line 229, in download_file if not kachery_download_file(uri=uri, dest=AnalysisNwbfile.get_abs_path(analysis_file_name)): File "/Users/loren/Src/NWB/spyglass/src/spyglass/sharing/sharing_kachery.py", line 63, in kachery_download_file fname = kcl.load_file(uri, dest=dest) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/kachery_cloud/load_file.py", line 41, in load_file return _load_sha1_file_from_cloud(sha1, verbose=verbose, dest=dest) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/kachery_cloud/load_file.py", line 117, in _load_sha1_file_from_cloud r.raise_for_status() File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://s3.us-east-1.wasabisys.com/kachery-cloud/projects/lqqrbobsev/sha1/89/d8/ba/89d8ba74d54d3be6f0e24a182e4f8e21d4e25524

On Jul 7, 2022, at 9:50 AM, Frank, Loren @.**@.>> wrote:

Still having trouble with the error message on the client. I’ve checked and it is using 0.1.26

Exception: None

take message interrupted: 'errorMessage' Exception in thread SubscribeMessageWorker: Traceback (most recent call last): File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/workers.py", line 41, in run self._take_message() File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/pubnub.py", line 365, in _take_message self._process_incoming_payload(msg) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/workers.py", line 177, in _process_incoming_payload self._listener_manager.announce_message(pn_message_result) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/pubnub/managers.py", line 198, in announce_message callback.message(self._pubnub, message) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/kachery_cloud/TaskBackend/PubsubListener.py", line 99, in message cb(channel=message.channel, message=message.message) File "/Users/loren/opt/anaconda3/envs/spyglass/lib/python3.8/site-packages/kachery_cloud/TaskBackend/TaskClient.py", line 49, in handle_message error = message['errorMessage'] KeyError: 'errorMessage'

-- Loren M. Frank

Professor Howard Hughes Medical Institute Kavli Institute for Fundamental Neuroscience Departments of Physiology and Psychiatry University of California, San Francisco Office Phone: 415-502-7357

On Jul 7, 2022, at 6:16 AM, Jeremy Magland @.**@.>> wrote:

This Message Is From an External Sender This message came from outside your organization.

@lfrankhttps://urldefense.com/v3/__https://github.com/lfrank__;!!LQC6Cpwp!vBQOdvdYQYhRvKdHAZhr_pi5djJ4n9H3JQaGY3eUeJ-fgaeVtBIZHXDbP4QELsUKOWA1I9JkYnsu4YvyncVFif_igQ$ There was a typo error in reporting the error, which makes this difficult to troubleshoot. I fixed the error reporting code, so please try again...

but you'll need to install kachery-cloud 0.1.26 from source, since that's not pushed to pypi yet.

pip install --upgrade git+https://github.com/flatironinstitute/kachery-cloud

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/flatironinstitute/kachery-cloud/issues/8*issuecomment-1177598263__;Iw!!LQC6Cpwp!vBQOdvdYQYhRvKdHAZhr_pi5djJ4n9H3JQaGY3eUeJ-fgaeVtBIZHXDbP4QELsUKOWA1I9JkYnsu4YvyncXeIcC-pw$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABV4PSJBWXBPYHBGK7R5MMLVS3KDXANCNFSM522VD3LA__;!!LQC6Cpwp!vBQOdvdYQYhRvKdHAZhr_pi5djJ4n9H3JQaGY3eUeJ-fgaeVtBIZHXDbP4QELsUKOWA1I9JkYnsu4YvyncVkbt4R2w$. You are receiving this because you were mentioned.