Mismatch in values for KACHERY_CLOUD_DIR

[samuelbray32]

Describe the bug

• Users having issues accessing some kachery files through the cloud.
• Digging into the request shows it arises from a file not found issue
• Current values for kachery_cloud_dir in spyglass.sharing.KacheryZone table are "/stelmo/nwb/.kachery_cloud" (with an underscore). This value is used to define where the linked sha1:// file is saved.
• Current install instructions say to set the KACHERY_CLOUD_DIR environment variable as "/stelmo/nwb/.kachery-cloud" (with a dash). This value in the server daemon's environment is what directory is used to look for sha1 file when requested
• This mismatch results in a file not found on the server end (note that there are files in "/stelmo/nwb/.kachery-cloud" as well. Unclear when these were made and with what settings.

Possible Solutions

1. Delete the entries in spyglass.sharing.KacheryZone and reinsert. (Not ideal due to disrupting data)
2. Edit the secondary key values for kachery_cloud_dir and then merge the file directories "/stelmo/nwb/.kachery_cloud" and "/stelmo/nwb/.kachery-cloud"
3. Other ideas?

[edeno]

I think it should be fairly easy to edit the secondary key values. Are there many entries in spyglass.sharing.KacheryZone?

@CBroz1 thoughts?

[samuelbray32]

I think it should be fairly easy to edit the secondary key values. Are there many entries in spyglass.sharing.KacheryZone?

@CBroz1 thoughts?

There's only two entries. One for franklab.default and one for franklab.collaborators .

[edeno]

Okay, I suspect this is dropable. Or we can just change the value for franklab.collaborators.

[lfrank]

As a follow-up: as far as I know everything in the Kachery tables was for testing only, so if it’s easier to just delete things, I think that’s fine.

On Nov 28, 2023, at 2:08 PM, Eric Denovellis @.***> wrote:

This Message Is From an External Sender This message came from outside your organization.

I think it should be fairly easy to edit the secondary key values. Are there many entries in spyglass.sharing.KacheryZone?

@CBroz1https://urldefense.com/v3/__https://github.com/CBroz1__;!!LQC6Cpwp!uqeMoTT3TzAyX14M9lchrKegKE0C2rM0CclZuud4BvlVDWiqhH5X2-wpvS8bUGPznVeSah6LUFFXoFygd0WJbWDS5CM$ thoughts?

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/LorenFrankLab/spyglass/issues/693*issuecomment-1830837478__;Iw!!LQC6Cpwp!uqeMoTT3TzAyX14M9lchrKegKE0C2rM0CclZuud4BvlVDWiqhH5X2-wpvS8bUGPznVeSah6LUFFXoFygd0WJ2jdxgLI$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABV4PSM4CTPWH7EQP6Z7KETYGZOFVAVCNFSM6AAAAAA76OMRI2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZQHAZTONBXHA__;!!LQC6Cpwp!uqeMoTT3TzAyX14M9lchrKegKE0C2rM0CclZuud4BvlVDWiqhH5X2-wpvS8bUGPznVeSah6LUFFXoFygd0WJznqnzUw$. You are receiving this because you are subscribed to this thread.

[CBroz1]

This might be caused by oversight on my part. In the SpyglassConfig class, I think I used an underscore in the default value for this environmental variable. I'll plan to

1. Edit existing table entries to reflect the hyphen - easy enough for secondary key values
2. Edit the settings.py reflect the hyphen
3. Add entries for the position-specific config items we've discussed separately

[samuelbray32]

Update on the progress for this issue:

I tried changing the environment variable for KACHERY_CLOUD_DIR in the daemons environment to where the data is. After doing so, a request for data from the user end Printed the following message: Error: Error executing kachery-cloud-store: Command exited with code 1

The server daemon print the following output: Upload request: sha1://ceac0c1995580dfdda98d6aa45b7dda72d63afe4 Starting upload: sha1://ceac0c1995580dfdda98d6aa45b7dda72d63afe4

This suggests the file is now being found. However, kachery reports no data being uploaded. Trying to see if this stems from user permissions when I run the daemon. Open to other suggestions of what could be the breaking point

[edeno]

I would suggest contacting @magland in the #sortingview channel to see if he has any thoughts.

[samuelbray32]

Fixed the remaining issue. For future reference, the terminal running the server daemon needs read access to /stelmo/nwb/.kachery_cloud/private.pem (or the equivalent for the kachery zone it's running)

• [x] Fix secondary keys in spyglass.sharing.KacheryZone to hyphen instead of underscore
• [x] Merge the underscore version of the KACHERY_CLOUD_DIR with the hyphen version in the stelmo filesystem

[CBroz1]

Thanks @samuelbray32 !