DigitalSlideArchive / digital_slide_archive

The official deployment of the Digital Slide Archive and HistomicsTK.
https://digitalslidearchive.github.io
Apache License 2.0
108 stars 49 forks source link

Change AssetStore type to S3 not working. #152

Closed hiracky16 closed 2 years ago

hiracky16 commented 3 years ago

Hi. I changed AssetType from filesystem to Amazon S3. After uploading, I can save an image at S3 bucket and mongodb documents. However, can't display the image.(Image extension is svs.)

At the time of local filesystem, mongodb item documents has largeImage but at the time of S3 don't has largeImage. (Please refer to the following documents.) I think this is the cause.

How I can save largeImage in an item document?

# the different of mongodb item document
# S3
{
    "_id": ObjectId("xxxxxxxxxxxxxx"),
    "name": "test.svs",
    "description": "",
    ...
}

# Filesystem
{
    "_id": ObjectId("xxxxxxxxxxx"),
    "name": "test.svs",
    "description": "",
    ...
    "largeImage": {
        "fileId": ObjectId("xxxxxxx"),
        "sourceName": "openslide"
    }
}

Error screen スクリーンショット 2021-06-27 14 40 40

manthey commented 3 years ago

How are you deploying the system? Specifically, the default deployments of the DSA use "girder mount" to give filesystem-like access to any assetstore. If this isn't available, most large image types can't be read from S3 directly. If you are using Docker, the default docker deployments ask for privileged access AND the local system needs libfuse installed (or its equivalent). If you are using a default deployment, you might just need to apt-get install fuse and restart the DSA.

hiracky16 commented 3 years ago

@manthey Thank you for your reply.

How are you deploying the system?

I try with using Docker on my local machine.(S3 is used from AWS.)

If you are using Docker, the default docker deployments ask for privileged access AND the local system needs libfuse installed (or its equivalent).

I'm probably using default Docker. At digital_slide_archive/devops/dsa First, I will try it.

hiracky16 commented 3 years ago

@manthey I installed macfuse (and sshfs) on My Macbook. MacFuse site https://osxfuse.github.io/

After installing, I recreated an assetstore using S3 and restarted DSA on docker compose. However, it isn't solved.

Do I have to do anything after installing?

hiracky16 commented 3 years ago

At the time of local file system, a large image was automatically generated after uploading. But at S3, genareted when clicking following icon.

Why automatically generate a large image at the time of using S3? スクリーンショット 2021-07-03 12 48 19

manthey commented 3 years ago

There is a setting in the Admin Console / Plugins / Large Image called "Automatically use new items as large images". By default, if the files (locally or on S3) have specific file extensions or report specific mime types they will automatically be used as large images. If they have names without file extensions, you might want to change the setting to "Automatically try to use all files as large images".

hiracky16 commented 3 years ago

@manthey I tryed changing setting of large images, but not working automatically. The extension of the uploaded file is "svs". However, There are files named by Hash value( ex. 035d2a5xxxxx ) on S3. They have no extension.

Is this the correct behavior for the assetstore ?

manthey commented 2 years ago

I'm closing this issue -- to the best of my knowledge, if you have "girder mount" running, everything should work on S3 even without extensions. If there is a specific instance that can be accessed showing otherwise, we can reopen and debug further.