dgutman
commented
1 year ago So big big point.. We can either IMPORT a file into our local assetstore, OR what I do 99% of the time, is we create indexes/references to files that already exist on a NAS somewhere. In that case, you have to make sure the docker container has been configured so the volume is bind-mounted into the container.
I often have a docker-compose.override.yml file (similar to below), that will mount
version: '3' services: girder: volumes:
For local development, uncomment the set of mounts associated with
the
local source files. Adding the editable egg directories first
allows
allow mounting source files from the host without breaking the
internal
data.
- /home/dagutman/devel/testData:/testData
- /myLocalNAS:/myLocalNasSo then under the existing admin console --> Assetstores, there's a magical IMPORT DATA button (in green), once your file system / nas /whatever is "visable" within the main girder docker container, you can then have it walk that file path and it will import the data into the filesystem.. again in this mode you are INDEXING the files and grabbing metadata from the file header, but are NOT copying 10TB of files into a mongo database. As you mentioned, that would be a total disaster.
The only thing the actual MONGO database stores is the actual internal DSA metadata, annotation documents, etc.
The assetstore is a separate folder / resource where things people have "manually" uploaded i.e. via drag & drop through the uI would appear.
[image: image.png]
[image: image.png]
On Sat, Feb 25, 2023 at 5:33 PM André Pedersen @.***> wrote:
I have a local database with about 10 TB of WSIs. It is not practical to have duplicates of these inside a separate MongoDB database which DSA uses. The original database is also used for other stuff than DSA, and thus, relying fully on MongoDB only is not a viable option.
I had the impression that DSA supported two ways of storing data:
- Upload WSI and store it directly in the MongoDB database (stores a copy of the image)
- Upload WSI directly from filesystem where the server is launched (only stores a reference/path to the image)
For option 2 I assumed, perhaps naively, that it only stored the reference to the WSI inside the database and not the entire WSI. However, I just noticed that the collection I was working with had stored ~22 GB of data when the original test cohort was on about ~16.3 GB. Hence, I would think that it stores the full images (in addition to annotations and other stuff). Hence, my understanding is likely wrong.
If you do not storing references in DSA, wouldn't that be a great idea? This is what QuPath does and enables the user to make "infinite" projects without running out of storage. I understand that it is convenient to have the images in the database itself as the DSA solution would be more stable and less prone to external forces, such as the user suddenly moving the images on disk, and then the reference would no longer be valid - resulting in corrupted IDs in the database and all that fun stuff.
For exactly this issue, QuPath has added a feature to enable users to fix broken image paths by enabling them to set the new path manually or asking for a directory of where the WSI should be where QuPath would then run a search to find the missing image. I find this extremely useful in a variety of situations and it is honestly the only practical solution for me.
What are your thoughts on this idea? I don't mean to replace storing images, but rather having an alternative and less storage-hungry alternative.
— Reply to this email directly, view it on GitHub https://github.com/DigitalSlideArchive/digital_slide_archive/issues/254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFODTQMZU3DHTFPLNHP5KLWZKCCTANCNFSM6AAAAAAVIDBSVY . You are receiving this because you are subscribed to this thread.Message ID: @.***>
-- David A Gutman, M.D. Ph.D. Associate Professor of Neurology Emory University School of Medicine
andreped
manthey
I have a local database with about 10 TB of WSIs. It is not practical to have duplicates of these inside a separate MongoDB database which DSA uses. The original database is also used for other stuff than DSA, and thus, relying fully on MongoDB only is not a viable option.
I had the impression that DSA supported two ways of storing data:
For option 2 I assumed, perhaps naively, that it only stored the reference to the WSI inside the database and not the entire WSI. However, I just noticed that the collection I was working with had stored ~22 GB of data when the original test cohort was on about ~16.3 GB. Hence, I would think that it stores the full images (in addition to annotations and other stuff). Hence, my understanding is likely wrong.
If you do not storing references in DSA, wouldn't that be a great idea? This is what QuPath does and enables the user to make "infinite" projects without running out of storage. I understand that it is convenient to have the images in the database itself as the DSA solution would be more stable and less prone to external forces, such as the user suddenly moving the images on disk, and then the reference would no longer be valid - resulting in corrupted IDs in the database and all that fun stuff.
For exactly this issue, QuPath has added a feature to enable users to fix broken image paths by enabling them to set the new path manually or asking for a directory of where the WSI should be where QuPath would then run a search to find the missing image. I find this extremely useful in a variety of situations and it is honestly the only practical solution for me.
What are your thoughts on this idea? I don't mean to replace storing images, but rather having an alternative and less storage-hungry alternative.