Consider revisiting optimization of load from bucket

[fedorov]

I do not know what is the current logic, but data loading for a DICOM study loaded from a bucket can be very sluggish - this one below takes minutes:

https://volview.kitware.app/?urls=[s3://idc-open-data/ed38c404-6a39-4d06-9484-86204e81f0b9,s3://idc-open-data/fbac83a8-ea0a-40a2-b415-5cbd0118dc3e,s3://idc-open-data/dd0ac33e-a435-4eaa-9f15-ee7ff4d86ee6,s3://idc-open-data/acde3185-defa-4204-9edc-702978604433,s3://idc-open-data/1714b9de-3a93-4f89-82b5-d3b5724410a2,s3://idc-open-data/4a4ee5ac-6e53-47f4-86cf-63536d4a3be0,s3://idc-open-data/1998d15b-17a2-4e7e-ad29-3cd2b9b70fab,s3://idc-open-data/d9bc4f80-67c6-40de-bf0d-7ca694df131d,s3://idc-open-data/16f94d2a-c3b6-40b2-844a-1df4bcc0dfe1,s3://idc-open-data/99c0f6d5-cbf3-4f4b-8572-e8be07518d79,s3://idc-open-data/66c22a8c-b29a-4a13-bcf1-fb12f6139bac,s3://idc-open-data/ac1c80d4-e11c-4203-8c8a-5009b045d18b,s3://idc-open-data/f57c1ac9-2d19-4861-a3fa-247c385a2f59,s3://idc-open-data/6936a577-eb26-4024-b492-59f15c725652,s3://idc-open-data/f01dcf47-b34d-4e34-b121-1927747c7af3,s3://idc-open-data/835d740c-f130-47a8-a5af-713b3e2257a0,s3://idc-open-data/4bfdb5f1-c00a-4e45-b028-e7b439633c28,s3://idc-open-data/8a31ee63-c33c-4c89-a422-cc1b44c2ef59,s3://idc-open-data/0f8d95a5-66a9-452e-8bc0-5ad357a49213]

In case of IDC, we guarantee there is one and only one series in each bucket folder, so one could optimize loading by, for example, creating one entry on the left for each series, and incrementally populating each item one by one. But in general, of course, there can be any kind of mix of instances across buckets.

I guess this speaks for the advantages of the DICOMweb interface, where you could query metadata for each series, instead of having to fetch all of the study into the browser to make sense out of it.

As I said, I am not sure there is a solution, but I wanted to raise this issue since loading for that case above is rather slow. You should at least be aware of this.

[fedorov]

Related to this, would it be possible to change the panel on the right to not prompt the user to drop the files, and instead indicate that they should wait until load is completed? The instructions right now are misleading. This was noted by David Clunie @dclunie.

[aylward]

That is a great idea! When the data loading process begins, we should replace that message with "Loading..."

This is an issue even when the data is local or when integrated into other platforms. Changing that message until the render windows appear and replace it is a perfect solution.

@floryst and @PaulHax - what do you think?

Thanks @fedorov !

[aylward]

@fedorov

I do not know what is the current logic, but data loading for a DICOM study loaded from a bucket can be very sluggish - this one below takes minutes:

It took 8 min and 39 sec to load using the free wifi at Starbucks.

I think a URL option/flag to indicate one series per folder could be used to speedup the initial parse. The system could grab a file from the "middle" of the list of files in the folder as the thumbnail and use it to populate the browser as you suggested. Then we treat the data similar to how we treat the sample data or a response from a DICOMWeb query and only load the data if the user selects it.

@floryst is working on a streaming system that (probably) could be (easily) extended to support this. It is an interesting and perhaps common use case we should consider.

Thanks!

[floryst]

@floryst is working on a streaming system that (probably) could be (easily) extended to support this. It is an interesting and perhaps common use case we should consider.

Yes, I'll be working on data streaming which should substantially improve responsiveness when loading from a bucket.

[fedorov]

Another observation: opening studies that contain both CT and SEG sometime result in opening SEG in the viewer, which I think is suboptimal. If the study contains CT and SEG, it should be CT that is opened by default on the initial open.

Example: https://volview.kitware.app/?urls=[s3://idc-open-data/c879e3b4-89de-4d81-8f94-e09f280b61bf,s3://idc-open-data/16ffa5df-3354-4755-b586-896ec12a9185,s3://idc-open-data/a5660b69-132f-4b6c-83cf-86ed9ec45907,s3://idc-open-data/8bd67531-ec26-4232-a8d4-95b60fe0fcf4,s3://idc-open-data/cdac3f73-4fc9-4e0d-913b-b64aa3100977,s3://idc-open-data/42566d54-25af-440d-94ce-fd553991462e,s3://idc-open-data/a1e77e5c-299f-49ef-a659-f4484e0cedd2,s3://idc-open-data/37cf5cd8-d103-4cd0-a446-960bc0ed92a3,s3://idc-open-data/a64bd0ee-03be-44ef-b46d-2c006e57a7e1]

[aylward]

Great idea! Eventually we'll have SEG objects appear directly in the annotation/segmentation tab, associated with the proper image. In the meantime, we should setup a heuristic for what image to display first when multiple are loaded.