DICOM to SVS conversion

[petroslk]

• Python version: 3.9.13
• Operating System: Ubuntu 18.04

Description

Hello, I attempted to convert a DICOM wsi to svs using the following command and it failed:

wsic convert -i ANONU3UAJH1JJ_Leica/ -o convert_test/test2.svs  -c jpeg

I manually verified that the DICOM image I was trying to convert was readable on python using wsidicom.

Any ideas on how to do this? Any help would be much appreciated!

Thanks,

Petros

Reading:   0%|                                                                                                                                                                   | 0/28272 [00:00<?, ?it/s$
data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/wsic/readers.py:385: UserWarning: Failed to get next tile after 100 attempts. Dumping debug information.       | 0/113088 [00:00<?, ?it/s$
  warnings.warn(                                                                                                                                                                                           
Reader Shape (58258, 126761, 3)                                                                                                                                                                            
Read Tile Size (512, 512)                                                                                                                                                                                  
Yield Tile Size (256, 256)                                                                                                                                                                                 
Read Mosaic Shape (114, 248)                                                                                                                                                                               
Yield Mosaic Shape (228, 496)                                                                                                                                                                              
Read Index (0, 0)                                                                                                                                                                                          
Yield Index (0, 0)                                                                                                                                                                                         
Remaining Reads (:10) [(0, 3), (0, 4), (0, 5), (0, 6), (0, 7), (0, 8), (0, 9), (0, 10), (0, 11), (0, 12)]                                                                                                  
Enqueued {(0, 1), (0, 2), (0, 0)}                                                                                                                                                                          
Reordering Dict (keys) dict_keys([])                                                                                                                                                                       
Queue Size 0                                                                                                                                                                                               
Intermediate Read slices (slice(0, 256, None), slice(0, 256, None))                                                                                                                                        
Reading:   0%|                                                                                                                                                                   | 0/28272 [00:10<?, ?it/s$
Writing:   0%|                                                                                                                                                                  | 0/113088 [00:11<?, ?it/s$
Traceback (most recent call last):                                                                                                                                                                         
  File "/data/home/pliako/anaconda3/envs/wsic/bin/wsic", line 8, in <module>                                                                                                                               
    sys.exit(main())                                                                                                                                                                                       
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/click/core.py", line 1128, in __call__                                                                                           
    return self.main(*args, **kwargs)                                                                                                                                                                      
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/click/core.py", line 1053, in main                                                                                               
    rv = self.invoke(ctx)                                                                                                                                                                                  
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/click/core.py", line 1659, in invoke                                                                                             
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/click/core.py", line 1395, in invoke                                                                                             
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                         
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/click/core.py", line 754, in invoke                                                                                              
    return __callback(*args, **kwargs)
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/wsic/cli.py", line 223, in convert
    writer.copy_from_reader(
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/wsic/writers.py", line 942, in copy_from_reader
    tif.write(
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/tifffile/tifffile.py", line 2556, in write
    chunk = next(dataiter)
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/wsic/readers.py", line 406, in __next__
    raise IOError(f"Tile read timed out at index {self.yield_index}")
OSError: Tile read timed out at index (0, 0)
Exception ignored in: <function MultiProcessTileIterator.__del__ at 0x7f179e266f70>
Traceback (most recent call last):
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/wsic/readers.py", line 509, in __del__
    self.close()
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/site-packages/wsic/readers.py", line 501, in close
    executor.map(lambda p: p.join(1), self.processes)
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/concurrent/futures/_base.py", line 598, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/concurrent/futures/_base.py", line 598, in <listcomp>
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "/data/home/pliako/anaconda3/envs/wsic/lib/python3.9/concurrent/futures/thread.py", line 169, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

[John-P]

@petroslk I would love to look into this for you. Are you able to share an image that are trying to convert so that I can try to replicate the issue?

[John-P]

You may also want to try installing the latest test version: https://test.pypi.org/project/wsic/0.6.1/

pip install -i https://test.pypi.org/simple/ wsic==0.6.1

[John-P]

Hi @petroslk, have you had any luck with this? I can try to replicate the issue if you can let me know the version of wsic. A sample image that you are having trouble with would also be helpful.

[petroslk]

Hey @John-P , the latest version also seems to be having the same issue. I've sent you a slide for you to try and replicate it! Thanks a lot for your help!

[John-P]

Thanks @petroslk, I'll try to replicate the issue and see if I can come up with a fix.

[John-P]

Hi @petroslk when I run the conversion with wsic using the file you sent to me I get an error from wsidicom which says that the transfer syntax in the DICOM file is not supported (1.2.840.10008.1.2.1 AKA Explicit VR Little Endian). Do you not see this message? It may also be a differently encoded file as the name is different from in the above example.

[John-P]

Hi @petroslk, I plan to close this due to inactivity. Please can you let me know if this has been resolved?

[petroslk]

Hey there,

Sorry for the late reply, the issue still persists.

I should soon have a properly formated DICOM WSI that does not throw this warning and that I can share with you to see if you can replicate the issue. I hope to have this by next week.

[petroslk]

Hey there @John-P , I hope you received my DICOM WSI! Let me know if you need anything else!

[John-P]

I have managed to get the conversion to work with the file that you sent me. I had to increase the time option (-to). This file is unusually slow to convert for some reason though. It will require some further investigation.

[John-P]

I was able to perform fast (and lossless) conversion of the pixel data in file that you sent me to a .tiff using wsic transcode (~1 min) and then convert this file to a .svs using wsic convert as a workaround. I have also confirmed that it opens in Aperio ImageScope. I hope to add transcode/re-packaing support from DICOM to SVS soon as this will be must faster.

[John-P]

I have figured out that the problem is that creating a wsidicom reader object is very expensive. It looks like pydicom is trying to iterate over all the frames (tiles) at startup (or similar), which is unnecessary and expensive. I can limit the creation of the reader to once at the start of the process. This leads to a long warm up but much faster conversion after. To properly fix this issue, there needs to be bug fixes to the wsidicom and pydicom packages, or I need to implement my own reader.

On my machine (a six core Intel i5 with 64 GB RAM), converting this file took over 15 minutes. I believe that this conversion time could be cut down significantly, simply by making the warm-up less expensive.

[petroslk]

Thanks a lot for your help, indeed, it takes way too long with the direct conversion.

I used the following command to transcode to a .tiff file:

wsic transcode -i /media/petros/PortableSSD/R22000049-1-A67-2___427399/ -o test.tiff

But it seems to flatten the pyramid and openslide then seems unable to find the mpp values when I run the second command:

wsic convert -i test.tiff -o test_svs_direct.svs -c jpeg

 UserWarning: OpenSlide could not find MPP.
  self.microns_per_pixel = self._get_mpp()

It seems like the following bfconvert command can do the trick when it comes to converting DICOM WSI to OME.TIFF:

bfconvert -noflat -compression JPEG R22000049-1-A67-2___427399/1.3.6.1.4.1.36533.147143106742241217322159521414914625468143.dcm conved_dicom.ome.tif

Would you mind sharing the commands you used? Thanks a lot.

[John-P]

Sure, I'll include some commands below.

The transcode function does not (currently) attempt to copy the pyramid resolutions. Wsic as a whole only operates on the baseline full resolution image. This is to keep the implementation simple, and because the downsample method used can vary from tool to tool and sometimes be of poor quality or corrupted. It was also targeted as computational applications where high levels are not typically needed, versus for viewing applications where they are commonly needed. You can, however, easily ask wsic to generate pyramid levels but supplying the -d options. Future versions of the tool could attempt to copy existing level data, however this may complicate the implementation and I would like to keep it as simple as possible. Wsic could have an option to recreate the scales of the levels without too much complexity, which may be quite useful and a nice middle ground.

I'll have to look into the resolution issue, that sounds like a bug.

DCM to TIFF

$ wsic transcode -i <input_path.dcm> -o <output_path.tiff>

TIFF to SVS

$ wsic convert -i <input_path.tiff> -o <output_path.svs> -c jpeg -cl 90 -d 4 -d 8 -d 32

You can change the pyramid scales to other values, but I found these to work well with ImageScope.

Your files are likely slow due to using sparse tiling. I have been working on a branch to improve performance with such DICOM files. I have also raised an issue on the wisidcom repo and got a helpful answer here. I am also working on a branch to ask the user if they would like to make a full tiled copy before conversion, and the maintainers of wsidicom have also just added some optimizations which may help.

[John-P]

I'll try and make a release soon with all of the PRs merged to fix or alleviate some of these issues.

[John-P]

Version 0.8.0 is now out on pypi which may help. Also using wsidicom 0.9.0 may improve things.

EDIT: I made a careless mistake in the update! Hotfix patch coming soon. EDIT: All sorted in 0.8.2 🤞

[John-P]

Using wsidicom 0.9.0 and the latest wsic (https://github.com/John-P/wsic/pull/102), copying is now much faster even without making a TILED_FULL copy first. Hopefully you find the same :)