Uploading cellSens VSI image fails

[anttikos]

Uploading a cellSens VSI image to Cytomine fails. The upload seems to work without issues, but after the upload Storage page shows Content type "undetermined", and the image does not get added to the Project, and I'm not able to display the image.

The image itself seems fine, I'm able to display it with OlyVIA viewer.

Below is pasted what is caught on the logs. I'm not sure how to continue to debug any further, so please let me know what other information I can provide to help fix the issue.

ims:/var/log/tomcat7/catalina.out 2021-05-14 08:47:10,334 [pool-2-thread-6] INFO client.HttpClient - Connection to IMS/api/uploadedfile/6805423.json ext : tiff source : /data/images/30/1620981946620/Image_129/Image_129.tiff target : /data/images/30/1620981946620/Image_129/Image_129_pyr.tif /usr/local/bin/vips tiffsave "/data/images/30/1620981946620/Image_129/Image_129.tiff" "/data/images/30/1620981946620/Image_129/Image_129_pyr.tif" --tile --pyramid --compression lzw --tile-width 256 --tile-height 256 --bigtiff tiff2vips: tiled separate planes not supported VipsRegion: valid clipped to nothing VipsRegion: valid clipped to nothing VipsRegion: valid clipped to nothing VipsRegion: valid clipped to nothing VipsRegion: valid clipped to nothing Command return value = 1

[gvincke]

Hi @anttikos VSI files are not natively supported in Cytomine. Once uploaded they are converted into TIFF. VSI format is sometime structured differently regarding the options of the scanner (mono zone, multi zones, etc). This conversion in TIFF do not cover all this variability, and that's why it is better to convert VSI files in pyramidal TIFF BEFORE to upload them to Cytomine. The log you provided here upper indicates that this conversion in TIFF have failed. If you want us to understand why, can you please provide us an example of such an image ? (by wetransfer or equivalent) Please sent it to support at cytomine dot com Thank you.

[anttikos]

Dear @gvincke,

Our team took the approach you suggested earlier, and we saved the image as Pyramidal Tiff instead of VSI, and this time the upload worked, although the original file still got converted. We are trying to avoid unnecessary conversions, as this wastes storage space, and it slows down the upload process - it would be better to upload the images in format the Cytomine supports natively.

I tried to do some digging of my own, to understand why the original file was not recognized by Cytomine, and why the conversion happened. I have attached the tiffinfo outputs of a) the original file and b) the file converted from the original file by Cytomine below.

I also found this message from IMS logs: 2021-12-08 09:05:32,196 [pool-2-thread-8] INFO utils.UploadService - Format = be.cytomine.formats.lightconvertable.specialtiff.BrokenTIFFFormat@eda90a4

Which lead me to checking this piece of code: https://github.com/cytomine/Cytomine-IMS/blob/507e95406aca78f2ada4a5ec903cd39b50d42b8a/src/groovy/be/cytomine/formats/lightconvertable/specialtiff/BrokenTIFFFormat.groovy

By looking at the code, it seems that Cytomine expects "Tile Width" and "Tile Length" tags to be present in the tif metadata, and otherwise it's considered as BrokenTIFFFormat, and requires a conversion. Is this true, and/or is there more to it? If the scanner would provide us .tif files that had these tags present, would it be enough to get the images uploaded into Cytomine without the need of conversion, or can you see other possible issues as well with the metadata pasted below?

Original file tiffinfo output: ` TIFF Directory at offset 0x2032f9f0 (540211696) Image Width: 17584 Image Length: 10191 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Compression Scheme: None Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Rows/Strip: 64 Planar Configuration: single image plane ImageDescription: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>

2021-12-07T08:40:15Z Slide Make: iDS Model: VS-264C DateTime: 2021:12:07 00:40:15 Artist: sqtini Tag 33560: 540211588 EXIFIFDOffset: 0x2032f5a0 TIFF Directory at offset 0x2032f5a0 (540210592) ExposureTime: 0.000257 ExifVersion: 0x30,0x32,0x31,0x30 DateTimeDigitized: 2021:12:07 00:40:15 ApertureValue: 0.130000 SubjectDistance: 0.017000 FlashpixVersion: 0x30,0x31,0x30,0x30 ColorSpace: 1 PixelXDimension: 17584 PixelYDimension: 10191 ExposureMode: 1 TIFF Directory at offset 0x203321d6 (540221910) Image Width: 8 Image Length: 6 Resolution: 96, 96 pixels/inch Bits/Sample: 16 Compression Scheme: None Photometric Interpretation: min-is-black Samples/Pixel: 1 Rows/Strip: 64 Planar Configuration: single image plane Make: Olympus Soft Imaging Solutions GmbH Software: XV Imaging 1.0 DateTime: 2021:12:07 00:40:15 Artist: sqtini Tag 33560: 540221810 ` File converted from the original file by Cytomine, tiffinfo output: ` TIFF Directory at offset 0xd4d9e56 (223190614) Image Width: 17584 Image Length: 10191 Tile Width: 256 Tile Length: 256 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Sample Format: unsigned integer Compression Scheme: LZW Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Planar Configuration: single image plane ImageDescription: 2021-12-07T08:40:15Z Slide TIFF Directory at offset 0x10ed0930 (283969840) Subfile Type: reduced-resolution image (1 = 0x1) Image Width: 8792 Image Length: 5095 Tile Width: 256 Tile Length: 256 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Compression Scheme: LZW Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Planar Configuration: single image plane ImageDescription: 2021-12-07T08:40:15Z Slide TIFF Directory at offset 0x11ca7bd6 (298482646) Subfile Type: reduced-resolution image (1 = 0x1) Image Width: 4396 Image Length: 2547 Tile Width: 256 Tile Length: 256 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Compression Scheme: LZW Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Planar Configuration: single image plane ImageDescription: 2021-12-07T08:40:15Z Slide TIFF Directory at offset 0x11f90478 (301532280) Subfile Type: reduced-resolution image (1 = 0x1) Image Width: 2198 Image Length: 1273 Tile Width: 256 Tile Length: 256 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Compression Scheme: LZW Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Planar Configuration: single image plane ImageDescription: 2021-12-07T08:40:15Z Slide TIFF Directory at offset 0x1205a8c6 (302360774) Subfile Type: reduced-resolution image (1 = 0x1) Image Width: 1099 Image Length: 636 Tile Width: 256 Tile Length: 256 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Compression Scheme: LZW Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Planar Configuration: single image plane ImageDescription: 2021-12-07T08:40:15Z Slide TIFF Directory at offset 0x120ae2a8 (302703272) Subfile Type: reduced-resolution image (1 = 0x1) Image Width: 549 Image Length: 318 Tile Width: 256 Tile Length: 256 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Compression Scheme: LZW Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Planar Configuration: single image plane ImageDescription: 2021-12-07T08:40:15Z Slide TIFF Directory at offset 0x120c9d50 (302816592) Subfile Type: reduced-resolution image (1 = 0x1) Image Width: 274 Image Length: 159 Tile Width: 256 Tile Length: 256 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Compression Scheme: LZW Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Planar Configuration: single image plane ImageDescription: 2021-12-07T08:40:15Z Slide TIFF Directory at offset 0x120d83a2 (302875554) Subfile Type: reduced-resolution image (1 = 0x1) Image Width: 137 Image Length: 79 Tile Width: 256 Tile Length: 256 Resolution: 96, 96 pixels/inch Bits/Sample: 8 Compression Scheme: LZW Photometric Interpretation: RGB color Orientation: row 0 top, col 0 lhs Samples/Pixel: 3 Planar Configuration: single image plane ImageDescription: 2021-12-07T08:40:15Z Slide `