search

comic / evalutils

evalutils helps users create extensions for grand-challenge.org
https://grand-challenge.org
MIT License
23 stars 9 forks source link

Could not load test .png images in segmentation challenge #326

Closed prash-p closed 2 years ago

prash-p commented 2 years ago

Description

I am testing a segmentation challenge container. I have 520 images in the ground-truth/ folder and 520 binary segmentations produced by a segmentation algorithm in the test/ folder

When I run ./test.sh I get the error: Could not load img_n.png using <evalutils.io.SimpleITKLoader object at 0x7f021d9a0860> 520 times where n is the image number [1-520]

and then evalutils.exceptions.FileLoaderError: Could not load any files in /input with <evalutils.io.SimpleITKLoader object at 0x7fd2158b62d0>.

Oddly, when I copy the ground-truth images to the test folder the ./test.sh script runs fine and I get 100% accuracy as expected. However when using the test images generated by the algorithm the test.sh fails with the above error.

Seems related to: https://github.com/comic/evalutils/issues/136

jmsmkn commented 2 years ago

As the ground truth images load fine this points to something being wrong with the images produced by the algorithm as SimpleITK cannot load them using ReadImage(...) (https://github.com/comic/evalutils/blob/d77c80d6420980a886302237ca321d09478a3db2/evalutils/io.py#L151).

Have you checked things like file permissions, and that the images can be read by some other python process? You could also try updating SimpleITK within the container.

SimpleITK probably isn't needed when using PNG images anyway, instead, there is another loader for images that your could try. Try setting file_loader=ImageIOLoader() (from evalutils.io.ImageIOLoader) in the super().__init__(...) method for your evaluation.

prash-p commented 2 years ago

Thanks! It was a file permission issue, which I fixed with chmod and now the test works as expected.

I'm running into another issue now. After exporting the container and uploading it to grand-challenge.org, I uploaded the submission (same as test images above). However it has been in the 'preparing' stage for multiple hours now, which has not happened before. Any idea why this could be?

jmsmkn commented 2 years ago

Please email support with details of your challenge and the evaluation ID.