Closed AjeyPaiK closed 1 month ago
After doing some fact finding, I have narrowed the issue down to the backend we use while loading the WSIs during dataset creation.
I made a manifest with just 2 images (one in fit and one in validate) and I started tracking the open file handlers using lsof
.
I looked for the obvious ones that could be accessed: the .log
files, the image files ending with .svs
, .json
files and the .cache
files
With just 2 images, I expect that only a reasonable number of file handlers for the .svs
images are open.
After about 51 epochs, I see that there are 144 open file handlers for the two .svs
files I'm using.
Moreover, these file handlers are all opened for just those two images.
When I change the backend to OPENSLIDE, there are no open file handlers corresponding to these .svs
files.
@jonasteuwen What do you think needs to be done about this?
This issue is closed since the bug should be handled in upstream dlup.
The PR mentioned above fixes it.
Describe the bug I am training a segmentation model with the
ZarrFileImageWriter
and theComputeWsiMetricsCallback
. The training runs successfully for many epochs (in this case, it was 82 epochs). Then abruptly crashes with the following error stack:To Reproduce Run a training for a segmentation model with the following settings -
Datamodule settings: BATCH_SIZE=64 VALIDATION_BATCH_SIZE=126 NUM_WORKERS=16
write_file_callback settings:
Expected behavior A clear and concise description of what you expected to happen.
Environment dlup version: 0.5.1 How installed: pip Python version: 3.11 Operating System: Linux
Additional context N/A