Mismatch size for test input during inference

[mustansarfiaz]

Hi

I found your paper interesting, I want to produce the inference score using the provided UNet weights. However, I observed that there is a mismatch for the test dataset.

Could you please look into it. here is the error:

Global seed set to 0 using checkpoint epoch=33-val_loss=0.59-auroc=0.985.ckpt Lightning automatically upgraded your loaded checkpoint from v1.8.2 to v1.9.5. To apply the upgrade to your files permanently, runpython -m pytorch_lightning.utilities.upgrade_checkpoint --file checkpoints/marinedebrisdetector/a_unet1/epoch=33-val_loss=0.59-auroc=0.985.ckpt` GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs

Dataset Composition total

train flobs_dataset (train): 39194 shipsdataset: 29833 MARIDA (train): 2017

val refinedflobs_val: 2451 MARIDA (val): 1185

test flobstestdataset: 2197 maridatestdataset: 872

Dataset Composition debris/non-debris train flobs_dataset (train): 19587/19607 shipsdataset: 0/29833 MARIDA (train): 930/1087

val refinedflobs_val: 868/1583 MARIDA (val): 616/569

test flobstestdataset: 903/1294 maridatestdataset: 270/602 LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2] /home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/lightning_fabric/loggers/csv_logs.py:184: UserWarning: Experiment logs directory ./checkpoints/marinedebrisdetector/a_unet1/test_log/version_0 exists and is not empty. Previous log files in this directory will be deleted when the new ones are saved! f"Experiment logs directory {self.log_dir} exists and is not empty." /home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:229: PossibleUserWarning: The dataloader, test_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument(try 128 which is the number of cpus on this machine) in theDataLoader` init to improve performance. category=PossibleUserWarning, Testing DataLoader 0: 48%|████▊ | 23/48 [00:31<00:34, 1.39s/it]Traceback (most recent call last):

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/spyder_kernels/py3compat.py", line 356, in compat_exec exec(code, globals, locals)

File "/nvme-data/change_former/Marine_debris_detection_sentinel_2/code/marinedebrisdetector-main/marinedebrisdetector/test.py", line 236, in main(parse_args())

File "/nvme-data/change_former/Marine_debris_detection_sentinel_2/code/marinedebrisdetector-main/marinedebrisdetector/test.py", line 92, in main trainer.test(model, marinedebris_datamodule)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 795, in test self, self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in _test_impl results = self._run(model, ckpt_path=self.ckpt_path)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run results = self._run_stage()

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1188, in _run_stage return self._run_evaluate()

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1228, in _run_evaluate eval_loop_results = self._evaluation_loop.run()

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/loops/loop.py", line 199, in run self.advance(*args, **kwargs)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/loops/loop.py", line 199, in run self.advance(*args, **kwargs)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 121, in advance batch = next(data_fetcher)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in next return self.fetching_function()

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/utilities/fetching.py", line 265, in fetching_function self._fetch_next_batch(self.dataloader_iter)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/utilities/fetching.py", line 280, in _fetch_next_batch batch = next(iterator)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next data = self._next_data()

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch return self.collate_fn(data)

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 84, in default_collate return [default_collate(samples) for samples in transposed]

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 84, in return [default_collate(samples) for samples in transposed]

File "/home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate return torch.stack(batch, 0, out=out)

RuntimeError: stack expects each tensor to be equal size, but got [12, 128, 128] at entry 0 and [12, 128, 127] at entry 32`

[MarcCoru]

Hi,

Thanks for the Issue!

Hmm. that is weird. I usually added padding to each dataset to make sure the dimensions match to the getitem function calls.

See for instance here: https://github.com/MarcCoru/marinedebrisdetector/blob/44f7f5898f37b4b1bfb3378a0ef5e587a3e3ead2/marinedebrisdetector/data/floatingobjects.py#L212

this comes at the cost of having a zero-value pixel border in some images, which is fine to some degree.

Unfortunately, I don't have a lot of time for coding and debugging nowadays. Can you identify which dataset class that mismatched image came from? that would help finding a missing padding or so.

I am available to help generally!

Marc

[mustansarfiaz]

Hi, thank you for your response.

I debugged the test code and found that the window generates an image of size (12,128,127) at item index 31. It should be (12,128,128). here is error line: https://github.com/MarcCoru/marinedebrisdetector/blob/44f7f5898f37b4b1bfb3378a0ef5e587a3e3ead2/marinedebrisdetector/data/refined_floatingobjects.py#L61

Here is the dataset path and outputs of size mismatch for /data/marinedebris/refinedfloatingobjects/durban_20190424.tif

Please look into it.

[MarcCoru]

thanks for running the tests.

interesting... It looks like the window rounds down width 127.999 to 127 instead of 128. I suppose it is some floating point issue that are handled differently on different computers, as I didn't have that issue.

I think the easiest fix would be to add a padding line like image, mask = pad(image, self.imagesize//2) using the pad function in utils.py

after the following line

https://github.com/MarcCoru/marinedebrisdetector/blob/44f7f5898f37b4b1bfb3378a0ef5e587a3e3ead2/marinedebrisdetector/data/refined_floatingobjects.py#L63

that will add a zero a zero row at one side, which is not optimal, but a quick fix to reproduce results.

Otherwise, there may be some further check to modify the rasterio window object with the correct width (128 instead of 127). But I don't know that out of the top of my head. That may need some tinkering with the object.

Hope that helps

Marc

[mustansarfiaz]

Hi Marc

Thank you for your help. I followed as you suggested but I'm not getting the numbers as in paper using the pre-trained weights. Moreover, I want to know if I am doing something wrong.

Here are my hyper parameters: comparison=ours, ckpt-folder=epoch=97-val_loss=0.63-auroc=0.982.ckpt, batch-size=64, workers=0 [should not effect anything], image-size= 128, topk=1

Here is the log for inference

Global seed set to 0 using checkpoint epoch=97-val_loss=0.63-auroc=0.982.ckpt Lightning automatically upgraded your loaded checkpoint from v1.8.2 to v1.9.5. To apply the upgrade to your files permanently, run python -m pytorch_lightning.utilities.upgrade_checkpoint --file checkpoints/marinedebrisdetector/author/a_unet2/epoch=97-val_loss=0.63-auroc=0.982.ckpt GPU available: True (cuda), used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs HPU available: False, using: 0 HPUs

Dataset Composition total

train flobs_dataset (train): 39194 shipsdataset: 29833 MARIDA (train): 2017

val refinedflobs_val: 2451 MARIDA (val): 1185

test flobstestdataset: 2197 maridatestdataset: 872

Dataset Composition debris/non-debris train flobs_dataset (train): 19587/19607 shipsdataset: 0/29833 MARIDA (train): 930/1087

val refinedflobs_val: 868/1583 MARIDA (val): 616/569

test flobstestdataset: 903/1294 maridatestdataset: 270/602 LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1] /home/mbzuaiser/anaconda3/envs/changeformer_s/lib/python3.7/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:229: PossibleUserWarning: The dataloader, test_dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument(try 128 which is the number of cpus on this machine) in theDataLoader` init to improve performance. category=PossibleUserWarning, Testing DataLoader 0: 100%|██████████| 48/48 [01:06<00:00, 1.38s/it] ──────────────────────────────────────────────────────────────────────────────── Test metric DataLoader 0 ──────────────────────────────────────────────────────────────────────────────── test_accra_20181031_accuracy 0.9250836120401338 test_accra_20181031_auroc 0.9689815643458027 test_accra_20181031_fscore 0.9223300970873787 test_accra_20181031_jaccard 0.8558558558558559 test_accra_20181031_kappa 0.8500756605750204 test_accra_20181031_precision 0.9472934472934473 test_accra_20181031_recall 0.8986486486486487 test_accra_20181031_threshold 0.07124878466129303 test_accuracy 0.9045291625936788 test_auroc 0.890273101535606 test_durban_20190424_accuracy 0.9045584045584045 test_durban_20190424_auroc 0.8274810202943419 test_durban_20190424_fscore 0.7432950191570882 test_durban_20190424_jaccard 0.5914634146341463 test_durban_20190424_kappa 0.6890807409072279 test_durban_20190424_precision 0.9897959183673469 test_durban_20190424_recall 0.5950920245398773 test_durban_20190424_threshold 0.07124878466129303 test_fscore 0.864539990753583 test_jaccard 0.761400651465798 test_kappa 0.7916408597710641 test_loss 1.1586157083511353 test_marida_accuracy 0.8692660550458715 test_marida_auroc 0.7314599483204134 test_marida_fscore 0.7521739130434784 test_marida_jaccard 0.6027874564459931 test_marida_kappa 0.666997186118183 test_marida_precision 0.9105263157894737 test_marida_recall 0.6407407407407407 test_marida_threshold 0.07124878466129303 test_precision 0.9444444444444444 test_recall 0.7971014492753623 test_threshold 0.07124878466129303 ────────────────────────────────────────────────────────────────────────────────

In addition to that,

I also ran three times to get mean and std by reproducing the numbers myself. I also used your pretrained weights and get the numbers. However, the numbers are different from paper. Could you please look into it. Here are the numbers:

Regards Mustansar

[mustansarfiaz]

Hi

Here the numbers produced using weights provided by you. please look into it. and please guide me if Im doing something wrong

Regards Mustansar

[MarcCoru]

Alright. I see the refined floating object numbers are slightly worse. The MARIDA numbers are on-spot. So, the model seems still fine, is must be connected with the refined Floating Objects dataset.

My assumption is that the "quick and dirty" padding fix above made the numbers on the refined-floating objects datasets worse, which is expected. The label of the samples in the refined floating objects dataset correspond exactly to the center pixel. I assume that the padding has shifted the center pixel in some cases. That would explain the slight drop in accuracy.

How did you address the site offset at the end? By padding? It may be better to fix it properly by adjusting the rasterio window size before loading the image (and padding afterwards)

Best of success, Marc

[mustansarfiaz]

Hi Marc

Thank you for such detailed help. I also believe that this might be due to rasterio window size. Now, I just want to confirm that I followed the correct inference process is right and I correctly read the values for accuracy, f-score, auroc, Jaccard, and kappa.

Could you provide your environment.yml file? I can try that and maybe if I follow the exact library versions you were using, I might not require this "quick and dirty" padding. BTW, I followed using padding as you suggested. Here is the padding function

''' def pad(image, output_size=64):

if feature is near the image border, image wont be the desired output size

H, W = output_size, output_size
c, h, w = image.shape
dh = (H - h) / 2
dw = (W - w) / 2
image = np.pad(image, [(0, 0), (int(np.ceil(dh)), int(np.floor(dh))),
                       (int(np.ceil(dw)), int(np.floor(dw)))])
return image
'''

Regards Mustansar