error during TEST/INFERENCE using percentile normalization in detection workflow [bug]

[pr4deepr]

Hi @danifranco I've trained a model using percentile normalization with the 3D detection workflow. I'm running into errors during the "TEST" part of the workflow.

I keep getting this error:

[10:49:22.790595] Loading checkpoint from file D:/traning/dropout_percentile/unet_default_lr/checkpoints/unet_default_lr_0-checkpoint-best.pth
[10:49:22.951760] Model weights loaded!
[10:49:22.951760] ############################
[10:49:22.951760] #  PREPARE TEST GENERATOR  #
[10:49:22.951760] ############################
[10:49:22.951760] Test normalization: trying to load lower bound percentile value from D:/traning/dropout_percentile/test/unet_default_lr/checkpoints/lower_bound_perc.npy
[10:49:22.951760] Test normalization: trying to load lower bound percentile value from D:/traning/dropout_percentile/test/unet_default_lr/checkpoints/upper_bound_perc.npy
Traceback (most recent call last):
  File "C:/Users/rajasekhar.p/Documents/Pradeep_BAC_Projects/Dewson_Lab/whole_brain_TH/biapy/biapy_run_inference.py", line 27, in <module>
    biapy.run_job()
  File "C:/Users/rajasekhar.p/Documents/BiaPy/biapy/_biapy.py", line 419, in run_job
    self.test()
  File "C:/Users/rajasekhar.p/Documents/BiaPy/biapy/_biapy.py", line 159, in test
    self.workflow.test()
  File "C:/Users/Public/miniconda3/envs/biapy_env/lib/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:/Users/rajasekhar.p/Documents/BiaPy/biapy/engine/base_workflow.py", line 835, in test
    self.prepare_test_generators()
  File "C:/Users/rajasekhar.p/Documents/BiaPy/biapy/engine/base_workflow.py", line 781, in prepare_test_generators
    self.test_generator, self.data_norm = create_test_augmentor(self.cfg, self.X_test, self.Y_test, self.cross_val_samples_ids)
  File "C:/Users/rajasekhar.p/Documents/BiaPy/biapy/data/generators/__init__.py", line 386, in create_test_augmentor
    raise FileNotFoundError("Not lower/upper bound percentile files found in {} and {}"
FileNotFoundError: Not lower/upper bound percentile files found in D:/traning/dropout_percentile/test/unet_default_lr/checkpoints/normalization_mean_value.npy and D:/traning/dropout_percentile/test/unet_default_lr/checkpoints/normalization_std_value.npy

I have the lower_bound_perc.npy and upper_bound_perc.npy.. I checked the lower and upper bound files and they are:

array([1.00e-01, 1.99e+02])
array([   99.9, 20337. ])

Looks like something to do with the logic in this code block:

https://github.com/BiaPyX/BiaPy/blob/2c3d6cc6aca04ed738e9a201a14b636b5b13d836/biapy/data/generators/__init__.py#L363

Cheers Pradeep

[pr4deepr]

looking at the code: https://github.com/BiaPyX/BiaPy/blob/2c3d6cc6aca04ed738e9a201a14b636b5b13d836/biapy/data/generators/__init__.py#L385

should this

if calc_percentiles: be

if not calc_percentiles: ???

[pr4deepr]

Ok, I didn't realize I had to explicitly provide the lower and upper percentile file paths in the config file when running TEST.

So, now I've specified the paths under the key:

• PATHS.LWR_VAL_FILE
• PATHS.UPR_VAL_FILE

Once I do this, it works!!

I assumed they would have been automatically detected as I provided the PATHS.CHECKPOINT_FILE.

Is this worth adding in the documentation? or did I miss this part?

Cheers Pradeep

[pr4deepr]

Also, to add the UPR_VAL_FILE and LWR_VAL_FILE were both in same directory as PATHS.CHECKPOINT_FILE..

[danifranco]

Hi,

Sorry for the delay... I'm pushing new changes (60e06cc): I separated the percentile clipping for the normalization, so you can apply it using 'custom' normalization. The idea of having these values stored is to load during test to do the same percentile clipping as the one used during train/val. This will be done when DATA.NORMALIZATION.APPLICATION_MODE == "dataset", otherwise a per image clipping will be made. I'm changing the documentation to reflect these changes and also information regarding the values stored under checkpoints folder so the users can now where they are.

Thank you again!

Feel free to close the issue if you want. Tell me when you also checked that it works as you expected so we can close also #73 ;)

[danifranco]

Hi @pr4deepr, did you have time to check this? Thanks a lot!

[pr4deepr]

Hi Sorry, I haven't. Will get onto this soon. Pradeep

[danifranco]

Hello,

Just to tell you that we make a new release of BiaPy solving minor bugs and some related to percentile clippling. Now it should be running in all the cases. It would be nice if you can try and reproduce the results you got by doing that clipping outside BiaPy. Our doc to use the clipping is also updated here.

[pr4deepr]

Hi @danifranco I gave it a go and I can run it using the CLI and the percentile clipping seems to be working for both training and inference. Thanks for updating the doc. I like the option of clipping and then having different normalization options.

Cheers Pradeep

[danifranco]

Thank you so much for trying it!! I'm closing the issue.