search

valeoai / bravo_challenge

BRAVO Challenge Toolkit and Evaluation Code
Other
17 stars 2 forks source link

Encoding submission error - Index Error #1

Closed mdsmith-cim closed 3 months ago

mdsmith-cim commented 3 months ago

In the process of trying to encode a submission, I run into an IndexError in the encode script, making it obviously impossible to upload any results. I've noticed this on the provided baseline code so I don't believe it's anything to do with my implementation.

Steps to reproduce:

  1. Download provided RbA baseline code and set it up
  2. Run said code to generate results: python evaluate_ood_bravo.py --score_func rba --dataset_mode selective --selected_datasets bravo_ACDC bravo_SMIYC bravo_outofcontext bravo_synflare bravo_synobjs bravo_synrain --model_mode selective --selected_models swin_b_1dl --models_folder ckpts/ --datasets_folder datasets --out_path test_bravocode_out (Resulting output structure: encoding_source_dir.txt)
  3. Run the encoding script: python -m bravo_toolkit.util.encode_submission baselines/RbA/test_bravocode_out/ encoded_output.tar --samples ../bravo_SAMPLING.tar
  4. Failure: Opening tar files... INFO: Listing input files... INFO: Reencoding submission... 97%|████████████████████████████████████████████████████████████████████████████▊ | 7675/7889 [03:45<00:06, 33.96it/s] Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/local/data/msmith/bravo_repo/bravo_toolkit/util/encode_submission.py", line 166, in main() File "/usr/local/data/msmith/bravo_repo/bravo_toolkit/util/encode_submission.py", line 162, in main raise e File "/usr/local/data/msmith/bravo_repo/bravo_toolkit/util/encode_submission.py", line 158, in main process_tar_files(args.input_path, args.output_tar_path, samples_tar_path=args.samples) File "/usr/local/data/msmith/bravo_repo/bravo_toolkit/util/encode_submission.py", line 130, in process_tar_files encoded_data = bravo_encode(pred_image, conf_image, confidence_indices=confidence_indices) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/data/msmith/bravo_repo/bravo_toolkit/codec/bravo_codec.py", line 126, in bravo_encode confidence_array = confidence_array[confidence_indices] 
    
IndexError: index 921609 is out of bounds for axis 0 with size 921600

Any suggestions/fixes would be appreciated.

tuanhungvu commented 3 months ago

Hello @mdsmith-cim , thanks for reporting the issue. We are actively investigating it.

dreavjr commented 3 months ago

Hi @mdsmith-cim — it's possible that our baselines' code is slightly outdated.

It used to be the case that the encoding script accepted subsampled prediction and confidence arrays, but for simplicity and reliability's sake the current script requires that the arrays have the exact same size as the input image.

We will update the baselines, but one possible workaround is that you upsample the arrays yourself (using, e.g., the nearest pixel on the prediction array, and bilinear interpolation on the confidence array).

mdsmith-cim commented 3 months ago

Thanks for pointing me in the right direction. I noted that in the baselines, the SMIYC set uses image resizing while the other datasets do not. Removing that solved the issue and I've confirmed I can now at least successfully upload a basic submission to the evaluation server.

tuanhungvu commented 3 months ago

Close #1 - updated codes to fix resolution mismatches