[Question] Usability evaluation and custom dataset

[jS5t3r]

Hi,

as I understand correctly that main.py will be replaced by `eval_ood.py'. I created a custom dataset.

I changed this script

sh scripts/ood/msp/cifar10_test_ood_msp.sh

the content of the old script:

PYTHONPATH='.':$PYTHONPATH \
#srun -p dsta --mpi=pmi2 --gres=gpu:${GPU} -n1 \
#--cpus-per-task=${CPU} --ntasks-per-node=${GPU} \
#--kill-on-bad-exit=1 --job-name=${jobname} -w SG-IDC1-10-51-2-${node} \
python main.py \
    --config configs/datasets/cifar10/cifar10.yml \
    configs/datasets/cifar10/cifar10_custom_dataset_ood.yml \  <------------- Here!!!!!! I added the custom dataset. !!!!!!!!!
    configs/networks/resnet18_32x32.yml \
    configs/pipelines/test/test_ood.yml \
    configs/preprocessors/base_preprocessor.yml \
    configs/postprocessors/msp.yml \
    --num_workers 8 \
    --network.checkpoint './results/cifar10_resnet18_32x32_base_e100_lr0.1_default/s0/best.ckpt' \
    --mark 0 \
    --merge_option merge

but how do I define the custom dataset in `eval_ood.py'?

python scripts/eval_ood.py \
   --id-data cifar10 \
   --root ./results/cifar10_resnet18_32x32_base_e100_lr0.1_default \
   --postprocessor msp \
   --save-score --save-csv

Thanks for your help in advance!

[zjysteven]

Hi @jS5t3r, the new evaluator fetches data by https://github.com/Jingkang50/OpenOOD/blob/18c6f5174a2f518e2a8e819ffb1cd1914bcf12e0/openood/evaluation_api/evaluator.py#L103-L110 where the get_id_ood_dataloader is defined in openood/evaluation_api/datasets.py. You would need to link your dataset in openood/evaluation_api/datasets.py so the evaluator can obtain your custom data.

[jS5t3r]

@zjysteven Ok. This means I need to change the code instead of having a provided API

[zjysteven]

The provided API is dedicated for a unified and consistent evaluation on carefully picked OOD datasets, which is why OpenOOD is created in the first place as it enables fair and straight comparison across methods. Including your own data is not hard: as I mentioned earlier, it should be good to go if you just encode relevant information in openood/evaluation_api/datasets.py (following how other datasets are encoded).

For "no ood.csv created" issue you mentioned in the pull request, did you pass the save_csv argument when running eval_ood_imagenet.py? https://github.com/Jingkang50/OpenOOD/blob/18c6f5174a2f518e2a8e819ffb1cd1914bcf12e0/scripts/eval_ood_imagenet.py#L172-L180

[zjysteven]

Closing now. Feel free to reopen if there are any other questions.