Incorrect and missed splits for some datasets

[teasgen]

I'm trying to use OxfordIIITPet dataset, and as indicated in Pets it has splits "trainval", "test". In your code are available only "train" and "test" splits. When I run the code it fails with error: ValueError: Unknown value 'train' for argument split. Valid values are {'trainval', 'test'}. Please, fix it.

The second question is how to use the validation splits with your code? For instance, DTD has "train", "val", "test" parts, and I think it will be great, if you add possibility to use it. I need it to perform linear evaluation with train and validation parts.

[mehdidc]

Hi @teasgen thanks for reporting the errors, will go over the datasets and fix this

[mehdidc]

fixed, please retry

[teasgen]

fixed, please retry

Thank you, it works. When this package will be updated in https://pypi.org/project/clip-benchmark? I think the 1.4.0 version is outdated and package hasn't been updated for 6 month.

[mehdidc]

@teasgen indeed would make sense to update it now. @rom1504 could you please let me know how you did that previously, from what I understand it was through github actions? or if you have time please just do it. btw HISTORY.rst is already updated with the (1.5.0) changes.

[rom1504]

You need a commit named "Release 1.5.0" and you need to update history file and the version in setup.py

On Tue, Aug 22, 2023, 17:08 Mehdi Cherti @.***> wrote:

@teasgen https://github.com/teasgen indeed would make sense to update it now. @rom1504 https://github.com/rom1504 could you please let me know how you did that previously, from what I understand it was through github actions? or if you have time please just do it. btw HISTORY.rst is already updated with the (1.5.0) changes.

— Reply to this email directly, view it on GitHub https://github.com/LAION-AI/CLIP_benchmark/issues/104#issuecomment-1688378529, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAR437T6ILQLQOBR4CIXV63XWTDNJANCNFSM6AAAAAA3D7ANAA . You are receiving this because you were mentioned.Message ID: @.***>

[mehdidc]

Alright will do, thanks @rom1504

[mehdidc]

Ok 1.5.0 is available now, closing.