Adding Fashion-MNIST and German Traffic Sign datasets

[max-andr]

Which datasets are considered? We include the datasets defined in the robustml framework (currently MNIST, CIFAR-10, and ImageNet ILSVRC 2012). If you wish to add a new dataset, please open an issue.

Would it be possible to also add Fashion-MNIST and German Traffic Sign datasets to the robustml framework?

Both datasets can be found as .mat-files at: https://github.com/max-andr/MIPVerify_data/tree/master/datasets

Thanks!

[anishathalye]

Yes, it would be great to add new datasets. I can work on writing the code for this soon (or if anyone else wants to work on it, it involves adding a couple lines of code to dataset.py and provider.py).

[max-andr]

I could make a pull request with this change.

Would it be ok if in my implementation I just rely on reading .mat-files like the ones in https://github.com/max-andr/MIPVerify_data/tree/master/datasets/gts ?

[anishathalye]

.mat files versus another format doesn't really matter, as long as it's downloaded from the official source (http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset#Imageformat ?)

[max-andr]

Sorry for the delay. I decided not to rely on any pre-computed .mat files that I tend to use internally, and implemented data reading directly from the original files: https://github.com/robust-ml/robustml/pull/1

[anishathalye]

Thank you for the pull request! We'll review it soon, merge it, update the website to reflect that we support FMNIST and GTS, and then merge in updates for #4.

[anishathalye]

Thank you again for your patience. We've merged your change to robustml, and we've updated the faq page in bf524c7ab5bbfedc4585aec42ea06a753491b532.