twosixlabs / armory

ARMORY Adversarial Robustness Evaluation Test Bed
MIT License
176 stars 67 forks source link

[ENHANCEMENT] Fail gracefully when num-eval-batches > ds.size #1893

Open jprokos26 opened 1 year ago

jprokos26 commented 1 year ago

When supplying --num-eval-batches from the CLI or from the config file, if it is larger than the size of the dataset armory will not fail until reaching the StopIteration error

jprokos26 commented 1 year ago

@christopherwoodall This never got merged into tfdsv4, https://github.com/twosixlabs/armory/blob/tfdsv4/armory/utils/config_loading.py#L81-L82 Simple fix by adding suggested change

        if num_batches > dataset.batches_per_epoch:
            # since num-eval-batches only applies at test time, we can assume there is only 1 epoch
            raise ValueError(
                f"{num_batches} eval batches were requested, but dataset has only {dataset.batches_per_epoch} batches of size {dataset.batch_size}"
            )