[Feature] Initial multi-tests support in the Tester

[vmarois]

In this PR, I am adding support for multi-tests in Tester. It is basic, and somewhat constrained, but it worked for SimpleConvNet/MNIST.

If the user would like to run multiple tests, he can use the multi_tests key in the testing section to indicate the keys which associated values will be different for each test config:

problem: batch_size: 64 data_folder: ~/data/mnist name: MNIST resize: [32, 32] use_train_data: false sampler: {} multi_tests: {batch_size: [500, 1000]}

The following constraints apply for now:

• Assume that the indicated varying values are leafs of the testing section
• The number of indicated varying values per key is the same for all keys
• The indicated order of the varying values will be respected, i.e. \

multi_tests: {batch_size: [64, 128], img_size: [128, 256]}

and

multi_tests: {batch_size: [64, 128], img_size: [256, 128]}

will lead to different test configs.

• At least one key has varying values (but this is implicit).

I am mainly opening this PR for code review & feedback, as this is still very basic for now. I want to test how this plays with more complex configs (e.g. when sampler is present).

[vmarois]

@tkornut I've pushed some commits that:

• Remove the multi_test key from the config so that the testing_configuration.yaml in each test folder does not contain it,
• Properly update the logger.FileHandler of the Tester so that tester.log in each test folder only contains the logs of the current experiment (wasn't the case previously),
• Pretty print some info about the current test experiment:

@aasseman : The bug where a config param is also present in the default ones is still there.. So only doing params.del_config_param(key='config_key') does not work on its own, we have to add params.del_default_param(key='config_key'). That's how I am getting around it, so let's not spend time on this bug for now (as this is the only place in mip where we need this feature). I'm opening an issue for reference though.

[vmarois]

Update: I'm fairly confident that I have fixed the bug where adding config params would also add them to the default ones. See https://github.com/IBM/mi-prometheus/commit/2ffcc710c7549d113f85de1d007d5668c43c9339 @tkornut Can you review that PR?

[vmarois]

Update: I've loosen the constraint on the 'testing problem' section: The user can now change leafs of the entire testing configuration section.