In the end of each iteration and sometimes even in the end of the irace, the configuration that appears on the best-so-far ignores new configurations when they didn't run for all instances, even if it has budget to run missing instances.
I ran again experiment 3 as requested on the mail group, but it wasn't exactly the same because Iwasn't using the seed inside the python script running on the target-runner. But I think it had the same problems, it still had 24 experiments on budget but it stopped without running some of the elite configurations for all instances.
output3b.zip
In the end of each iteration and sometimes even in the end of the irace, the configuration that appears on the best-so-far ignores new configurations when they didn't run for all instances, even if it has budget to run missing instances. I ran again experiment 3 as requested on the mail group, but it wasn't exactly the same because Iwasn't using the seed inside the python script running on the target-runner. But I think it had the same problems, it still had 24 experiments on budget but it stopped without running some of the elite configurations for all instances. output3b.zip