When running repeated evaluations of the same policy class, I noticed that we will occasionally infinite loop here because the policy directory already exists, but we aren't updating the policy directory path in the loop so the loop condition never changes and we never exit.
Fix: when we generate a new experiment id in the loop, also update the policy dir so that the loop condition will change.
One question I have is: why in successive calls to the policy are we generating the same initial experiment id. Seems like the odds of this happening are very low, so I suspect there may be some issue with seeding the randomness of the experiment id when running as a webservice.
When running repeated evaluations of the same policy class, I noticed that we will occasionally infinite loop here because the policy directory already exists, but we aren't updating the policy directory path in the loop so the loop condition never changes and we never exit.
Fix: when we generate a new experiment id in the loop, also update the policy dir so that the loop condition will change.
One question I have is: why in successive calls to the policy are we generating the same initial experiment id. Seems like the odds of this happening are very low, so I suspect there may be some issue with seeding the randomness of the experiment id when running as a webservice.