PeterAJansen
commented
2 years ago Hi Alex,
Thank you for this contribution!
I have worked a bunch to iron out the non-determinism in the environment (the simulator was originally designed to have a lot of random variability in it, so there has been a bit of work in making it deterministic. I think we have ironed out 99%+ of the issues).
For the two main sources of non-determinism in this commit (the order of the output text, e.g. "you see an apple and an orange" vs "you see an orange and an apple"), and the valid action ordering (e.g. ["eat apple", "eat orange"] vs ["eat orange", "eat apple"]), I intentionally kept these in to help encourage model robustness. But that was a decision I made entirely on my own and I'm very happy to be convinced it's better to be deterministic. I just worry about unintended consequences -- for example, I discovered an issue with another popular text environment with consistent deterministic output ordering/sorting, where the environment could be trivially solved by taking (something like) the 3rd action from the start then the 15th action from the end each time, and I'm hoping to avoid those issues as much as possible.
If the determinism in the output and valid action ordering are something we want in ScienceWorld, instead of sorting the lists, the technique I've been sometimes using is to shuffle the list using a random generator that's initialized on a seed that is the variation number for that environment. That way things are still repeatable, but they're not shuffled in alphabetical order, which might cause other opportunities for models to hijack regularities in the input. That might be an option here, too. The issue is that it's just slightly more complex, since the task variation/random generator/etc. isn't exposed all the way down to the object level, where a lot of this generation happens.
aphedges
MarcCote
While using ScienceWorld, I discovered multiple places where the program output was non-deterministic, even though the seed was set. This made my testing more difficult because each run could give a different output, even when given the same input, producing very large diffs. All problems I found were caused by the fact that Scala's and Python's set implementations have non-deterministic iteration orders. I did not do an exhaustive search, but I fixed all the cases I found during my testing. My solution was either to use a list if set access isn't needed or to sort the set contents before iteration. If this kind of contribution is welcome, I might make future PRs if other instances cause me problems.
While I was at it, I made various fixes and improvements to building the JAR file, such as fixing a bug in the
package.shand resolving somesbtwarnings. I updated the JAR itself in a separate commit in case you want to drop the commit and replace it with your own. I personally wouldn't trust a JAR file from a random person on the internet, so I completely understand if you don't, either.All code I used for testing is in test_files.zip. It includes a test script (
test_nondeterminism.py), console output before and after my fixes (output_before.txtandoutput_after.txt) and JSON output of actions to make non-determinism easier to spot (actions_sorted.jsonandactions_unsorted.json).