Open MarcCote opened 1 year ago
Hmm, given that EnvObject uses a Set() instead of an Array() for a storage class for objects (and most of the downstream functions in the e.g. object parser, etc., assume Sets() instead of ordered arrays), it would likely be a significant refactor to switch everything over to Arrays. I think the previous patch for determinism was limited to just showing the objects in the same order in the description text, which was solved with a call to sort the objects by name or object ID in the object description function.
@MarcCote, as @PeterAJansen stated, removing all non-determinism would be a large refactor, so I didn't attempt to fix most of it. I would have tried to fix a non-deterministic action if I were aware of it, but the only remaining non-determinism that I knew about was in gold path generation. For that, I only fixed a single task, and the fix interferes with our generation of training data, so I never tried to upstream my patch.
Some times running the same sequence of commands in ScienceWorld leads to different outcome (here making steam or not).
Steps to reproduce:
taskName=task-1-boil
,variationIdx=0
, andsimplificationStr='teleportAction'
.teleport to kitchen
,pour counter into sink
,activate sink
,use lighter on drawer
, andlook around
"a sink, which is turned on. In the sink is: a bowl (containing a red apple, a banana, an orange, a potato, a substance called water), a burned drawer, a substance called steam."
Running that procedure 8 times in a given Python instance, I got
Doing it a second time, I got
0/8 runs was expected.
@aphedges I'd love to see your changes to fix non determinism in the Scala code.