Closed sjkoo1989 closed 4 years ago
Thanks for filing. This is Will Not Fix, unfortunately.
The --shuffle
flag was always misleading (it caused shuffling in a few obscure places, but not the places you would expect). That's why it was removed. If you're copying it from a command, drop it and please report where you got the command from.
Randomization is always off in validation/test to enforce determinism of test results. If you want determinism in training, you can use -dt train:stream
.
ParlAI's method is very confusing, but I don't see a way of changing it without potentially breaking a lot of historic code. We face a very difficult balance between maintaining the code of older works, and bettering the platform for the future.
(The thing I am changing is updating the docs to remove references to --shuffle
)
Thanks for your kind response!
Bug description In training a poly-encoder with the given tutorial document, we found that --shuffle true tag doesn't work (Also is not properly processed by argument parser). We also found that a class variable 'random' is only dependent on whether the mode is "train" or not.
Expected behavior if --shuffle true tag is given, random variables should be set True, otherwise set False.
Logs Please paste the command line output:
Additional context We cloned and tested a master branch.