Sampling behavior for Imagenet not reproducible

[rpinsler]

Hi,

I'm trying to get reproducible sampling behavior by setting the random seed as follows:

def set_seed(seed):
    random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    np.random.seed(seed)
    tf.compat.v1.random.set_random_seed(seed)
    sampling.RNG = np.random.RandomState(seed)

This works for all datasets except ilsvrc_2012 with use_dag_hierarchy=True. I assume this is due to some non-deterministic operation during the construction of the DAG, but I haven't been able to find the root cause. Any ideas how to fix this?

Best, Robert

[lamblin]

I have noticed that generating the dataset_spec.json was not deterministic in the order of keys in dictionaries, but did not make a case of it, but it may be linked. I think children of a node in the DAG are represented as sets or dicts, and depending on the version of Python, they might or might not be iterated in the same order between executions.

The first thing would be to make sure you are using the same dataset_spec.json (if you are working with multiple machines), and then look for iteration over children of a node, replacing iterations over the un-ordered collection by sorting first (over the synset "n0...." string should be fine). Then if performance needs it, I can have a look at replacing dictionaries with OrderedDict or sorting only at strategic places.