Loader shuffled by different seed

[namikimeiko]

Thank you for your prompt contribution.

Maybe this is mistake, SimSiam_Pre_training.ipynb implement ...

Author's,

# f: backbone + projection mlp
# h: prediction mlp
for x in loader: # load a minibatch x with n samples
    x1, x2 = aug(x), aug(x) # random augmentation
    z1, z2 = f(x1), f(x2) # projections, n-by-d
    p1, p2 = h(z1), h(z2) # predictions, n-by-d
    L = D(p1, z2)/2 + D(p2, z1)/2 # loss
    L.backward() # back-propagate
    update(f, h) # SGD update

Your implementation means,

for (x_a, x_b) in (loader_a, loader_b): # load a minibatch x with n samples
    x1, x2 = aug(x_a), aug(x_b) # random augmentation
    z1, z2 = f(x1), f(x2) # projections, n-by-d
    p1, p2 = h(z1), h(z2) # predictions, n-by-d
    L = D(p1, z2)/2 + D(p2, z1)/2 # loss
    L.backward() # back-propagate
    update(f, h) # SGD update

I think the easiest fix is to use shuffle seed...

dataset_one = (
    train_ds
    .shuffle(1024, seed=0)
    .map(custom_augment, num_parallel_calls=AUTO)
    .batch(BATCH_SIZE)
    .prefetch(AUTO)
)

dataset_two = (
    train_ds
    .shuffle(1024, seed=0)
    .map(custom_augment, num_parallel_calls=AUTO)
    .batch(BATCH_SIZE)
    .prefetch(AUTO)
)

Regards.

[sayakpaul]

Correct. I will get it fixed asap. Thank you!