In training, same sample is being stored per batch.

[ntadimeti]

This bug was reported in #195 using v0.2.3 version. I am able to reproduce it on the same tag. Yet to reproduce on 0.3.0. To repro,

vi claragenomics/dl4atac/train.py

after line 61, add:

        for idx in range(0, len(x)):
            print (x[idx].sum())
            print (torch.all(torch.eq(x[idx], x[idx+1])))

Comment out rest of the code. This will show that for a batch size of 64, every sample in the batch is identical. Output:

INFO:2020-08-20 18:00:49,768:AtacWorks-main] Running on GPU: 0
Building model: resnet ...
Finished building.
Saving config file to ./trained_models_2020.08.20_18.00/configs/model_structure.yaml...
Num_batches 500; rank 0, gpu 0
64 64 64
tensor(402.)
tensor(True)
tensor(402.)
tensor(True)
tensor(402.)
tensor(True)
tensor(402.)
tensor(True)
tensor(402.)
tensor(True)