carbonscott / maxie

Masked Autoencoder for X-ray Image Encoding (MAXIE)
Other
1 stars 4 forks source link

Introduce round robin scheduling in the dataset class #9

Open carbonscott opened 1 month ago

carbonscott commented 1 month ago

This issue follows the PR discussion #7.

"Randomizes the order in which all events across all experiments are yielded by the IPCDistributedSegmentedDataset entry generator."

Proposing round robin scheduling. This approach involves processing a set number of examples for each (experiment, run) using a Python generator, then moving on to the next experiment. This cycle repeats until all generators are exhausted.

carbonscott commented 1 month ago

Getting an output below with entry_per_cycle = 1

[RANK 51] exp=mfxx49820, run=34, detector_name=epix10k2M, event=0.
[RANK 50] exp=mfxl1001521, run=1, detector_name=Rayonix, event=0.
[RANK 32] exp=mfxl1027522, run=59, detector_name=epix10k2M, event=0.
[RANK 31] exp=mfxly3820, run=413, detector_name=Rayonix, event=0.
[RANK 23] exp=mfxl1027522, run=105, detector_name=epix10k2M, event=0.
[RANK 47] exp=mfxx1001621, run=70, detector_name=Rayonix, event=0.
[RANK 30] exp=mfxx1001621, run=14, detector_name=Rayonix, event=0.
[RANK 48] exp=mfxly3820, run=78, detector_name=Rayonix, event=0.
[RANK 59] exp=mfxl1027522, run=239, detector_name=epix10k2M, event=0.
[RANK 18] exp=mfxx1001621, run=61, detector_name=Rayonix, event=0.
[RANK 34] exp=mfxl1001521, run=87, detector_name=Rayonix, event=0.
[RANK 49] exp=mfxl1001521, run=140, detector_name=Rayonix, event=0.
[RANK 35] exp=mfxly3820, run=216, detector_name=Rayonix, event=0.
[RANK 58] exp=mfxl1008021, run=86, detector_name=Rayonix, event=0.
[RANK 4] exp=mfxx49820, run=95, detector_name=epix10k2M, event=0.
[RANK 32] exp=mfxly3820, run=248, detector_name=Rayonix, event=0.
[RANK 57] exp=mfxly3820, run=338, detector_name=Rayonix, event=0.
[RANK 12] exp=mfxl1027522, run=252, detector_name=epix10k2M, event=0.
[RANK 59] exp=mfxl1008021, run=62, detector_name=Rayonix, event=0.
[RANK 21] exp=mfxl1027522, run=35, detector_name=epix10k2M, event=0.
[RANK 17] exp=mfxl1008021, run=138, detector_name=Rayonix, event=3.
[RANK 38] exp=mfxl1027522, run=171, detector_name=epix10k2M, event=0.