psana idx mode showing no events

[carbonscott]

2024-06-04 05:01:30.677 [WRN] {PSXtcInput::Index} Index.cpp:364 - File has no events. Skipping: /gpfs/alpine2/proj-shared/lrn044/foundation_models/lcls/ds/mfx/mfxx1001621/xtc/index/mfxx1001621-r0161-s03-c01.xtc.idx

Consult the Data System team about this issue.

• It is true that the event doesn't exist?

[carbonscott]

Here’s what might have happened: let’s say I have 10 events in a run and I requested 3 workers. So work 1, 2, 3 will process 3 events during the first pass, respectively, then worker 0 should just process the last event. However, I guess PyTorch’s dataloader will pad the event list with fake event numbers so the worker will process event 10, 11, 12. This may explain why the xtc reader complains about having no events (or sometimes negative event numbers) because event 11 and 12 don’t exist (an overshooting problem).

This behavior was documented https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler, where it says about the drop_last argument:

If False, the sampler will add extra indices to make the data evenly divisible across the replicas.

Need to check what event numbers are these.