Missing data collection

[fakerst]

I'm using Darshan to collect data for the MLperf storage benchmark, but I cannot collect the data for reading the dataset. I simply wrote a function call to serially detect the operation of reading a dataset, and everything was normal. I'm wondering if the function call is too deep or if there is a conflict between dlio and darshan. The process of reading a dataset is to use tf. data TFRecordDataset() reading the TFRecord file, I cannot collect data for any IO operations on call (), such as reads in the image, but I can collect data outside of this. Perhaps it's because call () is called too frequently?

Note: I modified the source code of darshan during configure so that it can collect the directory where my dataset is located

[kevin-harms]

In the log file, does it detect the open? If it's reasonably sized, you might be able to post the generated log as well.

kevin

On Sat, Jun 8, 2024 at 11:45 AM fakerst @.***> wrote:

I'm using Darshan to collect data for the MLperf storage benchmark, but I cannot collect the data for reading the dataset. I simply wrote a function call to serially detect the operation of reading a dataset, and everything was normal. I'm wondering if the function call is too deep or if there is a conflict between dlio and darshan. The process of reading a dataset is to use tf. data TFRecordDataset() reading the TFRecord file, I cannot collect data for any IO operations on call (), such as reads in the image, but I can collect data outside of this. Perhaps it's because call () is called too frequently?

Note: I modified the source code of darshan during configure so that it can collect the directory where my dataset is located

image.png (view on web) https://github.com/darshan-hpc/darshan/assets/119720309/7c6fde39-fd34-444c-b1bc-9cf4c5cd29b2

— Reply to this email directly, view it on GitHub https://github.com/darshan-hpc/darshan/issues/992, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVK4J6MIX32YFS2JLRK723ZGMYK3AVCNFSM6AAAAABJADXTFSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGM2DCNZTGUZTGOI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[fakerst]

2024-06-08-22-11-21.txt In the log, it only collected IO information for one TFRecord, but the count is 0. The others are the output information of the benchmark.The original logs can be found here.https://github.com/fakerst/darshan-script/blob/main/2024-06-08-22-11-21.darshan

Thank you very much for your help！

[shanedsnyder]

Looking at your log output, I guess this is the record of interest for you?

POSIX   -1  16398176509309012936    POSIX_OPENS 0   /opt/program/python/storage-main/resnet50_data/train/img_03_of_20.tfrecord  /   ext4
POSIX   -1  16398176509309012936    POSIX_FILENOS   0   /opt/program/python/storage-main/resnet50_data/train/img_03_of_20.tfrecord  /   ext4
POSIX   -1  16398176509309012936    POSIX_DUPS  0   /opt/program/python/storage-main/resnet50_data/train/img_03_of_20.tfrecord  /   ext4
POSIX   -1  16398176509309012936    POSIX_READS 0   /opt/program/python/storage-main/resnet50_data/train/img_03_of_20.tfrecord  /   ext4
POSIX   -1  16398176509309012936    POSIX_WRITES    0   /opt/program/python/storage-main/resnet50_data/train/img_03_of_20.tfrecord  /   ext4
POSIX   -1  16398176509309012936    POSIX_SEEKS 0   /opt/program/python/storage-main/resnet50_data/train/img_03_of_20.tfrecord  /   ext4
POSIX   -1  16398176509309012936    POSIX_STATS 4   /opt/program/python/storage-main/resnet50_data/train/img_03_of_20.tfrecord  /   ext4
...

And the issue is that this particular file is read by your code, but apparently not included in the instrumentation in the log file (i.e., READS value of 0 above)? Clearly, Darshan is able to instrument some access to this file (4 stat operations), but nothing else.

Without diving into your code or DLIO specifics, which I'm not sure I fully understand from your description above, maybe we can start with some simple things:

1. Are you sure there are not other Darshan logs generated by the application that could contain the information you're looking for? Darshan generates logs for each process, so if the application is using multiprocessing or otherwise spawning subprocesses, you should expect multiple logs.
2. If you're sure you're not overlooking any logs, then would it be possible for you to try our latest release (3.4.5) instead? There was some improvements in that release for handling Python multiprocessing that could possibly be related to what we're seeing here.

I'm mostly just speculating that some usage of multiprocessing (which I think is used a lot by TF) is confusing Darshan, but could be wrong. If none of the suggestions above are useful, then maybe you could also share strace -f output (the -f flag tells it to print info for child processes, too) for running this code?

[fakerst]

I'm sure there are not other Darshan logs generated and I have tried the latest release (3.4.5). There are only 4 stats in strace -f output. This means that there is indeed no posix_open，and it has openat open.py and mpiw.py，which is not in darshan logs. I feel so confused，thank you again for your help.The strace -f output can be found here.https://github.com/fakerst/darshan-hpc-darshan-issues-992/blob/main/storage_strace.txt

[shanedsnyder]

Hmm, yeah, I don't see any evidence of the .tfrecord file being read in the strace output either. As you point out, the only mentions of that file in the strace output are the stat calls that Darshan does instrument. Are you sure that particular file is actually being read in this benchmark configuration? Not really sure what to suggest, but may be useful to analyze the Darshan output for other benchmark configurations to see if it is consistently missing expected data, or if it's just this particular example?

By default, Darshan doesn't trace data in the /opt directory, so that would explain the lack of Darshan data for open.py and mpiw.py.