incomplete log error - Githubissues

shubh0155 commented 11 months ago

I've generated Darshan logs for Dlio benchmarking on my Luster file system. When I parse these logs using darshan-summary.pl, it generates a PDF, but the output appears to be incomplete. When I try to parse the logs using darshan-parser with the --show-incomplete flag, it gives the full report. I've also attempted to exclude all other directories by passing DARSHAN_EXCLUDE_DIRS=*, but I'm still facing the same issue. Can anyone provide assistance with this? Thank you. Screenshot (58) Screenshot (57)

kevin-harms commented 11 months ago

How many files are you trying to track per-rank? You might have rank0 or something that is using lots of files. If you use the --show-incomplete, do you see one rank (or many ranks) with lots of files?

shanedsnyder commented 11 months ago

See this section in darshan-runtime docs about configuring the runtime library: https://www.mcs.anl.gov/research/projects/darshan/docs/darshan-runtime.html#_configuring_darshan_library_at_runtime

Specifically, you have a couple of options to ensure you're not missing something instrumentation:

simply asking for Darshan to allocate more records for modules that are running out of memory using the MAX_RECORDS setting
filtering out records that aren't of interest using NAME_EXCLUDE setting

There should be some example config files shown there that illustrate how to set those parameters. Maybe start there and let us know if you have any questions?

darshan-hpc / darshan

incomplete log error #962