darshan-hpc / darshan

Darshan I/O characterization tool
Other
56 stars 27 forks source link

API to support injection of characterization data from external tools #247

Open shanedsnyder opened 3 years ago

shanedsnyder commented 3 years ago

In GitLab by @carns on Oct 2, 2018, 11:18

The use case for this would be instrumentation from an I/O library that wants to inject it's data into Darshan so that it travels with the Darshan log and uses it's compression and output infrastructure.

This would most likely mean adding an externally visible wrapper to the module register and record id functions, plus an extra function to retrieve a pointer to the memory region allocated during registration.

shanedsnyder commented 3 years ago

In GitLab by @glennklockwood on Oct 2, 2018, 11:47

This would be a great feature to have. There are a few times where I would have also liked the ability to insert new records/modules (or even arbitrary data, like HDF5's user block) into Darshan logs post-hoc. While this would probably be dangerous to do in production, it would be a more portable way for downstream analysis to attach indices or derived quantities (e.g., darshan-parser --perf counters) to existing logs so they don't have to be recalculated.

I'm not sure if use cases are compatible with the scope of this feature request, but hopefully they are.

shanedsnyder commented 3 years ago

In GitLab by @carns on Oct 2, 2018, 12:19

@glennklockwood Would you mind opening a separate ticket for the post-hoc injection idea? I can see how that would be useful, but it's probably a separate development effort from this because it will need to modify existing logs. I think it's definitely doable though.

shanedsnyder commented 3 years ago

In GitLab by @carns on Oct 4, 2018, 09:31

mentioned in merge request !24

shanedsnyder commented 3 years ago

In GitLab by @carns on Apr 25, 2020, 07:17

unassigned @carns