Add header and cell index to filter output csv-files

jomichaelis commented 2 weeks ago

The filter pipeline allows for the generation of csv output files, as is shown in the following exemplary couette.xml:

...
  <filter-pipeline>
    <per-instance output="md"/>
    <post-multi-instance output="md">
      <prefilter filtered-values="macro-mass macro-momentum" input="md">
        <write-to-file location="nofilter.csv"/>
      </prefilter>
    </post-multi-instance>
  </filter-pipeline>
...

This results in a file nofilter.csv:

1 ; 12.2347 ; -0.262493 ; -0.176776 ; 0.14687 ; 
1 ; 12.2231 ; -0.164591 ; 0.213655 ; 0.1595 ; 
1 ; 12.3207 ; -0.24711 ; -0.0794144 ; -0.0531393 ; 
1 ; 12.1926 ; -0.260807 ; -0.0856201 ; 0.069399 ; 
1 ; 12.1882 ; 0.128584 ; -0.00301881 ; -0.00504167 ; 
1 ; 12.2229 ; 0.237522 ; -0.0587471 ; -0.234311 ; 
1 ; 12.2467 ; -0.125776 ; -0.131807 ; 0.110567 ; 
1 ; 12.3983 ; 0.203651 ; 0.239832 ; -0.548851 ; 
...

The first column denotes the coupling cycle, while the second and the three following columns represent the mass and the 3 dimensional momentum, as specified in the couette.xml file above.

The developer/user is however left without information about the interpretation of the csv lines. It might be very beneficial to include the corresponding cell index to each row to prevent misinterpretation in post processing steps. Also, users need to be aware of their own configuration of the output file, as there is no header written to the csv file.

Example

The csv-files generated by MaMiCo's couette-test coupling functionality could serve as example. When specifying a postive value in the couette.xml:

...
<coupling
      coupling-cycles="1001"
      two-way-coupling="no"
      send-from-md-to-macro="yes"
      filter-init-cycles="0"
      write-csv-every-timestep="0"
      send-from-macro-to-md="yes"
      compute-snr="no"
    />
...

, the simulation writes out csv-files in the following format:

I01_x,I01_y,I01_z,vel_x,vel_y,vel_z,mass
4 , 4 , 4,-0.185251,-0.0592073,0.150312,13.28
5 , 4 , 4,0.328107,-0.177471,-0.198811,11.88
6 , 4 , 4,0.0565378,0.0561282,0.0155711,14.84
7 , 4 , 4,0.14806,-0.151455,0.216326,11.72
8 , 4 , 4,-0.346705,-0.420957,0.292326,11.16
9 , 4 , 4,-0.0708458,-0.093582,0.0397505,15.6
...

Goal

Include cell index to filter output files and allow for an optional header specification, like so: (couette.xml)

...
<write-to-file location="nofilter.csv" write-header="yes"/>
...

generates nofilter.csv like so:

c_cycle ; idx_x ; idx_y ; idx_z ; mass ; mom_x ; mom_y ; mom_z
1 ; 4 ; 4 ; 4 ; 12.2347 ; -0.262493 ; -0.176776 ; 0.14687 ; 
1 ; 5 ; 4 ; 4 ; 12.2231 ; -0.164591 ; 0.213655 ; 0.1595 ; 
1 ; 6 ; 4 ; 4 ; 12.3207 ; -0.24711 ; -0.0794144 ; -0.0531393 ; 
1 ; 7 ; 4 ; 4 ; 12.1926 ; -0.260807 ; -0.0856201 ; 0.069399 ; 
1 ; 8 ; 4 ; 4 ; 12.1882 ; 0.128584 ; -0.00301881 ; -0.00504167 ; 
1 ; 9 ; 4 ; 4 ; 12.2229 ; 0.237522 ; -0.0587471 ; -0.234311 ; 
1 ; 4 ; 5 ; 4 ; 12.2467 ; -0.125776 ; -0.131807 ; 0.110567 ; 
1 ; 5 ; 5 ; 4 ; 12.3983 ; 0.203651 ; 0.239832 ; -0.548851 ; 
...

Additional Info

This is not urgent, just a constructive input without any time constraints. The change would simplify post processing a lot and also decreases susceptibility to errors due to misinterpretation.

Thinkpiet commented 2 weeks ago

I remember that we used to have that (both the indices and the header). Maybe you can check if you find it somewhere, eg some stale branch or some commented out code, and then open a pull request

jomichaelis commented 2 weeks ago

I have found the following commit, where such funcitonality was commented out: https://github.com/HSU-HPC/MaMiCo/commit/c75e694e4620abb1a78fde78f855aa7ef2f94c1c#diff-033d765e13d57639ef36eda541784436456a1087b3b4187737dc22091bbc211a

It seems like the cell indexing has changed since that time, which makes adding the functionality a non-trivial task for me as "MaMiCo-newbie". I'll see what I can do as soon as I find the time.

HSU-HPC / MaMiCo