search

momentoscope / hextof-processor

Code for preprocessing data from the HEXTOF instrument at FLASH, DESY in Hamburg (DE)
https://hextof-processor.readthedocs.io/en/latest/
GNU General Public License v3.0
7 stars 4 forks source link

Enhancement needed when storing dataframes loaded with id interval #61

Closed balerion closed 3 years ago

balerion commented 3 years ago

The metadata gets overwritten by the last interval when storing dataframes in append mode. In other words, the data gets appended, but the metadata is only aware of the last append.

steinnymir commented 3 years ago

this has been fixed in this commit in the hackathon branch.

now metadata is correctly retained when appending parquets. The json metadata file for the appended parquets contains a dictionary with an aggregate of the runInfo of all included runs, to represent correctly the number of electrons/macrobunches, run numbers etc. The metadata of each run is stored next to this as dictionaries with the same structure as before.

zain-sohail commented 3 years ago

Should we close this issue if this has been fixed?

steinnymir commented 3 years ago

yes, should be fixed!

If there is any problem with metadata handling, we should make a new issue