operations with hdf files

[zaharko]

Combine (add) and subtract hdf files in a separate tab for this functionality.

1. When combining the files the following situations are the most common: 1a. the metadata of the files are identical (they were measured to improve statistics) Then the addition of counts should happen and estimated standard deviation should be recalculated 1b. One file is a continuation of the other (scanning motor values overlap in several points, but otherwise not) Then the total scanning area should be plotted, as presently hdf viewer does for one file 1c. More complicated cases (step is not the same, etc) should not be treated, but warning or error message communicated.

There should be an option to save the combined/subtracted file in the user's computer

[ivan-usov]

This functionality sounds almost exactly like the .ccl/.dat scans merging, but for hdf5 files. As a first question, does it make sense to, instead of having a tab for adding/subtracting files -> downloading them, have a tab where multiple hdf5 files can be open at the same time, similar to ccl_integrate?

[zaharko]

Actually, yes I just do not know how it is from the point of view of memory. The files are not giant, so opening a number of files is not going to affect the performance?

On 24 Jan 2022, at 15:18, Ivan Usov @.**@.>> wrote:

This functionality sounds almost exactly like the .ccl/.dat scans merging, but for hdf5 files. As a first question, does it make sense to, instead of having a tab for adding/subtracting files -> downloading them, have a tab where multiple hdf5 files can be open at the same time, similar to ccl_integrate?

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/pyzebra/issues/48#issuecomment-1020147881, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMBMFVGIFVBDTUQFJASCCA3UXVNR5ANCNFSM5MPSNGTA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.Message ID: @.***>

[ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/paulscherrerinstitute/pyzebra/issues/48#issuecomment-1020147881", "url": "https://github.com/paulscherrerinstitute/pyzebra/issues/48#issuecomment-1020147881", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[ivan-usov]

I don't think that memory will be a problem. And in case of appending files, the resulting file will also be large, so there will be not much difference between both methods anyways.

Ok, so a solution based on hdf_viewer and extended with options to open multiple files, could be a good starting point. At some point we might need to review the rules for merging hdf scans, but for now I'll apply the same rules as for .ccl/.dat files

[zaharko]

Sounds good!

On 24 Jan 2022, at 17:47, Ivan Usov @.**@.>> wrote:

I don't think that memory will be a problem. And in case of appending files, the resulting file will also be large, so there will be not much difference between both methods anyways.

Ok, so a solution based on hdf_viewer and extended with options to open multiple files, could be a good starting point. At some point we might need to review the rules for merging hdf scans, but for now I'll apply the same rules as for .ccl/.dat files

— Reply to this email directly, view it on GitHubhttps://github.com/paulscherrerinstitute/pyzebra/issues/48#issuecomment-1020305322, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMBMFVD75D62PCL43VX42MLUXV7BTANCNFSM5MPSNGTA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you authored the thread.Message ID: @.***>

[ { @.": "http://schema.org", @.": "EmailMessage", "potentialAction": { @.": "ViewAction", "target": "https://github.com/paulscherrerinstitute/pyzebra/issues/48#issuecomment-1020305322", "url": "https://github.com/paulscherrerinstitute/pyzebra/issues/48#issuecomment-1020305322", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { @.": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[ivan-usov]

The test server now should use an optimized version of hdf scans merging (https://github.com/paulscherrerinstitute/pyzebra/commit/098314e30dfd5ed5281da62ffe6d24c83e1972b2), bringing the time from minutes down to under a second.