search

ISISDataPolicy / policy

ISIS Neutron and Muon Source Data Policy
1 stars 1 forks source link

Should the term 'intermediate data' be defined? #22

Closed Anders-Markvardsen closed 6 months ago

Anders-Markvardsen commented 1 year ago

In an ISIS DAaaS Data Summary Nov 2022 it is stated "The analysis data occupies most of the disk space that DAaaS stores on the Deneb Ceph cluster for ISIS. It takes up 564TB of disk space. 4 instruments (LET, MERLIN, ALF, MAPS) are responsible for 555TB or 99% of the total." and "From closer inspection, a lot of this data consists of ‘intermediate’ data. Specifically, NSXPE files"

The above demonstrates an operational issue.

Should the ISIS data policy include the term 'intermediate data' (or other name) as data that instrument scientists classified as transient data that can be deleted after (say) 6 months?

Operationally there could be some form of cronjob that automatically deletes files with specific extensions for specific instruments and where this is made transparent to the user.

agbeltran commented 1 year ago

The policy reads: "4.2.2 Result data created by users of the DAaaS platform will be stored on medium-term storage, subject to volume restrictions."

martyngigg commented 6 months ago

As we already have the clause mentioned and its operational we don't feel the policy needs anything further in relation to this.