search

ISISComputingGroup / IBEX

Top level repository for IBEX stories
4 stars 2 forks source link

Data Archiving #36

Closed kjwoodsISIS closed 7 years ago

kjwoodsISIS commented 8 years ago

As a scientist, I want to migrate logged control system data to an archive, so that I have a secure record of logged control data that I can analyse at a later date.

'''Completion Criteria'''

  1. User can switch archiving on/off (default is on).
  2. All control system log files are archived while archiving is switched on.
  3. No control system log files are archived while archiving is switched off.

'''Notes'''

  1. Should the user have a choice about where the log file is archived?
  2. When does archiving happen - periodically, continuously, at defined events (e.g. end of experiment))?
  3. Is there a maximum archive frequency?
  4. What happens if the archive is full?
kjwoodsISIS commented 8 years ago

Imported from trac issue 36. Created by skn09965 on 2013-01-22T11:18:47, last modified: 2013-01-22T11:18:47

kjwoodsISIS commented 7 years ago

No longer relevant - superseded by other tickets.

ChrisM-S commented 7 years ago

Hi Kevin,

Is this really done? I know there is some ambiguity of naming in that the ISIS data archive is not the same as the EPICS archiver. But I’m not sure which archive this is referring to.

What for example happens if the archive fills up – is there a ticket which handles/answers this? Is the control data the log files which are exported to the ISIS Data Archive at the end of run – e.g. IMAT00001234.log ?

I’m not deliberately being pernickety but I’m not sure where the edge of the project is here.

Chris

From: kjwoodsISIS [mailto:notifications@github.com] Sent: 06 September 2016 17:02 To: ISISComputingGroup/IBEX Subject: Re: [ISISComputingGroup/IBEX] Data Archiving (#36)

Closed #36https://github.com/ISISComputingGroup/IBEX/issues/36.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/ISISComputingGroup/IBEX/issues/36#event-779574554, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AC5RDQz1ZU79NXr8k62bS2mttDqJJHtxks5qnY6JgaJpZM4Hd5u4.

kjwoodsISIS commented 7 years ago

Hi Chris,

I think this ticket, in its current format, is no longer relevant. It was one of the earliest tickets we created and our thinking has matured since then. Certainly, when I first wrote this ticket, the distinction between the sample environment archive, the detector data files and the ISIS data archive was not clear to me and is not reflected in the ticket.

As originally conceived, this ticket is about logging control system events (i.e. what we would now call logging of PVs and messages from IOCs). It is not about long-term archival of data.

We now have solutions for logging - the EPICS (PV) archiver and the Message archiver. These solutions don't conform to the Completion Criteria for the ticket, but that's OK, because, knowing what I now know about how things work at ISIS, I do not think the Completion Criteria are appropriate. The current EPICS and Message archivers behave sensibly.

As to the question of the "edge of the project", I would return to the project's official title "Instrument Control Project". The project's scope is the control of neutron & muon instruments at ISIS. On that basis, I would argue that once you have acquired your data you no longer have a need to control the instrument - the experiment is complete and it is time for the next scientist to do his/her experiment. What happens to the data that was acquired in the first experiment is no longer the concern of the control system.

I am sure we can improve the current EPICS and Message archivers. Any improvements (e.g. how do we handle the archives growing too large) we care to propose can be assessed against all the other things we want to do to improve IBEX and can be prioritised accordingly. I think the EPICS and Message archivers fall inside the "edge of the project" because you need them to diagnose and rectify problems (which is all part of instrument control).

When it comes to long-term data storage, my understanding of the current situation is that once an experiment is complete, it is the responsibility of the scientist to make sure that the data is safely stored somewhere (i.e. it is migrated off the control PC and that the safe location is the ISIS data archive). And, crucially, in this view, the migration is something that happens outside of IBEX. Of course, the Experimental Controls Team can assist in the migration process, and might even develop a system to automate the process, but I would argue such a system should not be part of IBEX; it should be independent.

Kevin