Infrastructure: Long Term Storage of diagnostic instrument log data

[John-Holt-Tessella]

As a future developer I want to be able to look at historic data so that I can perform some analysis, for example look for sticky motors. This is currently stored in the MySQL instrument archive and various log files in Instrument/Var/log/ioc

Create a meeting to discuss

1. Whether we want to store the data vs just delete it every cycle
2. How we would store it (format)
3. Where we can store it

Before the meeting find out:

1. How much there is
2. Whether Tom would allow us to store it with another group (e.g. is this too expensive)
3. Rough estimate of cost.

[ChrisM-S]

There is a dynamic side to this too, so it would be good to also consider and put an upper limit on

1. Number of GB/day and no. of individual file copies/day.
2. How long the data needs to be kept (and how securely - i.e. if it's lost does it matter much?)
3. How available must the data be (i.e. minimum time to access a useful amount).
4. How is the data to be indexed/found/retrieved/catalogued.
5. How long and how much would it cost to move the data to another storage provider.

[kjwoodsISIS]

See also #3034

[John-Holt-Tessella]

Impeded until: Organised 24 May 2018 15:00 Data being taken

[John-Holt-Tessella]

See https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/Data-Generation-and-Storage

[John-Holt-Tessella]

Impeded until notes are agreed or it is 1st July. At which point close and create tickets or have another meeting.

[Tom-Willemsen]

Latest iteration of this meeting documented at https://github.com/ISISComputingGroup/ibex_developers_manual/wiki/Data-Generation-and-Storage