ISISComputingGroup / IBEX

Top level repository for IBEX stories
5 stars 2 forks source link

Rearrange storage areas #5374

Open FreddieAkeroyd opened 4 years ago

FreddieAkeroyd commented 4 years ago

Leading on from #5339 I think we should rearrange some of our storage areas.

The \isis\shares\ISIS_Experimental_Controls etc areas are centrally maintained and backed up - we should use them for important ibex build stuff, it also gives us a good point to centrally mirror. This area is quotad so we should not use it as a general dumping ground, so we should move the very large \isis\shares\ISIS_Experiment_Controls\FilesMovedFromInstruments to kits$ for example

\isis\inst$\Kits$\CompGroup\ICP should be for built and derived files, large things we can regenerate, or things we can live without for some time if need be. So not things needed to build ibex.

I am looking at a gitlab LFS instance on site GitLab server as another option too, but I don't know yet how much storage space we may have. It is also not an off site solution. Similarly i have been able to attach additional storage to our stfc cloud RHEL7 node, so I could have ~1TB of space there as a mirror. I am not sure of the SLA for this service though, but again it is not an offsite solution.

acceptance criteria

DominicOram commented 4 years ago

As part of this we should document the decision of what lives where.

ChrisM-S commented 4 years ago

As this filled up this morning I've skimmed off the files into the central stage-deleted area where there is plenty of space. There is a .cmd file in the top of the area which does this but will need special permissions. There is no reason this skim could not be automated from the central archive servers (based on 3 months old backups, for example), it could also be included as part of the per cycle data moving off instruments. can be discussed when this ticket is picked up.

KathrynBaker commented 3 years ago

ChrisM-S has a script that does some of this already