search

fermilab-accelerator-ai / meetings

Meeting notes and information
2 stars 0 forks source link

Data storage #1

Open gnperdue opened 4 years ago

gnperdue commented 4 years ago

Data storage progress and requirements

gnperdue commented 4 years ago

Currently using /pnfs/ldrd/accelai for long-term storage. Should confirm how much storage is available, whether the data is tape-backed, and whether we have a scratch allocation. We must also have NAS disk available.

gnperdue commented 4 years ago

Note, we have 5 TB on tev.fnal.gov under /lfstev/accelai.

June 6, 2019 we got our dCache. /pnfs/ldrd/accelai is mounted on our GPVM, accelaigpvm01.fnal.gov. Everything in dCache except what is under scratch is tape backed. Not clear how much space we have in dCache.

We also have /accelai/app and /accelai/data NAS disk. 500 GB of app space, and 1 TB of data in /accelai. There are user quotas in place, possibly 10 GB in app and 100 GB in data. (I may have asked for a bump on the quota in app, but I can't find evidence of a response.)

gnperdue commented 4 years ago 
The nightly jobs collecting parameter and event histories have been 
running stably under cron jobs by the group user, 
gmpsai-prod@accelaigpvm01.fnal.gov.
gnperdue commented 4 years ago

@jasonstjohn - do we have any scripts running by cron that check whether copies were successful and how much was copied, etc.? When I was doing Minerva ops we had stuff like that data archival. Maybe I could dig those scripts up or it might be easier to write new ones, but it would probably be good to get a daily or weekly digest email that demonstrated copy success, etc.

gnperdue commented 4 years ago

Note that this issue has some pretty strong interactions with #31