Data storage for increasingly large datasets causes increased need for capacity planning

ices-eg / DIG

ICES Data and Information Group

Creative Commons Zero v1.0 Universal

3 stars 1 forks source link

Data storage for increasingly large datasets causes increased need for capacity planning #381

Open neil-ices-dk opened 1 year ago

neil-ices-dk commented 1 year ago

Aspect | Description -- | -- Challenges | Capacity planning and expectation management from the ICES Data Centre will be important. Realistic timelines for developing datasets and series (optimism bias) Mitigating strategies and controls | DIG to monitoring recommendations and evaluate cases. [Legacy entry](http://community.ices.dk/Committees/DIG/Lists/DIG%20future%20challenges%20and%20opportunities/DispForm.aspx?ID=6&Source=http%3A%2F%2Fcommunity%2Eices%2Edk%2FCommittees%2FDIG%2FLists%2FDIG%2520future%2520challenges%2520and%2520opportunities%2FAllItems%2Easpx&ContentTypeId=0x0103007DB8903096FFDD4BA4F2198AC9382C28)

OliverWilliamsDataManager commented 8 months ago

Receiving increasingly large datasets in itself I deemed a challenge only (opportunities from having large amounts of data not really an item for the DIG group to cover).

We know institutes are already accelerating the pace of data collection at higher frequencies due to tech advances so likelihood I deem high.

I chose medium disruption because getting more storage is in itself possible with cloud based solutions already available, however funding and managing this is not necessarily easy.