search

HDFGroup / hsds

Cloud-native, service based access to HDF data
https://www.hdfgroup.org/solutions/hdf-kita/
Apache License 2.0
126 stars 52 forks source link

'Verbose' parameters for dset reads returns inaccurate until scan #255

Closed mattjala closed 8 months ago

mattjala commented 10 months ago

The 'verbose' parameter for a dataset GET returns information based on all of the datasets' chunks, which is only aggregated whenever a bucket scan occurs. These scans occur once a minute by default, so requests will return inaccurate info until the next scan occurs.

This is relevant to HDFGroup/vol-rest#55, since that API call should return the total number of allocated bytes, which must be obtained by the verbose parameter.

jreadey commented 8 months ago

Closing this as it should be resolved in master. Since the task that collect storage statistics runs asynchronously and only looks at chunks in the storage system, clients need to do a domain PUT with rescan and flush params, followed by a dataset GET with a verbose param. See checkVerbose in tests/integ/value_test.py for an example.