search

stfc / rucio

Rucio - Scientific Data Management
http://rucio.cern.ch
Apache License 2.0
0 stars 0 forks source link

Existing reaper tests don't delete or assert anything #77

Closed patrick-austin closed 4 years ago

patrick-austin commented 4 years ago

Motivation

test_reaper.py and test_reaper2.py run the reaper, but nothing is ever deleted as the source column in rse_usage is set to srm when generating the test data, but (by default) this is storage for the RSE being used in the test. There replicas also need a tombstone set on them, which they don't have. As a result we never delete anything, but also never assert that we have so the test passes.

There's also a potential bug in the way we decide how many files to delete. If you set the RSE limits to require 10GB of free space, currently have 1GB of space free, and several 2GB files stored on the RSE, instead of deleting 5 files to give 11GB of free space, we delete 4 and end up with 9GB. This is due to the order in which we check file size vs adding the replica to a list of files to be deleted in core.replica.list_unlocked_replicas.

As I've made changes to the reaper (to allow VO to be specified), I need working reaper tests to make sure I haven't broken anything. However this is not a solely multi-VO issue, so this may need a PR against the mainline development too (rather than it going in as part of the larger multi-VO development PRs).

Modification

Change source and add tombstones to allow deletion, and add some asserts to the reaper tests. Change how we calculate the number of files that need reaping.

elichad commented 4 years ago

Closed by identical mainline issue https://github.com/rucio/rucio/issues/3726