XENONnT / outsource

Job submission code for XENONnT
3 stars 2 forks source link

Want to impose rucio file check after upload #110

Open FaroutYLq opened 10 months ago

FaroutYLq commented 10 months ago

So to avoid empty rules and incomplete upload. A typical rucio corruption is incompleteness in direct upload. Below is an extreme example where nothing was really uploaded for that dataset.

(rucio-venv-1.23) [_jlstephen@rucio-xenon ~]$ rucio ls xnt_048374:peaklets-euocvpkv3y
+--------------------------------+--------------+
| SCOPE:NAME                     | [DID TYPE]   |
|--------------------------------+--------------|
| xnt_048374:peaklets-euocvpkv3y | DATASET      |
+--------------------------------+--------------+
(rucio-venv-1.23) [_jlstephen@rucio-xenon ~]$ rucio list-files xnt_048374:peaklets-euocvpkv3y
+--------------+--------+-----------+------------+----------+
| SCOPE:NAME   | GUID   | ADLER32   | FILESIZE   | EVENTS   |
|--------------+--------+-----------+------------+----------|
+--------------+--------+-----------+------------+----------+
Total files : 0
Total size : 0.000 B
FaroutYLq commented 3 months ago

Right now we just keep running in --debug mode, which avoids rucio upload and we just gfal-copy to destination. It is slow and manual but working.