Ruciax tsm fixes

[XeBoris]

Dear all,

I finished a fairly huge bug fix and maintenance campaign in ruciax and tape upload related tools. This contains the following changes:

1. Fixed typo/bug in tsm-upload regarding the cross check of uploaded data and which information are shown by the tape upload tool (dsmc)
2. We can restore muon_veto files from tape now (Obviously I have never implemented this)
3. We can upload raw data sets with more then 1000 individual files in the raw data folder now. I never realized that there is an issue regarding the number of files in cax for tape uploads.
4. Changed massive-ruciax upload logic to handle upload request more clever: 4.1 Raw data sets which are marked by "error" are skipped 4.2 Re-upload raw data (status=RSEreupload) automatically with massive-ruciax 4.3 Uploads are separated from verification now.
5. Change the upload commands in massive-ruciax and massive-tsm to upload raw data by only the name (and not the run number). This reduces the complexity of the code a little. 6.Both tools (ruciax and cax-tsm) delete zero sized files in the raw data folder on xe1t-datamanager before the upload to tape or rucio is started. This becomes necessary to not be inconsistence with data on tape and rucio (In particular if we would like to restore them later). There are two consequences: 6.1. Incremental upload of old raw data to tape (in case something has changed on xe1t-datamanager and the raw data set was already uploaded to tape) is not possible anymore. We need to delete the tape backup and re-upload it again. Please not that this will not happen so often, or let's almost never. 6.2. We still need remove zero sized files from the rucio catalog. This will happen soon. 6.3. Latest pax version avoids to produce zero-sized files. No need to change anything here anymore. 6.4. Implication: Replace zero-sized raw data in the rucio catalog by one which has a size (additional script)
6. Handle random transfer rules by ruciax This request comes from Benedikt to distribute our raw data transfers on the "European" site to several locations. Therefore the the destination_rse and destination_lifetime key of the "rucio-rule" takes now arguments like this: "destination_rse": ["LNGS_USERDISK", "random:CCIN2P3_USERDISK,CNAF_USERDISK,NIKHEF_USERDISK,WEIZMANN_USERDISK", "UC_OSG_USERDISK"], "destination_livetime":{ "LNGS_USERDISK": "432000", "random:CCIN2P3_USERDISK,CNAF_USERDISK,NIKHEF_USERDISK,WEIZMANN_USERDISK": "-1", "UC_OSG_USERDISK": "-1" }, This example draw one RSE from the list (random:...) which is given and use this to create the transfer destinations.

I tested the code on xe1t-datamanager already

[pdeperio]

Try to also fix the codacy style issues (click "Details" below or run make lint in terminal).

[pdeperio]

I applied many lint/flake8 fixes to https://github.com/XENON1T/cax/pull/114, so you might want to merge that first, then sync with this branch. Then you can check what changes need to be made with make lint and automatically apply changes with e.g. autopep8 -i --ignore E501 cax/main.py, then checking again with make lint and fixing the rest manually.

Also, does your python really work with only 2-space indents?

[XeBoris]

@pdeperio If we can avoid creating zero sized acquisition monitor files in pax we need to think if we still want to keep my "protection" of uploading zero sized files to rucio/tape. For example there could also be a zero sized log file (I have seen this in the past). Therefore I will merge my branch then carefully with the latest master branch.

About the style issues: I will fix this latest before the final merging, thanks for reminding. And yes, "my python" works also with 2-space intends. But I think this is somehow pre-definied in Kate (Linux editor of the KDE environment) or the notebooks which I run on midway. I try to be more careful about it in the future.

[XeBoris]

@pdeperio I made some code review with pylint and flake8 now for: cax/main.py cax/tasks/data_mover.py cax/tasks/rucio_mover.py cax/tasks/tsm_mover.py

Should be fine now