Different proxies for different VOs / one daemon per VO option

[elichad]

Motivation

Different VOs may want to run the reaper or other daemons in different ways, with different constraints (e.g. greedy vs non-greedy deletion).

Modification

Provide an option for daemons such as reaper to be run on a one-per-VO basis.

[patrick-austin]

While scoping out the reaper files, noticed that both light and dark versions of the reaper were in a broken state. Changes had been made to handle multi-VO in line with the main reaper (namely adding include/exclude expressions and preventing the use of names only if in m-VO mode). However the bin for these daemons was unchanged (so these arguments couldn't be passed) and the daemons themselves were broken in the way they handled names/ids/full dictionaries for the RSEs. It doesn't appear as though there is any test for these daemons in the Travis suite, which explains why this went unnoticed. Even if we don't add tests for their functionality, it might be worth testing that we at least handle the m-VO logic, however this logic may change depending on how we handle the daemons so haven't written anything for this yet. As for the main change for specifying how we handle daemons for VOs independently, this will require further discussion as it's linked to how the instance is configured and where the daemons can be run from. Will look into how we could do it for the reaper and hopefully identify any other problematic daemons ahead of deciding how to implement the change(s).

[patrick-austin]

Have skimmed the options for all the daemons, and tried to identify which ones may need work. There may still be some advantage in making them all accept VO as an argument (maybe so you can turn off the daemons at one VO for troubleshooting while leaving the other VOs unaffected?) but most don't seem like they critically need this adding as they're just performing routine operations.

Inherently one per VO daemons due to client use:

• Automatix: injects test data so not essential for operation

• C3PO: being retired anyway

• Sonar: being retired anyway

Daemons with no or minor (e.g. sleep time, number of threads) command line options that should be able to run for all VOs concurrently:

• Abacus (all variants)

• Minos (and the temporary expiration variant)

• Necromancer

• Cache Consumer

• Conveyor Finisher/Poller/Reciever/Throttler

• Hermes

• Judge(s)

• Kronos

• OAuthmanager

• Transmogrifier

• Undertaker

Daemons with command line options that VOs may want to be different (e.g. lifetime for rule expiry):

• Atropos

Daemons with RSE related command line options that should have VO control if two VOs wanted different options (though at present you can achieve this by formatting expressions like vo=tst&(…) this isn't secure):

• Conveyor Submitter/Stager

• Reaper (all variants)

• Suspicious Replica Recoverer

Other:

• Auditor: on hold pending mainline development?

• BB8: has to be given an RSE ID which should make it restricted by VO as these should only be available to people at that VO, but may want a further check on VO for extra security?

[elichad]

Found an old note - it might be that moving to running daemons in containers (rather than with supervisord) could make it easier to pass individual VO options if the running multiple instances of the daemon. Possibly worth considering but I don't know if it's actually better.

[patrick-austin]

Having the option to specify the VO was included with #3888

The proxy however is still somewhat of an issue, as the config file only specifies one usercert under [conveyor], so even though the conveyor-submitter can pick up requests from multiple VOs it cannot select which certificate to use. Currently you would have to run the daemons off a different config file (so Docker container or another VM) to get them to be effective.

[Thysk]

I believe that I have worked on the code to allow conveyor to select certs based on VO information enabling a single submitter to run multiple VOs with no issue caused by a single usercert

rucio#4732