Write recurring action for data recall

[mmascher]

The action has to take care of two task states, RECALLING and KILLRECALL. A task goes into RECALLING after the splitting if we detect the blocks of dataset are on tape. It goes into KILLRECALL upon a user request through crab kill (he REST detects the state RECALLING)

1. RECALLING:

   • We get the list of tasks in recalling and their dynamo request id. We can use the already existing API we use for the TW passinge the RECALLING state: https://github.com/dmwm/CRABServer/blob/master/src/python/TaskWorker/MasterWorker.py#L176-L179
   • We need to add the dynamo request id to the list of fields returned by the associated query in the REST, otherwise we won't get it: https://github.com/dmwm/CRABServer/blob/master/src/python/Databases/TaskDB/Oracle/Task/Task.py#L59-L87
   • For each of the tasks obtained we check the status of the dynamo request.
     • If dynamo tells us the request has been done then we put the task in NEW (HOLDING?) using the already existing API: https://github.com/dmwm/CRABServer/blob/master/src/python/TaskWorker/MasterWorker.py#L194-L210
     • If dynamo tell us the request has failed then we get the error reson and fail the task with the appropriate error message (e.g.: https://github.com/dmwm/CRABServer/blob/master/src/python/TaskWorker/Worker.py#L81-L91).
   • We need to check if we have the timestamp of when the task has been put in STAGING and FAIL for timeout (same API as above). Notice that the timestamp of the task is not updated when we change the state https://github.com/dmwm/CRABServer/blob/master/src/python/TaskWorker/Worker.py#L81-L91 :( We probably need to change this query as well

2. KILLRECALL:

   • Get the list of task in state KILLRECALL
   • KILL the request in dynamo
   • Update the state to killed

We might get some progress info from dynamo that we might want to show to the user. We can explore the usage of the warning mechanism for that. Maybe BaseRecurringAction can inherit from TaskAction so that it has addWarning? (https://github.com/dmwm/CRABServer/blob/master/src/python/TaskWorker/Actions/Recurring/BaseRecurringAction.py and https://github.com/dmwm/CRABServer/blob/master/src/python/TaskWorker/Actions/TaskAction.py#L41-L52)

PS Please suggest better names for the states if you don't like them, they are going to be exposed to the user!

[belforte]

about the timestamp, it will be enough to base the timeout decision on the submission time or any other time that we have already in the DB, we will never be to

about the state name: all I can imagine to be nice to users is RECALLING --> RECALLING DATA but it may be enough to do it in crab status. I hate long names

[belforte]

@lecriste I guess this can be closed. Right ?

[lecriste]

No, the KILLRECALL part is missing.

[belforte]

@lecriste what's status of this ? Maybe close and open a new one for what's missing ?

[lecriste]

I started to change the code some time ago, got busy with higher priority things. No need to open another issue.

[belforte]

@lecriste can you remind me the status here ? Do you have work in progress ? All in all if we lived w/o this for 2 years, maybe we can simply say that we never kill a tape recall request once made.

[lecriste]

No work in progress. I noticed that a user could kill a request submitted by another user... shall we allow that?

[belforte]

I am a bit lost. How can a user kill a transfer request (no matter who created it) if this code is not written ?

[lecriste]

He cannot now. If we write the code he could. This may happen when a user B tries to submit a task on a dataset for which another user A has already triggered a tape recall request. The task will go in TAPERECALL status with the same Dynamo request ID of user-A task and if user B issues crab kill, he will kill the transfer request.

[belforte]

THanks for pointing that out. This adds weight to the "let's forget about this". Let's not delete recall requests. Period. We can always revisit later if there is a problem. I will put this on Hold for the time being.

[belforte]

Deleting recall requests is possible now with Rucio since each user places a rule in a different namespace.