Lessons learned: running and failing behaviour

[jsturdy]

TIF issues have brought to light several improvements that must be made Some of these may better fit into the gem-light-dqm package, but we'll summarize here

• When the LightDQM triggers the dqm processing of a run (including the unpacker), the following should be guaranteed:
  • If the processing of a run fails any step (dqm or unpacker), the chunk file should not be removed, but somehow marked as failed.
  • If all dqm jobs fail for a given call, then the entire run should be marked as problematic, and the LightDQM should not process any more chunks from that run
  • The location of the PROOF processing should be configurable, so that it is not hardcoded to the users home directory
  • Any cases we have seen that can result in a processing and then in an output file not being produced but the input file being removed need to be handled, for the moment, never removing the chunk files may be the safest alternative (possibly by moving them to a run000XXX_chunks directory)
• Further daemonization of the LightDQM should be undertaken, so there is an init script that takes care of setting up ports, cleaning up old logs, putting logs in system locations...

Probably there are other immediately necessary improvements, so let's try to get them sorted as quickly as possible

[bdorney]

to help alleviate the issue I have added a crontab to execute at 00h30 everyday on the user account to delete all files older than 2 days from both the ~/.Trash and ~/Downloads directory:

29 0 find /home/GE11COSMICS/Downloads/ -mtime +2 -delete 29 0 find /home/GE11COSMICS/.Trash/ -mtime +2 -delete