Closed ericvaandering closed 10 years ago
ericvaandering
commented
10 years ago Comment by belforte on Tue Oct 15 05:16:52 2013
I do not understand what you say here.
I would have expected the start point to always be the AND of the dataset lumis and the user's lumimask.
Then crab -report makes the list of that MINUS the processed lumis.
This should work in all cases.
Am I missing something ? Otherwise maybe there's a bug ?
ericvaandering
commented
10 years ago Comment by fanzago on Wed Oct 16 05:02:35 2013
My point is:
I have created a task using a lumimask file, and in the config of crab I have decided to split the task in this way:
lumis_per_job = 50 number_of_jobs = 10
Then 4 of them failed. Using the option -report I know the lumis analyzed by the corrected terminated job of the task and the missing lumi as difference between the original lumimask used in the config and the analyzed lumi.
I want to provide also a file missingLumi_of_task that contains only the missing lumi of the task, not the missing lumi related to the original lumimask. So I can create a new task using the "missing_lumi_of_task" , change the splitting and be sure to complete the analysis only of data required in the first task.
ericvaandering
commented
10 years ago Comment by belforte on Wed Oct 16 05:19:52 2013
I see. This is a bug then.
In the end, regardless of how it is described, the first term of the difference must be the same list as in arguments.xml for that task. Should fix ASAP. Let me know if I should help.
ericvaandering
commented
10 years ago Comment by fanzago on Wed Oct 16 05:24:37 2013
Hi, it isn't a bug, It was decided to be implemented in this way, but it work if an user want to analyze the complete dataset via lumimask, not a subset of lumi. I let you know.
ericvaandering
commented
10 years ago Comment by fanzago on Thu Oct 17 03:57:14 2013
With the -report option two files are now created:
1) total_missingLumiSummary.json about the missing lumi between the original lumi mask used in the crab.cfg and the lumi analyzed by correctly finished jobs
2) task_missingLumiSummary.json about the missing lumi between the arguments lumi of task's jobs and the lumi analyzed by correctly finished jobs
Changes committed in cvs, Reporter.py
belforte
commented
10 years ago need to commit again to GITHUB since only 2_9_1 was imported
belforte
commented
10 years ago merged into main branch for next release
Original Savannah ticket 102859 reported by fanzago on Tue Oct 15 05:09:19 2013.
In order to complete tasks created using a lumimask file and a splitting based by lumi with a limited number of tutal lumi to analize, we have to calculate in the report also the missingLumiMask_of_task starting from the inputLumi file (json file of input lumi to analize in the created task) and not only the missingLumi starting from the original lumimask provided by user. Using the missingLumi_of_task as new lumimask for a new task, it is possible to complete the analysis avoiding the creation of new jobs with new lumi section as happens now using as new lumiMask the missingLumi provided by report.