belforte
commented
10 years ago thanks, but... I have to reject this. this may be for the better good. But Crab2 has been like that for many years and it is a few months from end of life. So I do not want to change something that would require extensive validation. It is a part of Crab that I am not familiar with and person(s) who developed that are long gone. While certinaly this works finely for you it is difficult to be sure that there will never be an issue in other cases. At the moment we are already making lots of changes for DBS2/3 transition and need to keep the changes at minimum.
Hope you understand and do not mind keeping using this in your private copy. Maybe you can get a similar speed up if you place your crab working directory in a separete tree, not as a subdir of CMSSW area ?
Limit recursion to the 3rd level to keep data folder from path like Package/SubPackage/data
I have a lot of crab tasks hanging around, and it takes almost 10 minutes to create the tar file when creating a new task. Indeed, crab walk into all directories and stat all files to see if it's a folder. I have tasks with sometimes 3000 jobs, and crab spend all its time stating jobs res files...
With this change, it now takes ~10s to create the tar file!
One drawback is that it now tar only 'data' folder up to the 3rd level from src,
but IMO, this is the intended behavior.