reorganizye scripts directory, simplity creation of TWTarballs

belforte commented 1 month ago

in current scripts directory we mix files uses for different purposes, which forces us to keep explicit lists of file names in buildTWTarballs.sh

Instead we can structure scripts so that there is a top directory for "things which need to go in the scheduler's SPOOL_DIR" and "things which need to go in the grid job directory"

That will simplify both tarball creations and use and will be safer against adding, renaming files.

See https://github.com/dmwm/CRABServer/issues/8699#issuecomment-2374495276

belforte commented 1 month ago

I paste here text from https://github.com/dmwm/CRABServer/issues/8699#issuecomment-2374495276 for convenience

add two subdirectories in CRABServer/scripts

scripts
├── dagman
│   └── task_process
└── job_wrapper

so that in https://github.com/dmwm/CRABServer/blob/e8149dda5ff7a4fee80d3b25d5c038ead60c711d/cicd/crabtaskworker_pypi/buildTWTarballs.sh#L76-L84

we only copy well named directories around, like done now for task_process

and also move expanding of job_wrapper tarball https://github.com/dmwm/CRABServer/blob/e8149dda5ff7a4fee80d3b25d5c038ead60c711d/scripts/CMSRunAnalysis.sh#L49

to the top wrapper script between these lines https://github.com/dmwm/CRABServer/blob/e8149dda5ff7a4fee80d3b25d5c038ead60c711d/scripts/gWMS-CMSRunAnalysis.sh#L225-L227

belforte commented 1 month ago

better do right after #6544 "while memory is fresh" so that I can capture all changes in a documentation #8728

dmwm / CRABServer

reorganizye scripts directory, simplity creation of TWTarballs #8727