search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

Copy submit.sh, Unpacker.py with deploy script, include in jobs #1028

Closed cinquo closed 12 years ago

cinquo commented 13 years ago

As summary, following Jose' comment:

"...having to copy by hand the submit.sh and Unpacker.py scripts into the intput sandbox directory of the gridFTP server is very fragile. They might change from Tag to Tag. The copy should be done internally in WMAgent when preparing the jobs."

submit.sh and Unpacker.py will be placed in some known location (identified by $GRIDFTP_CACHE) whose contents should be copied into each sandbox at submit time

cinquo commented 13 years ago

mcinquil: Please Review

cinquo commented 13 years ago

mcinquil: If it is fine I will change the doc accordingly.

sfoulkes commented 13 years ago

sfoulkes: This will break when we deploy it: unpackerScript = os.path.join(getWMBASE(), 'src/python/WMCore/WMRuntime/Unpacker.py')

When we install via RPM we don't have a src directory. Take a look at the condor plugin, I hacked it up to work around this. This is still really ugly, we need to find a better way to handle getWMBASE() with the RPM deployment.

cinquo commented 13 years ago

mcinquil: Thanks Steve. I see, I will provide a new patch.

sfoulkes commented 13 years ago

sfoulkes: Anyway we could have this handled by the deployment scripts for wmagent? It'd be easier to add it there, and we wouldn't have to clutter the gLite plugin with this. How is the gridftp server currently deployed?

cinquo commented 13 years ago

mcinquil: It is being deployed separately with the root account on the machine. Then in the agent configuration the jobcache is being speficied to be in the path where gridFtp is working on and the plugin picks that parameter to copy the needed files there. The aim of the ticket was the one in the first description from the Jose feedback on wave.

evansde77 commented 13 years ago

evansde: In a comment on the wrong ticket, I asked what was going on in Unpacker.py but as I understand it, this just relates to moving the Unpacker.py script.

We can add something to deploy/manage scripts to move these files if we can agree on some setting like GRIDFTP_ROOT or something to copy the files to if it is set.

spigad commented 13 years ago

spiga: taking care of it at deploy/manage scripts level makes sense to me.. Probably something like GRIDFTP_CACHE instead of _ROOT which suffix is usually meant for the source code location (if I'm not wrong.)

drsm79 commented 13 years ago

metson: +1 on _CACHE (or maybe _LOCATION?)

spigad commented 13 years ago

spiga: to me it has the concept of a cache.. but it is also fine location

evansde77 commented 13 years ago

evansde: As long as it doesnt change every 10 minutes GFTP_MELOS_UNDERPANTS will work, just need to pick one and stick to it ;-)

cinquo commented 13 years ago

mcinquil: So, probably we should close this and open a new one on the Infrastructure component for the deployment. Is there already a ticket for CrabServer deployment in Infrastructure?

drsm79 commented 13 years ago

metson: Lets open a second ticket - there is still the need to copy from $GRIDFTP_CACHE in the WMCore code.

drsm79 commented 13 years ago

metson: I opened #1434 for the changes to infrastructure

cinquo commented 13 years ago

mcinquil: #1434 + #1682 are tracking all that needs to be done.