Lobster CVMFS repo - Githubissues

NDCMS / lobster

A userspace workflow management tool for harnessing non-dedicated resources for high-throughput workloads.

MIT License

3 stars 14 forks source link

Lobster CVMFS repo #615

Open PerilousApricot opened 6 years ago

PerilousApricot commented 6 years ago

Hi all,

It's surprisingly simple to get software installed into CVMFS [1] -- perhaps it would be neat to have lobster put in there, so people could use it without needing to install themselves?

[1] http://openhtc.org/docs/data/external-oasis-repos/

annawoodard commented 6 years ago

OMGOMGOMG :+1: * 1000

annawoodard commented 6 years ago

According to @bbockelm in cooperative-computing-lab/cctools#1798 CCTools would fit well into /cvmfs/oasis.opensciencegrid.org. Maybe we can sneak in there too? Thoughts, @matz-e, @klannon, @khurtado?

bbockelm commented 6 years ago

Yes!

I'm fairly sure that LIGO deploys to /cvmfs/oasis.opensciencegrid.org/ligo/sw/pycbc from Travis-CI when the release is tagged.

annawoodard commented 6 years ago

Thanks for the heads up, we could use their script for inspiration!

matz-e commented 6 years ago

I wonder if this could be looped back into Travis to be used for more extensive unit-testing before tagging a release (which we currently don't really do). Aka #231, but with real CMSSW + cctools?

annawoodard commented 6 years ago

I think that's a great idea, but that it makes sense to do this in stages. I think getting tagged releases into CVMFS is higher-priority and would be a good start, and improved tests can be tacked on afterwards.

bbockelm commented 6 years ago

If you're copying their script, there's a few tweaks to do (e.g., remove the setting of setting --no-check-certs...)

bbockelm commented 6 years ago

Probably also related to #575

matz-e commented 6 years ago

Suggested approach:

[ ] Test procedure creating a local directory and installing a Lobster release with all packages needed there
[ ] Find ways to override PYTHONPATH etc like when developing CRAB
[x] Look into the whole CVMFS stuff
[ ] Clone script from ligo
[ ] Hook it up into infrastructure via tags

matz-e commented 6 years ago

After reading the software update page that @btovar dug out, I'm under the impression that we need a VO to be in CVMFS, even under OASIS. That sounds a little more evolved, and should potentially cover both cctools and Lobster…

matz-e commented 6 years ago

@klannon maybe this would be something to discuss in the meeting today from the Lobster side?