Documentation on using HTC

[gwarf]

While I was trying to test the UI role/package we have in UMD, I took some notes and drafted a tutorial on using HTC. It needs some reviews/polishing, and as I haven't been doing this kind of things since long, all suggestions are very welcome!

See:

• https://docs.egi.eu/documentation/535/users/tutorials/htc-job-submission/
• https://docs.egi.eu/documentation/535/users/compute/high-throughput-compute/querying-information-system/
• https://docs.egi.eu/documentation/535/internal/operations-portal/using-vapor/

In the HTC section we only cover the Workload Manager, but I feel it may be handy to have this covered, at least for us or for more advanced use cases/users.

It's also based on ansible-roles that are being updated, same for the repo for setting up the UI.

Missing (at least):

• [x] provide proper documentation for querying the information system using GLUE 2
• [x] Document querying the information system using GLUE 2 to find resources accessible for a given VO
• [x] documenting/managing credential setup ~and possible transfer to the VM~ (this part was moved to the related repo)
• [x] add intro with references and so on
• [x] cover DIRAC CLI? (added references to existing documentation)
• [x] ~cover or link to basic data management~ (postponed to later)

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[enolfc]

I think this is good to have. I was struggling to do something similar a few months ago, but never managed to submit an actual job.

Are you happy with it already? I see there are some FIXME but maybe this is for later

Also the current HTC docs are rather useless, we should probably migrate some of the content of this tutorial there (in a followup PR)

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[gwarf]

I think this is good to have. I was struggling to do something similar a few months ago, but never managed to submit an actual job.

I was able to submit a few 🎉

Are you happy with it already? I see there are some FIXME but maybe this is for later

I've made various improvements and some cleaning, I think now the main missing point is some updated instructions on querying the information system to identify the resources available to a specific VO, as @paolini78 pointed out that lcg-info is only looking at GLUE 1 and thus lacking information that are only available via GLUE 2.

Also the current HTC docs are rather useless, we should probably migrate some of the content of this tutorial there (in a followup PR)

I agree we can move some content later, will be already better than nothing. Maybe in the meantime we could link from the HTC page to the tutorial, will push a change for this.

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[gwarf]

It's very valuable to have all this information together in a tutorial!

Just a comment: Should we clarify who's the target reader? Maybe I am biased but I would expect this tutorial to be followed by tech savvy people rather than end users (researchers) wanting to simply submit HTC jobs.

It depends how the specific user community wants to manage their jobs, from what I know, usually you can have different profile in a single user community, some with basic needs and knowledge, and others needing or enjoying a more advanced or low level stuff, that they can script or whatever.

I will try to add something in the intro.

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[sebastian-luna-valero]

Should we add a sentence to reflect the addition in https://github.com/EGI-Federation/documentation/pull/539 ?

[gwarf]

Should we add a sentence to reflect the addition in #539 ?

We currently have this: https://docs.egi.eu/documentation/535/users/tutorials/htc-job-submission/#via-the-egi-workload-manager Any thing special to suggest?

I will consider this PR as ready to be merged once we will have a few updated ldapsearch queries describing how to find all resources (CE, so including GLUE1 and GLUE2 as some may only be in one of them, in the Top BDII.

[sebastian-luna-valero]

Any thing special to suggest?

That's all, thanks!

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[andrea-manzi]

@gwarf i think it's ready now right?

[gwarf]

Still missing some bits on looking into GLUE 2 for finding the resources available to a given VO, like dteam. See https://docs.egi.eu/documentation/535/users/tutorials/htc-job-submission/#use-case-identifying-all-the-computing-elements-supporting-the-dteam-vo. Need also to review/test all the updated queries, to be sure they are OK

If you have something to suggest here... :) (it's more a mater of time for testing/doing in fact...)

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[andrea-manzi]

@gwarf i was wondering if we should include the BDII queries in a kind of "advanced users" section, and instead document how to use VAPOR to get the endpoints.

[gwarf]

@gwarf i was wondering if we should include the BDII queries in a kind of "advanced users" section, and instead document how to use VAPOR to get the endpoints.

In this tutorial I was rally willing to focus on using the CLI tools first, including for the query. But we should also probably document using VAPOR, or give a hint about this or point to some doc somewhere, if there is some. And in fact the tutorial itself is meant for advanced users, cf the info at the start.

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[gwarf]

There is now some documentation about identifying/selecting some CE, I'm not really happy with it as it's quite unclear how one could either select a given CE, or identify the proper info to be used with condor_submit and arcsub, but I'm not sure how to improve that part. It's also probably a bit too complex/long, but I guess we can later move some content under the main HTC section and simplify this one. Ideally someone else should look at testing the queries/following the tutorial, but if nobody is having time for this, we can probably get this version merged. Unless someone is having any other comments or requests, thanks!

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[gwarf]

If someones in EGI Foundation wants to test, there i an UI to whom people in EGI Foundation can get access, just ping me on Slack.

[andrea-manzi]

If someones in EGI Foundation wants to test, there i an UI to whom people in EGI Foundation can get access, just ping me on Slack. i would be happy to have access to the UI and follow the tutorial steps!

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535

[github-actions[bot]]

Documentation preview deployed!

Available at https://docs.egi.eu/documentation/535