search

fermitools / jobsub_lite

jobsub_lite is a wrapper for HTCondor job submission
Apache License 2.0
1 stars 7 forks source link

Log web sandbox creation works only with proxy #561

Closed vitodb closed 7 months ago

vitodb commented 7 months ago

When submitting jobs with token only using:

jobsub_submit --debug -G fermilab --memory 500MB --disk 500MB --auth-methods token file:///usr/bin/sleep 300

in the submission log we see

[...]
Transferring files to web sandbox...
gfal-mkdir error: 13 (Permission denied) - HTTP 401 : Authentication Error 
warning: error creating sandbox, web logs will not be available for this submission: Error: Unable to make directory https://fndcadoor.fnal.gov:2880/fermigrid/jobsub/jobs/2024_04_04/4111a9ea-f1e1-432d-a4f3-31109f3fee22
[...]
Checking if fermilab_5c91474037 -S ",compute.read,compute.create,compute.cancel,compute.modify,storage.read:/fermilab/users/vito,storage.create:/fermilab/users/vito,storage.create:/fermigrid/jobsub/jobs," credentials exist
[...]

the token scope is supposed to have write access for the web sandbox storage.create:/fermigrid/jobsub/jobs so the error seems to imply the code creating the web sandbox is using proxy authentication. As result the job runs and is able to upload its log, I guess this time using tokens for auth, but in the web sandbox we are missing:

simple.cmd
simple.sh
sleep

If the submission doesn't use --auth-methods token all is working as expected, the web sandbox is created at job submission time and all expected files are present in there.

marcmengel commented 7 months ago

So, I'm thinking maybe the token with storage.create doesn't let us make directories under the /fermigrid/jobsub/jobs area? And even then, we probably need to exempt that from the token weakening... (i.e. act like we got "--need-storage-modify /fermigrid/jobsub/jobs")

shreyb commented 7 months ago

The fermilab Analysis token, for example, does have the following scope:

storage.create:/fermigrid/jobsub/jobs

See https://landscape.fnal.gov/monitor/d/56WoQ2cVk/token-scopes-capability-sets?orgId=1&var-experiment=fermilab&var-role=Analysis

Does gfal-mkdir, or the version present on interactive nodes, default to proxy authentication or tokens?

marcmengel commented 7 months ago

Hm... it will use tokens, but I think it needs BEARER_TOKEN set, as opposed to just BEARER_TOKEN_FILE... And that's our problem. If I set BEARER_TOKEN to the contents of my token file, the submission works. So we just need to do that in fake_ifdh.py...