search

fermitools / jobsub_lite

jobsub_lite is a wrapper for HTCondor job submission
Apache License 2.0
1 stars 7 forks source link

jobsub_cleanup_cred doesn't work for non-Analysis users #549

Closed shreyb closed 1 month ago

shreyb commented 8 months ago

@kherner reported this issue. He was trying to remove DUNE production credentials from the jobsub schedds, and got this error. This is a known issue, and I thought we had a bug open for it, but it looks like we don't. Here was the stderr:

$ jobsub_cleanup_cred -G dune --role production --global-pool dune
Schedds to clean: [REDACTED]
credentials on REDACTED:
dune_production_HASHREDACTED.top = 1701890372
dune_production_HASHREDACTED.use = 1710251842
Traceback (most recent call last):
 File "/opt/jobsub_lite/bin/jobsub_cleanup_cred", line 131, in <module>
  main()
 File "/opt/jobsub_lite/bin/jobsub_cleanup_cred", line 123, in main
  del_cred(schedd_host, tname, nflag)
 File "/opt/jobsub_lite/bin/jobsub_cleanup_cred", line 89, in del_cred
  tname, handle = tname.split("_")
ValueError: too many values to unpack (expected 2)
shreyb commented 1 month ago

Did a lot of cleanup, and added error handling. Ready to test on the following cases:

  1. Own token (should prompt, and then delete)
  2. hypotpro token (should prompt, and then delete)
  3. Own token, --force (should not prompt for deletion, then delete)
  4. Own token, --no-delete (should not prompt for deletion, and should not delete)
  5. Own token, --force and --no-delete (same as (4))
  6. Own token, specified schedd (-name) (should prompt, ONLY delete on specified schedd. Set this up by submitting jobs to multiple schedds using -n, --schedd-for-testing flags for 2 schedds.)
  7. Own token, specified schedd (-name), --no-delete (should prompt, NOT delete anything. Set this up by submitting jobs to multiple schedds using -n, --schedd-for-testing flags for 2 schedds.)
  8. Own token, specified schedd (-name), --force (should not prompt, ONLY delete on specified schedd. Set this up by submitting jobs to multiple schedds using -n, --schedd-for-testing flags for 2 schedds.)
  9. Own token, specified schedd (-name), --force, --no-delete (should not prompt, NOT delete anything. Set this up by submitting jobs to multiple schedds using -n, --schedd-for-testing flags for 2 schedds.)
shreyb commented 1 month ago

All tests pass:

1. Own token (should prompt, and then delete) - PASS

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab
This will delete all of your tokens on the following schedds: schedd04.domain, schedd05.domain, schedd01.domain, schedd02.domain, schedd03.domain.
Are you sure you want to proceed (Y/n)? n
Not deleting tokens.

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab
This will delete all of your tokens on the following schedds: schedd04.domain, schedd05.domain, schedd01.domain, schedd02.domain, schedd03.domain.
Are you sure you want to proceed (Y/n)? Y
Schedds to clean: ['schedd04.domain', 'schedd05.domain', 'schedd01.domain', 'schedd02.domain', 'schedd03.domain']
credentials on schedd04.domain:
credentials on schedd05.domain:
credentials on schedd01.domain:
credentials on schedd02.domain:
credentials on schedd03.domain:
fermilab.top = 1727294118
fermilab.use = 1727464408
Running: _condor_CREDD_HOST=schedd03.domain condor_store_cred delete-oauth -s fermilab
Account: <current> (<user>)
CredType: oauth

Operation succeeded.

After:
credentials on schedd04.domain:
credentials on schedd05.domain:
credentials on schedd01.domain:
credentials on schedd02.domain:
credentials on schedd03.domain:
Done.

2. hypotpro token (should prompt, and then delete) - PASS

Setup: pushed hypotpro managed token to dev machine, then got bearer token with that.

$ jobsub_lite/bin/jobsub_cleanup_cred -G hypot --role production
This will delete all of your tokens on the following schedds: schedd04.domain, schedd05.domain, schedd01.domain, schedd02.domain, schedd03.domain.
Are you sure you want to proceed (Y/[n])? Y
Schedds to clean: ['schedd04.domain', 'schedd05.domain', 'schedd01.domain', 'schedd02.domain', 'schedd03.domain']
credentials on schedd04.domain:
hypot_production.top = 1727409512
hypot_production.use = 1727465182
hypot_production_af81385931.top = 1726319073
hypot_production_af81385931.use = 1727465490
Running: _condor_CREDD_HOST=schedd04.domain condor_store_cred delete-oauth -s hypot_production -H af81385931
Account: <current> (hypotpro)
CredType: oauth

Operation succeeded.

Running: _condor_CREDD_HOST=schedd04.domain condor_store_cred delete-oauth -s hypot -H production
Account: <current> (hypotpro)
CredType: oauth

Operation succeeded.

credentials on schedd05.domain:
hypot_production.top = 1727409512
hypot_production.use = 1727464294
Running: _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s hypot -H production
Account: <current> (hypotpro)
CredType: oauth

Operation succeeded.

credentials on schedd01.domain:
hypot_production.top = 1727370751
hypot_production.use = 1727465254
Running: _condor_CREDD_HOST=schedd01.domain condor_store_cred delete-oauth -s hypot -H production
Account: <current> (hypotpro)
CredType: oauth

Operation succeeded.

credentials on schedd02.domain:
hypot_production.top = 1727409510
hypot_production.use = 1727464295
Running: _condor_CREDD_HOST=schedd02.domain condor_store_cred delete-oauth -s hypot -H production
Account: <current> (hypotpro)
CredType: oauth

Operation succeeded.

credentials on schedd03.domain:
hypot_production.top = 1727409511
hypot_production.use = 1727464657
Running: _condor_CREDD_HOST=schedd03.domain condor_store_cred delete-oauth -s hypot -H production
Account: <current> (hypotpro)
CredType: oauth

Operation succeeded.

After:
credentials on schedd04.domain:
credentials on schedd05.domain:
credentials on schedd01.domain:
credentials on schedd02.domain:
credentials on schedd03.domain:
Done.
$

3. Own token, --force (should not prompt for deletion, then delete) - PASS

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab --force
Schedds to clean: ['schedd04.domain', 'schedd05.domain', 'schedd01.domain', 'schedd02.domain', 'schedd03.domain']
credentials on schedd04.domain:
credentials on schedd05.domain:
fermilab.top = 1727465661
fermilab.use = 1727465662
fermilab_941320c798.top = 1727465661
fermilab_941320c798.use = 1727465662
Running: _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab -H 941320c798
Account: <current> (<user>)
CredType: oauth

Operation succeeded.

Running: _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab
Account: <current> (<user>)
CredType: oauth

Operation succeeded.

credentials on schedd01.domain:
credentials on schedd02.domain:
credentials on schedd03.domain:
After:
credentials on schedd04.domain:
credentials on schedd05.domain:
credentials on schedd01.domain:
credentials on schedd02.domain:
credentials on schedd03.domain:
Done.

4. Own token, --no-delete (should not prompt for deletion, and should not delete) - PASS - mods made

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab --no-delete
Schedds to clean: ['schedd04.domain', 'schedd05.domain', 'schedd01.domain', 'schedd02.domain', 'schedd03.domain']
credentials on schedd04.domain:
credentials on schedd05.domain:
credentials on schedd01.domain:
        fermilab.top = 1727465805
        fermilab.use = 1727465805
        fermilab_941320c798.top = 1727465805
        fermilab_941320c798.use = 1727465805
I would run:  _condor_CREDD_HOST=schedd01.domain condor_store_cred delete-oauth -s fermilab -H 941320c798

I would run:  _condor_CREDD_HOST=schedd01.domain condor_store_cred delete-oauth -s fermilab

credentials on schedd02.domain:
credentials on schedd03.domain:

--no-delete was specified, so no credentials were deleted. Done.

5. Own token, --force and --no-delete (same as (4)) - PASS

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab --no-delete --force
Schedds to clean: ['schedd04.domain', 'schedd05.domain', 'schedd01.domain', 'schedd02.domain', 'schedd03.domain']
credentials on schedd04.domain:
credentials on schedd05.domain:
credentials on schedd01.domain:
        fermilab.top = 1727465805
        fermilab.use = 1727465805
        fermilab_941320c798.top = 1727465805
        fermilab_941320c798.use = 1727465805
I would run:  _condor_CREDD_HOST=schedd01.domain condor_store_cred delete-oauth -s fermilab

I would run:  _condor_CREDD_HOST=schedd01.domain condor_store_cred delete-oauth -s fermilab -H 941320c798

credentials on schedd02.domain:
credentials on schedd03.domain:

--no-delete was specified, so no credentials were deleted. Done.

6. Own token, specified schedd (-name) (should prompt, ONLY delete on specified schedd. Set this up by submitting jobs to multiple schedds using --schedd-for-testing flags for 2 schedds.) - PASS

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab -name schedd05.domain
This will delete all of your tokens on the following schedds: schedd05.domain.
Are you sure you want to proceed (Y/[n])? Y
Schedds to clean: ['schedd05.domain']
credentials on schedd05.domain:
        fermilab.top = 1727466325
        fermilab.use = 1727466325
        fermilab_941320c798.top = 1727466325
        fermilab_941320c798.use = 1727466327
Running: _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab -H 941320c798

Account: <current> (<user>)
CredType: oauth

Operation succeeded.

Running: _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab

Account: <current> (<user>)
CredType: oauth

Operation succeeded.

After:
credentials on schedd05.domain:
Done.

7. Own token, specified schedd (-name), --no-delete (should prompt, NOT delete anything. Set this up by submitting jobs to multiple schedds using --schedd-for-testing flags for 2 schedds.) - PASS

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab -name schedd05.domain --no-delete
Schedds to clean: ['schedd05.domain']
credentials on schedd05.domain:
        fermilab.top = 1727466374
        fermilab.use = 1727466375
        fermilab_941320c798.top = 1727466374
I would run:  _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab

I would run:  _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab -H 941320c798

--no-delete was specified, so no credentials were deleted. Done.

8. Own token, specified schedd (-name), --force (should not prompt, ONLY delete on specified schedd. Set this up by submitting jobs to multiple schedds using --schedd-for-testing flags for 2 schedds.) - PASS

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab -name schedd05.domain --force
Schedds to clean: ['schedd05.domain']
credentials on schedd05.domain:
        fermilab.top = 1727466374
        fermilab.use = 1727466375
        fermilab_941320c798.top = 1727466374
        fermilab_941320c798.use = 1727466379
Running: _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab -H 941320c798

Account: <current> (<user>)
CredType: oauth

Operation succeeded.

Running: _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab

Account: <current> (<user>)
CredType: oauth

Operation succeeded.

After:
credentials on schedd05.domain:
Done.

Had credds on schedd01, and they were left alone.

9. Own token, specified schedd (-name), --force, --no-delete (should not prompt, NOT delete anything. Set this up by submitting jobs to multiple schedds using --schedd-for-testing flags for 2 schedds.) - PASS

$ jobsub_lite/bin/jobsub_cleanup_cred -G fermilab -name schedd05.domain --force --no-delete
Schedds to clean: ['schedd05.domain']
credentials on schedd05.domain:
        fermilab.top = 1727466516
        fermilab.use = 1727466516
        fermilab_941320c798.top = 1727466516
        fermilab_941320c798.use = 1727466517
I would run:  _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab -H 941320c798

I would run:  _condor_CREDD_HOST=schedd05.domain condor_store_cred delete-oauth -s fermilab

--no-delete was specified, so no credentials were deleted. Done.
shreyb commented 1 month ago

I made a few UX modifications after tests, but it looks fine. Ready for PR.