ganga-devs / ganga

Ganga is an easy-to-use frontend for job definition and management
GNU General Public License v3.0
100 stars 159 forks source link

Making Dirac proxy stuck in loop #2280

Closed ghost closed 8 months ago

ghost commented 8 months ago

When using the Dirac backend ganga (8.7.2) cannot find my existing proxy and gets stuck in a loop making new proxies. Each of these proxies also is in a new file (the <vo>_user string gets appended in pairs each time)

# First proxy made
path         : /tmp/x509up_u<num>:<vo>_user
# Second proxy made
path         : /tmp/x509up_u<num>:<vo>_user:<vo>_user:<vo>_user
# Third proxy made
path         : /tmp/x509up_u<num>:<vo>_user:<vo>_user:<vo>_user:<vo>_user:<vo>_user

I managed to get this error with the following

j = Job()
j.backend = Dirac()
j.submit()
mesmith75 commented 8 months ago

What happens if you do

credential_store.create(DiracProxy())
ghost commented 8 months ago

This also spawns a proxy creation loop

mesmith75 commented 8 months ago

What exactly is the printout?

ghost commented 8 months ago

Here is the full printout (with minor edits to remove personal information). This loop of proxies just keeps going.

[15:51:23]                                                                                                                                                                                                                                                                                                                                                                               
Ganga In [9]: credential_store.create(DiracProxy())                                                                                                                                                                                                                                                                                                                                      
Generating proxy...                                                                                                                                                                                                                                                                                                                                                                      
Enter Certificate password:                                                                                                                                                                                                                                                                                     
Contacting CS...                                                                                                                                                                                                                                                                                                                                                                         
Checking DN /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>                                                                                                                                                                                                                                                                                                                       
Username is <first>.<last>                                                                                                                                                                                                                                                                                                                                                                 
Creating proxy for <first>.<last>@<vo>_user (/C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>)                                                                                                                                                                                                                                                                                         
Added VOMS attribute /<vo>                                                                                                                                                                                                                                                                                                                                                                 
Uploading proxy..                                                                                                                                                                                                                                                                                                                                                                        
Uploading /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name> proxy to ProxyManager...                                                                                                                                                                                                                                                                                                
Loading user proxy                                                                                                                                                                                                                                                                                                                                                                       
Uploading proxy on-the-fly                                                                                                                                                                                                                                                                                                                                                               
Cert file /home/hep/<USER>/.globus/usercert.pem                                                                                                                                                                                                                                                                                                                                          
Key file  /home/hep/<USER>/.globus/userkey.pem                                                                                                                                                                                                                                                                                                                                           
Loading cert and key                                                                                                                                                                                                                                                                                                                                                                     
User credentials loaded                                                                                                                                                                                                                                                                                                                                                                  
 Uploading...                                                                                                                                                                                                                                                                                                                                                                            
Proxy uploaded                                                                                                                                                                                                                                                                                                                                                                           
Proxy generated:                                                                                                                                                                                                                                                                                                                                                                         
subject      : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>/CN=.../CN=...                                                                                                                                                                                                                                                                                          
issuer       : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>/CN=...                                                                                                                                                                                                                                                                                                       
identity     : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>                                                                                                                                                                                                                                                                                                                    
timeleft     : 23:53:58                                                                                                                                                                                                                                                                                                                                                                  
DIRAC group  : <vo>_user                                                                                                                                                                                                                                                                                                                                                                   
path         : /home/hep/<USER>/.x509_proxy:<vo>_user:<vo>_user                                                                                                                                                                                                                                                                                                                              
username     : <first>.<last>                                                                                                                                                                                                                                                                                                                                                              
properties   : NormalUser, JobSharing, AccountingMonitor                                                                                                                                                                                                                                                                                                                                 
VOMS         : True
VOMS fqan    : ['/<vo>']

Proxies uploaded:
 DN                                                     | Group | Until (GMT)
 /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name> |  | 2024/12/19 16:27
Generating proxy...
Enter Certificate password:   
Contacting CS...
Checking DN /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>
Username is <first>.<last>
Creating proxy for <first>.<last>@<vo>_user (/C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>)
Added VOMS attribute /<vo>
Uploading proxy..
Uploading /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name> proxy to ProxyManager...
Loading user proxy
Uploading proxy on-the-fly
Cert file /home/hep/<USER>/.globus/usercert.pem
Key file  /home/hep/<USER>/.globus/userkey.pem
Loading cert and key
User credentials loaded
 Uploading...
Proxy uploaded
Proxy generated:
subject      : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>/CN=..../CN=...
issuer       : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>/CN=...
identity     : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>
timeleft     : 23:53:59
DIRAC group  : <vo>_user
path         : /home/hep/<USER>/.x509_proxy:<vo>_user:<vo>_user:<vo>_user:<vo>_user
username     : <first>.<last>
properties   : NormalUser, JobSharing, AccountingMonitor
VOMS         : True
VOMS fqan    : ['/<vo>']

Proxies uploaded:
 DN                                                     | Group | Until (GMT)
 /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name> |  | 2024/12/19 16:27
Generating proxy...
Enter Certificate password:   
Contacting CS...
Checking DN /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>
Username is <first>.<last>
Creating proxy for <first>.<last>@<vo>_user (/C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>)
Added VOMS attribute /<vo>
Uploading proxy..
Uploading /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name> proxy to ProxyManager...
Loading user proxy
Uploading proxy on-the-fly
Cert file /home/hep/<USER>/.globus/usercert.pem
Key file  /home/hep/<USER>/.globus/userkey.pem
Loading cert and key
User credentials loaded
 Uploading...
Proxy uploaded
Proxy generated:
subject      : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>/CN=.../CN=...
issuer       : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>/CN=...
identity     : /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name>
timeleft     : 23:53:58
DIRAC group  : <vo>_user
path         : /home/hep/<USER>/.x509_proxy:<vo>_user:<vo>_user:<vo>_user:<vo>_user:<vo>_user:<vo>_user
username     : <first>.<last>
properties   : NormalUser, JobSharing, AccountingMonitor
VOMS         : True
VOMS fqan    : ['/<vo>']

Proxies uploaded:
 DN                                                     | Group | Until (GMT)
 /C=UK/O=eScience/OU=<OU>/L=Physics/CN=<full-name> |  | 2024/12/19 16:27
Generating proxy...
Enter Certificate password:
egede commented 8 months ago

Can you test if you see the same behaviour when using the /cvmfs installation of Ganga. That would help us to narrow down the problem.

ghost commented 8 months ago

I observe the same behaviour when using /cvmfs/ganga.cern.ch/runGanga-dirac.sh

mesmith75 commented 8 months ago

What do you see in ganga for:

config.DIRAC.proxyInfoCmd
config.DIRAC.proxyInitCmd

And what do the commands listed there produce on the command line?

ghost commented 8 months ago

So I see dirac-proxy-info and dirac-proxy-init as I would have expected.

dirac-proxy-info returns the information about my proxy.

dirac-proxy-init by itself does not work. I use dirac-proxy-init -g <vo>_user -M -C $HOME/.globus/usercert.pem -K $HOME/.globus/userkey.pem $@ whenever I make a proxy from the command line.

I have also just noticed that the dirac proxy does not resepct the "X509_USER_PROXY" env variable I have been using for generating voms proxies

mesmith75 commented 8 months ago

So it sounds like you need to modify your ~/.gangarc. In there you can find the proxyInitCmd option, uncomment it and replace it with what you use on the command line.

ghost commented 8 months ago

Okay that appears to have solved the looping proxy issue. Thanks to you both for your assistance