Dirac proxy multi VO problems

[alexanderrichards]

Here is an email from simon quoting an LSST user:

We've been given the long report below from a user testing the LSST VO
using ganga + our DIRAC server. The gist of it seems to be that ganga is
getting a vanilla proxy, which the DIRAC server will then attach a VOMS
proxy to at job submission time. Unfortunately this user is a member of
multiple VOs and DIRAC sometimes picks a different VO to the one they're
trying to test... I guess the questions we need to answer are:

 - Is this behaviour reproducible by us?
 - Is there some way to get ganga to get a VOMS proxy so that there is no
   room for the DIRAC server to make any decisions on the VO?

Would you be able to have a look at this?

The original email is below. Hopefully this is a small fix but obviously the new credentials system will be the proper solution.

Most of the jobs following those 4 failed with a mixture of

Stalling for more than 11700 sec and Job stalled: pilot not running

at all sites but Birmingham where they weren't supposed to run.

Since I put the right dirac-proxy-init in .gangarc I looked a bit better at what happens and it seems >not to care, it just generates a plain proxy.

if I run the dirac command standalone I get this proxy {quote} aforti@vm7>dirac-proxy-init -g lsst_user -M Generating proxy... Enter Certificate password: Added VOMS attribute /lsst Uploading proxy for lsst_user... Proxy generated: subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy/CN=proxy issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti timeleft : 23:53:59 DIRAC group : lsst_user path : /tmp/x509up_u500 username : alessandra.forti properties : NormalUser VOMS : True VOMS fqan : ['/lsst']

Proxies uploaded: DN | Group | Until (GMT) /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti | vo.northgrid.ac.uk_user | 2016/11/03 11:48 /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti | gridpp_user | 2016/11/03 11:48 /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti | lsst_user | 2016/11/03 11:48 aforti@vm7>voms-proxy-info -all subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy/CN=proxy issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=proxy type : proxy strength : 1024 bits path : /tmp/x509up_u500 timeleft : 23:53:42 key usage : Digital Signature, Key Encipherment, Data Encipherment === VO lsst extension information === VO : lsst subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti issuer : /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=Services/CN=voms1.fnal.gov attribute : /lsst/Role=NULL/Capability=NULL timeleft : 23:53:42 uri : voms1.fnal.gov:15003 {quote}

when I put that command in ganga this is what happen instead

{quote} aforti@vm7>grep dirac-proxy-init .gangarc [defaults_GridCommand]init = dirac-proxy-init -g lsst_user -M

aforti@vm7>ganga Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti Enter GRID pass phrase for this identity: Creating proxy ........................................................................................................................... Done Your proxy is valid until: Fri Nov 20 23:16:25 2015

* Welcome to Ganga * Version: Ganga-6-1-6-hotfix1 Documentation and support: http://cern.ch/ganga Type help() or help('index') for online help.

This is free software (GPL), and you are welcome to redistribute it under certain conditions; type license() for details.

Ganga.Utility.Config : INFO reading config file /home/aforti/.gangarc

In [1]: Do you really want to exit ([y]/n)? y Ganga.Core.MonitoringComponent : INFO Stopping the monitoring component... aforti@vm7>voms-proxy-info -all subject : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti/CN=400330830 issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti identity : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti type : RFC compliant proxy strength : 1024 bits path : /tmp/x509up_u500 timeleft : 23:59:43 key usage : Digital Signature, Key Encipherment, Data Encipherment {quote}

it generates a plain proxy without VOMS information. With LHCb this still works because they have >only LHCb on their servers but with the multi-VO gridpp Dirac it picks the first VO I belong to to run >the jobs if the jobs are submitted without VOMS credentials.

[drmarkwslater]

I can't actually recreate this behaviour outside Ganga - i.e. I can't get dirac-proxy-init to NOT add the VOMS extensions - can anyone else recreate this outside Ganga?

[rob-c]

@drmarkwslater We're not using this tool to create destroy proxies. We use the voms-proxy-init most of the time I think (I've not played around with the vanilla install for a while) If the dirac-proxy-init does something more sensible than voms-proxy-init or grid-proxy-init currently in the credentials code then surely the best thing to do is to use this as the default tool? (also this goes more hand in hand with the GridPP way of presenting Ganga+Dirac to be the best way to use the grid)

[afortiorama]

Hi,

dirac-proxy-init in .gangarc is suggested in this documentation written by Mark

https://www.gridpp.ac.uk/wiki/Guide_to_Ganga#Installation_and_Configuration

At first I thought the problem was -M missing from the command, but even with that it didn't work. We are heavily relying on that wiki in the UK for smaller or local groups. If it needs to be corrected please correct it.

[drmarkwslater]

I put this in as it is what is used by the Dirac docs on the GridPP wiki. This will mean we get the Dirac user group as well as a default voms proxy. For LHCb, this is set to lhcb-proxy-init I believe.

In any case, @afortiorama are you able to get dirac-proxy-init to NOT create the VOMS extensions? I don't seem to be able to and it also picks the correct ones even without -M:

bash-4.1$ dirac-proxy-init -g gridpp_user Generating proxy... Enter Certificate password: Added VOMS attribute /gridpp Uploading proxy for gridpp_user... Proxy generated: subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy/CN=proxy issuer : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy identity : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater timeleft : 23:53:59 DIRAC group : gridpp_user path : /tmp/x509up_u34811 username : mark.slater properties : NormalUser VOMS : True VOMS fqan : ['/gridpp']

Proxies uploaded: DN | Group | Until (GMT) /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater | na62.vo.gridpp.ac.uk_user | 2016/05/26 13:17 /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater | gridpp_user | 2016/05/26 13:17 bash-4.1$ voms-proxy-info --all subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy/CN=proxy issuer : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy identity : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy type : proxy strength : 1024 bits path : /tmp/x509up_u34811 timeleft : 23:53:36 key usage : Digital Signature, Key Encipherment, Data Encipherment === VO gridpp extension information === VO : gridpp subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater issuer : /C=UK/O=eScience/OU=Manchester/L=HEP/CN=voms.gridpp.ac.uk attribute : /gridpp/Role=NULL/Capability=NULL timeleft : 23:53:36 uri : voms.gridpp.ac.uk:15000 bash-4.1$ dirac-proxy-init -g na62.vo.gridpp.ac.uk_user Generating proxy... Enter Certificate password: Added VOMS attribute /na62.vo.gridpp.ac.uk Uploading proxy for na62.vo.gridpp.ac.uk_user... Proxy generated: subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy/CN=proxy issuer : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy identity : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater timeleft : 23:53:59 DIRAC group : na62.vo.gridpp.ac.uk_user path : /tmp/x509up_u34811 username : mark.slater properties : NormalUser VOMS : True VOMS fqan : ['/na62.vo.gridpp.ac.uk']

Proxies uploaded: DN | Group | Until (GMT) /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater | na62.vo.gridpp.ac.uk_user | 2016/05/26 13:17 /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater | gridpp_user | 2016/05/26 13:17 bash-4.1$ voms-proxy-info --all subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy/CN=proxy issuer : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy identity : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater/CN=proxy type : proxy strength : 1024 bits path : /tmp/x509up_u34811 timeleft : 23:53:37 key usage : Digital Signature, Key Encipherment, Data Encipherment === VO na62.vo.gridpp.ac.uk extension information === VO : na62.vo.gridpp.ac.uk subject : /C=UK/O=eScience/OU=Birmingham/L=ParticlePhysics/CN=mark slater issuer : /C=UK/O=eScience/OU=Imperial/L=Physics/CN=voms03.gridpp.ac.uk attribute : /na62.vo.gridpp.ac.uk/Role=NULL/Capability=NULL timeleft : 23:53:37 uri : voms03.gridpp.ac.uk:15501

I'll keep looking to see if I can see any other issues...

[drmarkwslater]

Hi Again!

@afortiorama Putting some debugging messages in shows that Ganga is running the command:

dirac-proxy-init -g gridpp_user -M -valid 24:00

and this works for me (as I say, even without the -M). What happens if you run this on the command line? Does it produce the voms extension correctly? Could you maybe send me your Dirac setup script? Maybe theres some differences there...

[afortiorama]

Hi Mark,

the problem wasn't with dirac-proxy-init but with what ganga runs. If you read the initial post it tells you what I did.

cheers alessandra

[drmarkwslater]

@afortiorama so looking at the specific output of your Ganga session it looks like it's not actually running dirac-proxy-init at all. In Ganga, it has the following:

Your identity: /C=UK/O=eScience/OU=Manchester/L=HEP/CN=alessandra forti Enter GRID pass phrase for this identity:

Which is different from the output of dirac-proxy-init:

Generating proxy... Enter Certificate password:

So could you send me your .gangarc (either attached to this or privately)? I suspect there's some setting that it making Ganga use grid-proxy-init instead.

[drmarkwslater]

Found the problem! It seems that python (or at least Ganga) doesn't like:

[group]param=value

instead of:

[group] param=value

I've updated the docs for the GridPP wiki and will check to see if this is a Python or Ganga limitation.

Thanks!

Mark