search

radical-cybertools / radical.saga

A Light-Weight Access Layer for Distributed Computing Infrastructure and Reference Implementation of the SAGA Python Language Bindings.
http://radical-cybertools.github.io/saga-python/
Other
83 stars 34 forks source link

Error with ec2 example #204

Closed ashleyz closed 11 years ago

ashleyz commented 11 years ago

Hello!

Not sure what is causing the below with the ec2 example invoked from (saga-python-env)ashley@effort:~/saga-python$ python examples/resource/ec2.py) on effort (rock physics machine)

2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.myproxy
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.myproxy for saga.Context API with URL scheme(s) ['myproxy://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.x509
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.x509 for saga.Context API with URL scheme(s) ['x509://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.ssh
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.ssh for saga.Context API with URL scheme(s) ['ssh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.context.userpass
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.context.userpass for saga.Context API with URL scheme(s) ['userpass://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.shell.shell_job
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_job for saga.job.Service API with URL scheme(s) ['fork://', 'local://', 'ssh://', 'gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_job for saga.job.Job API with URL scheme(s) ['fork://', 'local://', 'ssh://', 'gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.shell.shell_file
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.namespace.Directory API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://', 'ssh://', 'gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.namespace.Entry API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://', 'ssh://', 'gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.filesystem.Directory API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://', 'ssh://', 'gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.shell.shell_file for saga.filesystem.File API with URL scheme(s) ['file://', 'local://', 'sftp://', 'gsiftp://', 'ssh://', 'gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.sge.sgejob
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.sge.sgejob for saga.job.Service API with URL scheme(s) ['sge://', 'sge+ssh://', 'sge+gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.sge.sgejob for saga.job.Job API with URL scheme(s) ['sge://', 'sge+ssh://', 'sge+gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.pbs.pbsjob
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.pbs.pbsjob for saga.job.Service API with URL scheme(s) ['pbs://', 'pbs+ssh://', 'pbs+gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.pbs.pbsjob for saga.job.Job API with URL scheme(s) ['pbs://', 'pbs+ssh://', 'pbs+gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.condor.condorjob
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.condor.condorjob for saga.job.Service API with URL scheme(s) ['condor://', 'condor+ssh://', 'condor+gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.condor.condorjob for saga.job.Job API with URL scheme(s) ['condor://', 'condor+ssh://', 'condor+gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.slurm.slurm_job
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.slurm.slurm_job for saga.job.Service API with URL scheme(s) ['slurm://', 'slurm+ssh://', 'slurm+gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.slurm.slurm_job for saga.job.Job API with URL scheme(s) ['slurm://', 'slurm+ssh://', 'slurm+gsissh://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.http.http_file
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.http.http_file for saga.namespace.Entry API with URL scheme(s) ['http://', 'https://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.http.http_file for saga.filesystem.File API with URL scheme(s) ['http://', 'https://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Loading  adaptor saga.adaptors.aws.ec2_resource
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.aws.ec2_resource for saga.Context API with URL scheme(s) ['ec2://', 'ec2_keypair://', 'openstack://', 'eucalyptus://', 'euca://', 'aws://', 'amazon://', 'http://', 'https://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.aws.ec2_resource for saga.resource.Manager API with URL scheme(s) ['ec2://', 'ec2_keypair://', 'openstack://', 'eucalyptus://', 'euca://', 'aws://', 'amazon://', 'http://', 'https://']
2013:07:16 10:26:33 139982509999872 saga.engine           : [INFO    ] Register adaptor saga.adaptors.aws.ec2_resource for saga.resource.Compute API with URL scheme(s) ['ec2://', 'ec2_keypair://', 'openstack://', 'eucalyptus://', 'euca://', 'aws://', 'amazon://', 'http://', 'https://']
2013:07:16 10:26:33 139982509999872 saga.Session          : [DEBUG   ] adding  context : {'Server' : 'http://149.165.146.135:8773/services/Eucalyptus', 'Type' : 'ec2', 'UserID' : 'HNUFUQSHAISICNDWBFRTI', 'UserKey' : 'PPUjb3CFGupxPcYZHMhkznuZY7M9OjstWXmLwYmK'}
2013:07:16 10:26:33 139982509999872 saga.Session          : [DEBUG   ] adding  context : {'Token' : 'effort', 'Type' : 'ec2_keypair', 'UserCert' : '/home/ashley/india/futuregrid_rsa', 'UserID' : 'ubuntu'}
2013:07:16 10:26:33 139982509999872 saga.Session          : [DEBUG   ] adding  context : {'Type' : 'ssh', 'UserCert' : '/home/ashley/india/futuregrid_rsa', 'UserID' : 'ubuntu', 'UserKey' : 'None', 'UserPass' : 'None'}
2013:07:16 10:26:33 139982509999872 ContextSSH            : [INFO    ] init SSH context for key  at '/home/ashley/india/futuregrid_rsa' done
2013:07:16 10:26:34 139982509999872 EC2Keypair            : [INFO    ] keypair check : 'NoneType' object has no attribute 'strip'
Traceback (most recent call last):
  File "examples/resource/ec2.py", line 191, in <module>
    s.contexts.append (c2)
  File "/home/ashley/saga-python-env/local/lib/python2.7/site-packages/saga_python-0.9.8-py2.7.egg/saga/session.py", line 60, in append
    raise saga.exceptions.BadParameter (msg)
saga.exceptions.BadParameter: BadParameter: Cannot add context, initialization failed (BadParameter: 'ec2_keypair' invalid: 'NoneType' object has no attribute 'strip')

My environment looks like:

declare -x EC2_ACCESS_KEY="censored"
declare -x EC2_ACCOUNT_NUMBER="censored"
declare -x EC2_CERT="/home/ashley/india/euca2-azebro1-dabbebcd-cert.pem"
declare -x EC2_JVM_ARGS="-Djavax.net.ssl.trustStore=/home/ashley/india/jssecacerts"
declare -x EC2_KEYPAIR="/home/ashley/india/futuregrid_rsa"
declare -x EC2_KEYPAIR_ID="effort"
declare -x EC2_PRIVATE_KEY="/home/ashley/india/euca2-azebro1-dabbebcd-pk.pem"
declare -x EC2_SECRET_KEY="censored"
declare -x EC2_URL="http://149.165.146.135:8773/services/Eucalyptus"
declare -x EC2_USER_ID="censored"

which is similar to the environment on a working machine (Sullivan)

declare -x EC2_ACCESS_KEY="censored"
declare -x EC2_KEYPAIR="/glusterfs/users/sjha-bigjob/azebrowski/.ssh/osdc_rsa"
declare -x EC2_KEYPAIR_ID="OSDC"
declare -x EC2_SECRET_KEY="censored"
declare -x EC2_URL="http://10.103.114.3:8773/services/Cloud"

Any ideas? :)

andre-merzky commented 11 years ago

Hi Ashley,

I don't really see what is going wrong. Your env settings look good. The exception comes from this call in the ec2 adaptor:

keypair = conn.ex_describe_keypairs (token)

so the only parameter from SAGA-Python is token which seems to be set correctly, and is recognized earlier:

2013:07:16 10:26:33 139982509999872 saga.Session          : [DEBUG   ] adding  context : {'Token' : 'effort', 'Type' : 'ec2_keypair', 'UserCert' : '/home/ashley/india/futuregrid_rsa', 'UserID' : 'ubuntu'}

Would you mind checking once more if the UserCert path is correct? If thst is the case, we need to debug libcloud I'm afraid. Not sure if I should ask you that, but would you mind to wlk through that callstack with a debugger to check on what string variable the lib barfs? No worries if you don't want to, really -- I'll try to reproduce it then, but will probably need some days to get to it...

Best, Andre.

ashleyz commented 11 years ago

The UserCert path is correct:

(saga-python-env)ashley@effort:~/saga-python$ ls /home/ashley/india/futuregrid_rsa                
/home/ashley/india/futuregrid_rsa
(saga-python-env)ashley@effort:~/saga-python$ cat /home/ashley/india/futuregrid_rsa
-----BEGIN RSA PRIVATE KEY-----
...  (key info goes here) ...

I will give the debugger a try...

andre-merzky commented 11 years ago

Thanks Ashley! :)

ashleyz commented 11 years ago

Here is the trace...

...
c2.user_cert = os.environ['EC2_KEYPAIR']
c2.user_id   = 'ubuntu'         # the user id on the target VM                                    

# we create a session for all SAGA interactions, and attach the respective                        # security contexts.  Those are now avail for all SAGA objects created in                         
# that session                                                                                    
s = saga.Session (False)  # FALSE: don't use any other (default) contexts                         
s.contexts.append (c1)
s.contexts.append (c2) <----------------- crashes here

Here are the pertinent variables:

c1: {'Server' : 'http://149.165.146.135:8773/services/Eucalyptus', 'Type': 'ec2', 'UserID': '(censored)', 'UserKey' : '(censored)'}

c2: {'Token' : 'effort', 'Type' 'ec2_keypair', 'UserCert':/home/ashley/india/futuregrid_rsa', 'UserID' : 'ubuntu'}

Also session.py:57 is the endpoint at where the error is generated, but it has the information from c2 above...

ashleyz commented 11 years ago

I think this has been fixed on OSDC side... will reopen if needed.