Closed GoogleCodeExporter closed 9 years ago
Hi Mohammed
either your gc3pie.conf is incomplete or is erroneous.
the ec2 credentials should be specified in a dedicated
[auth/ ] section (e.g. [auth/aws] )
and then the specific resource should indicate what 'auth' to use:
in your case in
[resource/ec2] you should have (according to the example above)
auth=aws
keep also in mind that you need to specify the units for memory and walltime
so, for example:
max_memory_per_core = 1GB
max_walltime = 8 hours
Sergio
Original comment by sergio.m...@gmail.com
on 22 Jul 2014 at 8:12
Hi Sergio,
Thanks for your response. I corrected the config file and now I got an other
error message that I think it is related to keypair:
I tried 2 tests:
1. created a keypair 'metapiga' using AWS web interface and download it to may
local home. Then I extracted 'metapiga.pub' using 'openssh rsa ...' -> I got a
error message telling that the fingerprint does not match.
2. I removed the 'metapiga' from my local machine and AWS web, and kept only
'metapiga.pub'. I got this error message :
gc3.gc3libs: INFO: Computational resource 'amazon-ec2' initialized successfully.
gc3.gc3libs: DEBUG: Checking auth: EC2Auth
gc3.gc3libs: INFO: Creating main sequential task
gc3.gc3utils: DEBUG: Added task 'MetaPIGA_ParallelWK' to session.
gc3.gc3libs: DEBUG: Submitting MetaPIGATask.5 ...
gc3.gc3libs: DEBUG: Performing brokering ...
gc3.gc3libs: DEBUG: Checking resource 'amazon-ec2' for compatibility with
application requirements
gc3.gc3libs: DEBUG: Application scheduler returned 1 matching resources
gc3.gc3libs: DEBUG: Attempting submission to resource 'amazon-ec2'...
gc3.gc3libs: INFO: Keypair `metapiga` not found: creating it using public key
`/home/mohamed/.ssh/metapiga.pub`
gc3.gc3libs: INFO: Error in submitting job to resource 'amazon-ec2': TypeError:
not enough arguments for format string
Traceback (most recent call last):
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 242, in __submit_application
lrms.submit_job(app)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 811, in submit_job
user_data=user_data)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 403, in _create_instance
self._import_keypair()
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 515, in _import_keypair
% self.keypair_name, ex)
TypeError: not enough arguments for format string
Thanks,
Mohamed
Original comment by mohamed....@gmail.com
on 22 Jul 2014 at 10:32
Dear Mohamed,
the line numbers in the traceback do not match the current sources.
What version of GC3Pie are you using?
Thanks,
Riccardo
Original comment by riccardo.murri@gmail.com
on 22 Jul 2014 at 10:39
Dear Riccardo,
I used the command line to install it
sh -c "$(wget -O- http://gc3pie.googlecode.com/svn/install.sh)"
Thanks,
Mohamed
Original comment by mohamed....@gmail.com
on 22 Jul 2014 at 10:43
Dear Mohamed,
apparently that bug has already been fixed, but the fixed version was never
released.
I have now released the up-to-date sources, can you please install GC3Pie again
or upgrade, and then retry?
Thanks,
Riccardo
Original comment by riccardo.murri@gmail.com
on 22 Jul 2014 at 11:06
Dear Recardo,
Thanks for this fix. I re-installed the GC3Pie and the error message changed.
For the keypair. I generate it from AWS web interface and download it to may
local machine (I remove the .pem extension). Then I extract the public key
using:
openssl rsa -in ~/.ssh/metapiga -pubout -out ~/.ssh/metapiga.pub
...
gc3.gc3libs: INFO: Computational resource 'amazon-ec2' initialized successfully.
gc3.gc3libs: DEBUG: Checking auth: EC2Auth
gc3.gc3utils: WARNING: Session already exists, some command-line arguments
might be ignored.
gc3.gc3libs: DEBUG: Submitting MetaPIGATask.11 ...
gc3.gc3libs: DEBUG: Performing brokering ...
gc3.gc3libs: DEBUG: Checking resource 'amazon-ec2' for compatibility with
application requirements
gc3.gc3libs: DEBUG: Application scheduler returned 1 matching resources
gc3.gc3libs: DEBUG: Attempting submission to resource 'amazon-ec2'...
gc3.gc3libs: DEBUG: File `/home/mohamed/.ssh/metapiga` is not a valid DSS
private key: not a valid DSA private key file
gc3.gc3libs: ERROR: Keypair `metapiga` is present but has different
fingerprint: 51:42:b5:9f:81:75:65:71:84:5a:8d:37:0f:f2:05:34 !=
d5:92:e0:dc:72:5b:42:ab:64:0c:dd:20:cc:17:48:10:e4:45:bd:0f. Aborting!
gc3.gc3libs: INFO: Error in submitting job to resource 'amazon-ec2':
UnrecoverableError: Keypair `metapiga` is present but has different
fingerprint: 51:42:b5:9f:81:75:65:71:84:5a:8d:37:0f:f2:05:34 !=
d5:92:e0:dc:72:5b:42:ab:64:0c:dd:20:cc:17:48:10:e4:45:bd:0f. Aborting!
Traceback (most recent call last):
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 242, in __submit_application
lrms.submit_job(app)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 811, in submit_job
user_data=user_data)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 452, in _create_instance
keypairs[self.keypair_name].fingerprint))
UnrecoverableError: Keypair `metapiga` is present but has different
fingerprint: 51:42:b5:9f:81:75:65:71:84:5a:8d:37:0f:f2:05:34 !=
d5:92:e0:dc:72:5b:42:ab:64:0c:dd:20:cc:17:48:10:e4:45:bd:0f. Aborting!
...
Original comment by mohamed....@gmail.com
on 22 Jul 2014 at 11:25
Dear Mohammed,
The `openssl` command is not suitable for converting SSH
private/public keys; the `ssh-keygen` command should be used instead.
Can you please extract your private and public keys according to steps
1.-4. of this:
http://www.question-defense.com/2009/10/01/convert-amazon-web-services-private-r
sa-key-for-use-with-securecrt
You might want to substitute `amazon-ec2-key` with `metapiga` to match
the filenames you have already created.
Once you have extracted the keys according to the new procedure, can
you please retry and let us know?
Original comment by riccardo.murri@gmail.com
on 22 Jul 2014 at 11:42
Using the AWS generated keypair gave the fingerprint error message.
So I generated locally my keypair using ssh-keygen and upload the public key on
AWS. Then, after launching the program, a virtual machine was created on AWS
web interface.
Is that how the output message should look if the submission is ok?
Thanks,
Mohamed
gc3.gc3libs: INFO: Computational resource 'amazon-ec2' initialized successfully.
gc3.gc3libs: DEBUG: Checking auth: EC2Auth
gc3.gc3utils: WARNING: Session already exists, some command-line arguments
might be ignored.
gc3.gc3libs: DEBUG: Submitting MetaPIGATask.11 ...
gc3.gc3libs: DEBUG: Performing brokering ...
gc3.gc3libs: DEBUG: Checking resource 'amazon-ec2' for compatibility with
application requirements
gc3.gc3libs: DEBUG: Application scheduler returned 1 matching resources
gc3.gc3libs: DEBUG: Attempting submission to resource 'amazon-ec2'...
gc3.gc3libs: INFO: Keypair `metapiga` not found: creating it using public key
`/home/mohamed/.ssh/metapiga.pub`
gc3.gc3libs: INFO: Successfully imported key `metapiga` with fingerprint
`90:8d:a3:55:7a:e0:54:e7:5a:4f:c3:9a:f6:db:81:f2` as keypair `metapiga`
gc3.gc3libs: INFO: Creating security group metpiga_ssh
gc3.gc3libs: DEBUG: Adding rule {'to_port': 22, 'from_port': 22, 'ip_protocol':
'tcp', 'cidr_ip': '0.0.0.0/0'} to security group metpiga_ssh.
gc3.gc3libs: DEBUG: Adding rule {'to_port': -1, 'from_port': -1, 'ip_protocol':
'icmp', 'cidr_ip': '0.0.0.0/0'} to security group metpiga_ssh.
gc3.gc3libs: DEBUG: Create new VM using image id `ami-9d996bea`
gc3.gc3libs: INFO: VM with id `i-c8c64788` has been created and is in pending
state.
gc3.gc3libs: INFO: No available resource was found, but some VM is still in
`pending` state. Waiting until the next iteration before creating a new VM.
Pending VM ids: i-c8c64788
gc3.gc3libs: INFO: Submission of job MetaPIGATask.11 delayed
gc3.gc3libs: ERROR: Ignored error in submitting task 'MetaPIGATask.11':
LRMSSkipSubmissionToNextIteration: Delaying submission until some of the VMs
currently pending is ready. Pending VM ids: i-c8c64788
gc3.gc3libs: DEBUG: Ignored error in submitting task 'MetaPIGATask.11':
LRMSSkipSubmissionToNextIteration: Delaying submission until some of the VMs
currently pending is ready. Pending VM ids: i-c8c64788
Traceback (most recent call last):
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 969, in progress
self._core.submit(task)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 168, in submit
return self.__submit_application(app, resubmit, **extra_args)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 271, in __submit_application
raise ex
LRMSSkipSubmissionToNextIteration: Delaying submission until some of the VMs
currently pending is ready. Pending VM ids: i-c8c64788
gc3.gc3libs: DEBUG: Submitting MetaPIGATask.12 ...
gc3.gc3libs: DEBUG: Performing brokering ...
gc3.gc3libs: DEBUG: Checking resource 'amazon-ec2' for compatibility with
application requirements
gc3.gc3libs: DEBUG: Application scheduler returned 1 matching resources
gc3.gc3libs: DEBUG: Attempting submission to resource 'amazon-ec2'...
gc3.gc3libs: INFO: No available resource was found, but some VM is still in
`pending` state. Waiting until the next iteration before creating a new VM.
Pending VM ids: i-c8c64788
gc3.gc3libs: INFO: Submission of job MetaPIGATask.12 delayed
gc3.gc3libs: ERROR: Ignored error in submitting task 'MetaPIGATask.12':
LRMSSkipSubmissionToNextIteration: Delaying submission until some of the VMs
currently pending is ready. Pending VM ids: i-c8c64788
gc3.gc3libs: DEBUG: Ignored error in submitting task 'MetaPIGATask.12':
LRMSSkipSubmissionToNextIteration: Delaying submission until some of the VMs
currently pending is ready. Pending VM ids: i-c8c64788
Traceback (most recent call last):
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 969, in progress
self._core.submit(task)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 168, in submit
return self.__submit_application(app, resubmit, **extra_args)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 271, in __submit_application
raise ex
LRMSSkipSubmissionToNextIteration: Delaying submission until some of the VMs
currently pending is ready. Pending VM ids: i-c8c64788
gc3.gc3libs: INFO: Closing connection to cloud 'amazon-ec2'...
Starting metapiga; use the '-v' command-line option to get a more verbose
report of activity.
Status of jobs in the 'metapiga' session: (at 15:15:25, 07/22/14)
NEW 3/3 (100.0%)
RUNNING 0/3 (0.0%)
STOPPED 0/3 (0.0%)
SUBMITTED 0/3 (0.0%)
TERMINATED 0/3 (0.0%)
TERMINATING 0/3 (0.0%)
UNKNOWN 0/3 (0.0%)
total 3/3 (100.0%)
Original comment by mohamed....@gmail.com
on 22 Jul 2014 at 1:25
Yes, that is how a successful submission log looks like.
To cross check: after a while, the command should show that 3 tasks are in
RUNNING state.
Can we consider the issue solved?
Original comment by riccardo.murri@gmail.com
on 22 Jul 2014 at 1:32
After a while, I got an error message. Sorry for that.
I relaunched the same command, and I got an error message gc3.gc3utils:
CRITICAL: KeyError: 'ubuntu'. I can manually log-in the instance using
ssh-client.
gc3.gc3libs: DEBUG: Creating resource
'ec2-54-73-28-204.eu-west-1.compute.amazonaws.com@amazon-ec2' defined by:
architecture=set(['x86_64']), auth='ubuntu', enabled=True,
frontend=u'ec2-54-73-28-204.eu-west-1.compute.amazonaws.com',
ignore_ssh_host_keys=True, keyfile='/home/mohamed/.ssh/metapiga', max_cores=30,
max_cores_per_job=2, max_memory_per_core=Memory(2, unit=GB),
max_walltime=Duration(8, unit=h), metapiga_image_id='ami-9d996bea',
metapiga_instance_type='m1.small',
name=u'ec2-54-73-28-204.eu-west-1.compute.amazonaws.com@amazon-ec2',
override='True', security_group_name='metpiga_ssh',
security_group_rules='tcp:22:22:0.0.0.0/0, icmp:-1:-1:0.0.0.0/0',
transport='ssh', type='shellcmd', vm_auth='ubuntu'.
gc3.gc3libs: INFO: Computational resource
'ec2-54-73-28-204.eu-west-1.compute.amazonaws.com@amazon-ec2' initialized
successfully.
gc3.gc3libs: ERROR: Could not create resource
'ec2-54-73-28-204.eu-west-1.compute.amazonaws.com@amazon-ec2': 'ubuntu'.
Configuration file problem?
gc3.gc3utils: CRITICAL: KeyError: 'ubuntu'
Traceback (most recent call last):
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/cmdline.py", line 437, in run
return cli.app.CommandLineApp.run(self)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/cli/app.py", line 245, in run
return self.post_run(returned)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/cli/app.py", line 241, in run
returned = self.main(*args)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/cmdline.py", line 1382, in _main
self._controller.close()
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 1177, in close
self._core.close()
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 642, in close
lrms.close()
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 900, in close
self.get_resource_status()
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 701, in get_resource_status
resource = self._get_remote_resource(vm)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 482, in _get_remote_resource
self.resources[vm.id] = self._make_resource(vm.public_dns_name)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 539, in _make_resource
resource = cfg._make_resource(args)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/config.py", line 600, in _make_resource
return cls(**dict(resdict))
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/shellcmd.py", line 199, in __init__
auth = self._auth_fn()
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/config.py", line 465, in <lambda>
return (lambda **extra_args: self.auth_factory.get(name, **extra_args))
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/authentication/__init__.py", line 111, in get
a = self._ctors[auth_name](**dict(params))
KeyError: 'ubuntu'
Starting metapiga; use the '-v' command-line option to get a more verbose
report of activity.
Status of jobs in the 'metapiga' session: (at 15:35:21, 07/22/14)
NEW 3/3 (100.0%)
RUNNING 0/3 (0.0%)
STOPPED 0/3 (0.0%)
SUBMITTED 0/3 (0.0%)
TERMINATED 0/3 (0.0%)
TERMINATING 0/3 (0.0%)
UNKNOWN 0/3 (0.0%)
Original comment by mohamed....@gmail.com
on 22 Jul 2014 at 1:38
If you have `vm_auth=ubuntu` in your `gc3pie.conf` configuration file, then you
should have a matching "auth" section:
[auth/ssh_gc3user]
# Default user on "Hobbes" VM images is `gc3-user`
type=ssh
username=gc3-user
Can you please try adding this and see if it solves the problem?
Original comment by riccardo.murri@gmail.com
on 22 Jul 2014 at 1:50
I re-installed the gc3pie environment.
I run my application: $> python metapiga.py -v
/home/mohamed/VENUSC/demo/metapiga/metapiga_input.tar
Status of jobs in the 'metapiga' session: (at 17:05:38, 07/22/14)
NEW 3/3 (100.0%)
RUNNING 0/3 (0.0%)
STOPPED 0/3 (0.0%)
SUBMITTED 0/3 (0.0%)
TERMINATED 0/3 (0.0%)
TERMINATING 0/3 (0.0%)
UNKNOWN 0/3 (0.0%)
total 3/3 (100.0%)
I wait until the virtual machine is started on AWS interface.
I re-launch the command: $> python metapiga.py -v
/home/mohamed/VENUSC/demo/metapiga/metapiga_input.tar
I got this message.
gc3.gc3utils: INFO: Starting metapiga at Tue Jul 22 17:05:16 2014; invoked as
'metapiga.py -v /home/mohamed/VENUSC/demo/metapiga/metapiga_input.tar'
gc3.gc3libs: INFO: Computational resource 'amazon-ec2' initialized successfully.
gc3.gc3utils: WARNING: Session already exists, some command-line arguments
might be ignored.
gc3.gc3libs: INFO: Creating remote ShellcmdLrms resource for ip
ec2-54-195-126-166.eu-west-1.compute.amazonaws.com
gc3.gc3libs: INFO: Computational resource
'ec2-54-195-126-166.eu-west-1.compute.amazonaws.com@amazon-ec2' initialized
successfully.
gc3.gc3libs: INFO: Ignoring ssh host key file.
gc3.gc3libs: INFO: creating resource file folder: /home/ubuntu/.gc3/shellcmd.d
...
gc3.gc3libs: WARNING: `max_cores` value on resource
ec2-54-195-126-166.eu-west-1.compute.amazonaws.com@amazon-ec2 mismatch:
configurationfile says `30` while it's actually `1`. Updating current value.
gc3.gc3libs: WARNING: `max_memory_per_core` value on resource
ec2-54-195-126-166.eu-west-1.compute.amazonaws.com@amazon-ec2 mismatch:
configuration file says `2GB` while it's actually `1.69132e+06KiB`. Updating
current value.
gc3.gc3libs: ERROR: Keypair `metapiga` is present but has different
fingerprint: 3b:80:f6:a4:90:3e:4b:1c:16:ef:79:67:f0:47:8b:02 !=
99:15:ec:98:21:09:7a:c4:a3:a9:73:87:4d:91:b2:dd. Aborting!
gc3.gc3libs: INFO: Error in submitting job to resource 'amazon-ec2':
UnrecoverableError: Keypair `metapiga` is present but has different
fingerprint: 3b:80:f6:a4:90:3e:4b:1c:16:ef:79:67:f0:47:8b:02 !=
99:15:ec:98:21:09:7a:c4:a3:a9:73:87:4d:91:b2:dd. Aborting!
Traceback (most recent call last):
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/core.py", line 242, in __submit_application
lrms.submit_job(app)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 811, in submit_job
user_data=user_data)
File "/home/mohamed/gc3pie/local/lib/python2.7/site-packages/gc3libs/backends/ec2.py", line 452, in _create_instance
keypairs[self.keypair_name].fingerprint))
Original comment by mohamed....@gmail.com
on 22 Jul 2014 at 3:08
Please find attached the log file of an execution. It seems that there is still
an issue with the public key fingerprint.
Thanks.
Mohamed
Original comment by mohamed....@gmail.com
on 23 Jul 2014 at 9:15
Attachments:
Dear Mohamed,
I believe this is a configuration issue: somehow the SSH key injected
into the virtual machine is not the one GC3Pie is using to connect to
it. Your earlier reports and logs list 5 different key fingerprints,
whereas there should be only one!
So, can you please post the output of the following commands:
1) ls -l ~/.ssh/
2) for each `.pub` file in the above output, run the following
command::
ssh-keygen ~/.ssh/file.pub
(of course, substitute "file.pub" with the actual file name)
Thanks,
Riccardo
Original comment by riccardo.murri@gmail.com
on 23 Jul 2014 at 4:23
Dear Riccardo,
Thank you for your response.
Yes, It seems to me that there is a format problem in calculating the
fingerprint. What I did is:
1. remove all public private key from my ~/.ssh and from AWS-EC2 web portal.
2. generate a keypair using AWS-EC2 web portal and download it into
~/.ssh/metapiga
3. run the application: python metapiga.py metapiga_input.tar -vv -C 25
So, on the AWS-EC2 web portal, one VM is started. If I take a look on the
uploaded publickey on AWS-EC2, I find the value
'1a:5e:92:4a:cd:8e:32:32:cd:65:da:59:90:14:45:06'.
On my machine, I run these two commands:
$ ssh-keygen -lf ~/.ssh/metapiga
2048 0c:a9:83:3f:99:ae:5c:2e:49:19:5c:84:74:4b:5f:d1
/home/benbelgacem/.ssh/metapiga.pub (RSA)
$ openssl pkey -in ~/.ssh/metapiga -pubout -outform DER | openssl md5 -c
(stdin)= 1a:5e:92:4a:cd:8e:32:32:cd:65:da:59:90:14:45:06
In the log file of the submission, I got this error message:
...
gc3.gc3libs: DEBUG: Ignoring error while submit to resource
ec2-54-75-64-207.eu-west-1.compute.amazonaws.com@amazon-ec2: Resource
ec2-54-75-64-207.eu-west-1
.compute.amazonaws.com@amazon-ec2 does not have enough available memory:
1.73192e+09B < 2GB..
gc3.gc3libs: DEBUG: File `/home/benbelgacem/.ssh/metapiga` is not a valid DSS
private key: not a valid DSA private key file
gc3.gc3libs: ERROR: Keypair `metapiga` is present but has different
fingerprint: 0c:a9:83:3f:99:ae:5c:2e:49:19:5c:84:74:4b:5f:d1 !=
1a:5e:92:4a:cd:8e:32:32:
cd:65:da:59:90:14:45:06. Aborting!
gc3.gc3libs: INFO: Error in submitting job to resource 'amazon-ec2':
UnrecoverableError: Keypair `metapiga` is present but has different
fingerprint: 0c:a9:
83:3f:99:ae:5c:2e:49:19:5c:84:74:4b:5f:d1 !=
1a:5e:92:4a:cd:8e:32:32:cd:65:da:59:90:14:45:06. Aborting!
...
For me, it seems like the gc3Pie uses the SHA1 algorithm and AWS-EC2 the MD5
algorithm to compute the fingerprint.
What do you think ?
Thanks,
Mohamed
Original comment by mohamed....@gmail.com
on 23 Jul 2014 at 7:08
Dear Mohamed,
The logs you posted so far show no issue with the GC3Pie script or
application class. Rather, there is evidently an authentication
problem which probably originates into different key pairs being used
by Amazon AWS and your local GC3Pie installation.
However, GC3Pie is able to *upload* its SSH key to Amazon, provided
the keypair does not exist on Amazon already.
So can you please make another attempt, following the steps I write here?
1. Create a new keypair on the computer where you run GC3Pie::
ssh-keygen -f rsa -C "gc3pie" -f ~/.ssh/gc3pie
Please make sure that you type the `-t rsa` option; Amazon AWS does
not accept DSA keys, and your configuration file states that your
SSH key is called `myec2_dsa.pub` which I guess is a DSA key :-)
This should create *two* files:
~/.ssh/gc3pie
~/.ssh/gc3pie.pub
2. Modify your `~/.gc3/gc3pie.conf` configuration file to use this new keypair:
(a) Replace `keypair_name=myec2_dsa` on line 19 with: `keypair_name=gc3pie`
(b) Replace `public_key=~/.ssh/myec2_dsa.pub` on line 24 with: `public_key=~/.ssh/gc3pie.pub`
3. Now try to re-run your GC3Pie driver script, with max debugging level
(`-vvvv`)
Thanks,
Riccardo
Original comment by riccardo.murri@gmail.com
on 24 Jul 2014 at 12:45
Dear Riccardo,
Now things seem better. I generated the keypair as you described: if the
'vm_pool_max_size'=1 in my `~/.gc3/gc3pie.conf` , then the creation of a VM
succeeds and the output file of my job is downloaded on my local machine.
However, If 'vm_pool_max_size' is greater than 1, then the error of
fingerprint appears. So, I commented the lines 447-452 in the file 'ec2.py'
and re-launched the submission of a parallel workflow of 10 jobs with
'vm_pool_max_size=5'
(see my gc3pie.conf content below) -> it works :)
I would like please to check the meaning of :
- 'max_cores = N': means that we can use at max N cores per/VM?
- 'vm_pool_max_size=x': means that the number of running VMs should be <=
x ?
My application is a ParallelTaskCollection based workflow which is a kind
of Map/reduce workflow: it submits m independent jobs on AWS and retrieve
their output files.
During execution, I remarked that the application (the script session):
- spawns the x virtual machines in sequential ( where, vm_pool_max_size=x),
i.e., its starts the first Vn and wait until it is available, then, it
start the second one ...
- if the job is finished, it shutdowns its corresponding VM and starts a
new VM for the next job
Is it possible to starts x virtual machines in parallel? how?
Is there any way to re-use VMs without shutting them down during the same
computation?
Many thanks !!
Mohamed
[auth/aws]
ec2_access_key=AAAAAAAAAAAAAAA
ec2_secret_key=BBBBBBBBBBBBBBBBBBBB
type=ec2
[auth/sshaws]
type=ssh
username=ubuntu
[resource/amazon-ec2]
enabled=true
auth=aws
ec2_url=https://ec2.eu-west-1.amazonaws.com
ec2_region=eu-west-1
type=ec2+shellcmd
max_cores_per_job = 1
max_memory_per_core =1 GB
max_walltime = 8
max_cores = 1
architecture = x86_64
keypair_name=gc3pie
public_key=~/.ssh/gc3pie.pub
vm_auth=sshaws
instance_type=m1.small
image_id=ami-9d996bea
security_group_name=metpiga_ssh
security_group_rules=tcp:22:22:0.0.0.0/0, icmp:-1:-1:0.0.0.0/0
vm_pool_max_size = 5
#user_data=#!/bin/bash
# command -v java || (sudo apt-get update ; sudo apt-get install -y
openjdk-6-jre-headless)
2014-07-24 14:45 GMT+02:00 <gc3pie@googlecode.com>:
Original comment by mohamed....@gmail.com
on 25 Jul 2014 at 10:18
[deleted comment]
Dear Mohamed,
Thank you for your patience and for replaying the steps; it seems we're
approaching
regarding the issue of fingerprints:
> However, If 'vm_pool_max_size' is greater than 1, then the error of
> fingerprint appears. So, I commented the lines 447-452 in the file 'ec2.py'
> and re-launched the submission of a parallel workflow of 10 jobs with
> 'vm_pool_max_size=5'
> (see my gc3pie.conf content below) -> it works :)
Ok, then it seems we have a real issue here: AWS EC2 and Paramiko (the
Python library for doing SSH connections) are computing the key
fingerprint in different ways, which we cannot compare.
I'll add some code to disable the check.
> I would like please to check the meaning of :
> - 'max_cores = N': means that we can use at max N cores per/VM?
> - 'vm_pool_max_size=x': means that the number of running VMs should be <=
> x ?
> Is it possible to starts x virtual machines in parallel? how?
That's not currently supported. The issue is that the AWS EC2 API
does not provide details on the VM features (how many cores, GBs of
RAM, etc.) that correspond to a certain "flavor" (e.g., "m1.small").
So, GC3Pie has to launch a VM, wait for it to be networked, log in and
find out. It could well be that the VM has enough cores to launch all
the pending jobs, or maybe we need to spawn another VM...
So, if you are interested in job throughput, use a larger flavor.
Any ideas on how to handle this is welcome, but please open another
issue ticket if you want to discuss this issue further -- let's not
intertwine discussion threads.
> Is there any way to re-use VMs without shutting them down during the same
> computation?
That should be done automatically already (but maybe not in the 2.1.x
release -- I need to check).
Thanks,
Riccardo
Original comment by riccardo.murri@gmail.com
on 25 Jul 2014 at 1:28
Thanks a lot for you response and help!
Original comment by mohamed....@gmail.com
on 25 Jul 2014 at 3:11
Dear Mohamed,
I have just released GC3Pie 2.1.5, which should contain a fix for the digest
incompatibility issue.
Can you please test it?
Thanks,
Riccardo
Original comment by riccardo.murri@gmail.com
on 25 Jul 2014 at 5:45
Dear Riccardo,
Thanks for the fix. I tested the release 2.1.5 and it works :)
Bests,
Mohamed
2014-07-25 19:45 GMT+02:00 <gc3pie@googlecode.com>:
Original comment by mohamed....@gmail.com
on 26 Jul 2014 at 10:57
Fixed in SVN r3957 for the 2.1.x series, and r3965-r3966 on the "trunk"
development line (shortly thereafter released as 2.2).
Original comment by riccardo.murri@gmail.com
on 29 Jul 2014 at 12:08
Original issue reported on code.google.com by
mohamed....@gmail.com
on 22 Jul 2014 at 8:01Attachments: