Closed GoogleCodeExporter closed 9 years ago
| Ability to start more than 8*14GB VMS for 256GB quota. Only 8 will start
Is this issue still valid? I currently see 13 started instances on
Hobbes, each with 14GB of RAM.
Original comment by riccardo.murri@gmail.com
on 24 Jul 2013 at 10:14
Original comment by riccardo.murri@gmail.com
on 24 Jul 2013 at 10:16
Hi Riccardo,
I believe so - Tyanko has made a workaround by increasing quota to
around 500GB to allow the full 256GB to be exploited. i.e. actual quota
values are not interpreted correctly.
Cheers,
Joel
Original comment by joelfiddes
on 24 Jul 2013 at 10:21
Hi Joel,
| I believe so - Tyanko has made a workaround by increasing quota to
| around 500GB to allow the full 256GB to be exploited. i.e. actual quota
| values are not interpreted correctly.
If this is the case, you should not be able to start more than 16 VMs
now. Can you confirm?
Original comment by riccardo.murri@gmail.com
on 24 Jul 2013 at 10:28
Hi Riccardo,
Just launched another 10 VM session to test this, but prevented by VM
limit =15
1 | NEW | Submission failed: RecoverableError: Already running the
maximum number of VM on resource hobbes: 15 >= 15. at Wed Jul 24
13:59:09 2013 |
J
Original comment by joelfiddes
on 24 Jul 2013 at 12:01
Hi Joel,
| Just launched another 10 VM session to test this, but prevented by VM
| limit =15
|
| 1 | NEW | Submission failed: RecoverableError: Already running the
| maximum number of VM on resource hobbes: 15 >= 15. at Wed Jul 24
| 13:59:09 2013 |
This depends on your configuration file; please change the line:
vm_pool_max_size = 15
to a larger number, e.g.:
vm_pool_max_size = 30
Original comment by riccardo.murri@gmail.com
on 24 Jul 2013 at 12:36
Hi Riccardo,
20 VM's start now. gc3pie cyclically tries to start 21st but then gives
an error and removes.
gc3.gc3libs: ERROR: VM with id `i-000056ab` is in ERROR state.
Terminating it!
So 280GB of 500GB is used. This is enough capacity for me but still
interesting as quota should be 500GB. But now perhaps an ip issue?
| GTSubControllApplication.427 |
GTSC_nS12_Nc200_snow_depth.mm._01102006_01102011 | NEW | Submission
failed: ValueError: _make_resource: `remote_ip` must be a valid IP or
hostname. at Wed Jul 24 16:23:52 2013
Cheers,
Joel
Original comment by joelfiddes
on 24 Jul 2013 at 2:25
Hi Joel,
| 20 VM's start now. gc3pie cyclically tries to start 21st but then gives
| an error and removes.
| [...]
| So 280GB of 500GB is used. This is enough capacity for me but still
| interesting as quota should be 500GB. But now perhaps an ip issue?
Indeed, there was a limit on 20 IP addresses that you could allocate max.
I've now raised that limit to 80 VMs.
Please try again and let us know.
Original comment by riccardo.murri@gmail.com
on 24 Jul 2013 at 3:22
I think still an ip issue
gc3.gc3libs: ERROR: VM with id `i-0000583e` is in ERROR state.
Terminating it!
gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.464': ValueError: _make_resource: `remote_ip`
must be a valid IP or hostname.
I have still a limit of 20 VMs.
Cheers
Joel
Original comment by joelfiddes
on 25 Jul 2013 at 8:45
Hi Joel,
| I think still an ip issue
|
| gc3.gc3libs: ERROR: VM with id `i-0000583e` is in ERROR state.
| Terminating it!
| gc3.gc3libs: ERROR: Ignored error in submitting task
| 'GTSubControllApplication.464': ValueError: _make_resource: `remote_ip`
|
| must be a valid IP or hostname.
|
| I have still a limit of 20 VMs.
I've revised the quotas for the `geo.uzh` tenant; there was a
"Gigabytes" limit of 2000, which I take to be the limit on the
aggregate disk space used by all VMs; I've raised it to 8000GB, can
you please check if you are now able to start (up to) 80 VMs?
Original comment by riccardo.murri@gmail.com
on 25 Jul 2013 at 3:27
Hi Riccardo,
Have launched 49 VM's, but still capped at 20:
--------------------------------------------------------------------------------
+
gc3.gc3libs: ERROR: VM with id `i-00005902` is in ERROR state.
Terminating it!
gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.511': ValueError: _make_resource: `remote_ip`
must be a valid IP or hostname.
^Cgtsub_control: Exiting upon user request (Ctrl+C)
and
gc3.gc3libs: ERROR: VM with id `i-00005903` is in ERROR state.
Terminating it!
Cheers
Joel
Original comment by joelfiddes
on 26 Jul 2013 at 11:51
ip issue still remains I think.
Currently only possible to start 18 VM's *14GB RAM/100GB disk on 483GB RAM
resource (geo.uzh).
Error message when GC3Pie attempts to start 19th VM:
gc3.gc3libs: ERROR: VM with id `i-00005ff3` is in ERROR state. Terminating it!
gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.795': ValueError: _make_resource: `remote_ip` must be
a valid IP or hostname.
Cheers
Joel
Original comment by joelfiddes
on 8 Aug 2013 at 11:52
Hi Joel,
> gc3.gc3libs: ERROR: VM with id `i-00005ff3` is in ERROR state. Terminating it!
> gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.795': ValueError: _make_resource: `remote_ip` must be
a valid IP or hostname.
this is actually Issue 408 masking the real error message :-(.
We would need to fix that first so that we can see why the 19th VM
goes into ERROR state. If you have any log lines regarding the
lifecycle of the 19th VM, could you please post them?
Thanks,
Riccardo
Original comment by riccardo.murri@gmail.com
on 13 Aug 2013 at 8:57
Hi Riccardo,
inconsistent behaviour now - memory quota maxes out after 7 * 14GB VM's
launched. NOT getting IP error stated above.
Cheers
Joel
#============================================
# SHORT LINES
#============================================
gc3.gc3libs: WARNING: Option `public_key` in configuration file should contain
the path to a public key file (with `.pub` ending), but
'/home/joel/.ssh/joel.pem' was found instead. Continuing anyway.
gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.831': UnrecoverableError: Error starting instance:
EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-550b26b0-785f-46cc-902b-20eedb57249
1</RequestID></Response>
gc3.gc3libs: WARNING: Option `public_key` in configuration file should contain
the path to a public key file (with `.pub` ending), but
'/home/joel/.ssh/joel.pem' was found instead. Continuing anyway.
gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.830': UnrecoverableError: Error starting instance:
EC2ResponseError: 400 Bad Request
#============================================
# VERBOSE LINES
#============================================
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-6085ce32-a7fc-4e23-8385-183f2ed52a8
6</RequestID></Response>
Traceback (most recent call last):
File "/home/joel/gc3pie/src/gc3libs/core.py", line 242, in __submit_application
lrms.submit_job(app)
File "/home/joel/gc3pie/src/gc3libs/backends/ec2.py", line 870, in submit_job
user_data=user_data)
File "/home/joel/gc3pie/src/gc3libs/backends/ec2.py", line 452, in _create_instance
raise UnrecoverableError("Error starting instance: %s" % str(ex))
UnrecoverableError: Error starting instance: EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-6085ce32-a7fc-4e23-8385-183f2ed52a8
6</RequestID></Response>
gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.825': UnrecoverableError: Error starting instance:
EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-6085ce32-a7fc-4e23-8385-183f2ed52a8
6</RequestID></Response>
gc3.gc3libs: DEBUG: Ignored error in submitting task
'GTSubControllApplication.825': UnrecoverableError: Error starting instance:
EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-6085ce32-a7fc-4e23-8385-183f2ed52a8
6</RequestID></Response>
Traceback (most recent call last):
File "/home/joel/gc3pie/src/gc3libs/core.py", line 969, in progress
self._core.submit(task)
File "/home/joel/gc3pie/src/gc3libs/core.py", line 168, in submit
return self.__submit_application(app, resubmit, **extra_args)
File "/home/joel/gc3pie/src/gc3libs/core.py", line 271, in __submit_application
raise ex
UnrecoverableError: Error starting instance: EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-6085ce32-a7fc-4e23-8385-183f2ed52a8
6</RequestID></Response>
gc3.gc3libs: DEBUG: Submitting GTSubControllApplication.834 ...
gc3.gc3libs: DEBUG: Performing brokering ...
gc3.gc3libs: DEBUG: Checking resource 'hobbes' for compatibility with
application requirements
gc3.gc3libs: DEBUG: Application scheduler returned 1 matching resources
gc3.gc3libs: DEBUG: Attempting submission to resource 'hobbes'...
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1401
gc3.gc3libs: DEBUG: Reading resource file for pid 1401
gc3.gc3libs: DEBUG: Recovered resource information from files in
/home/gc3-user/.gc3/shellcmd.d: Available memory: 7.15593e+08B, used memory:
1.4e+10B
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1409
gc3.gc3libs: DEBUG: Reading resource file for pid 1409
gc3.gc3libs: DEBUG: Recovered resource information from files in
/home/gc3-user/.gc3/shellcmd.d: Available memory: 7.15593e+08B, used memory:
1.4e+10B
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1404
gc3.gc3libs: DEBUG: Reading resource file for pid 1404
gc3.gc3libs: DEBUG: Recovered resource information from files in
/home/gc3-user/.gc3/shellcmd.d: Available memory: 7.15593e+08B, used memory:
1.4e+10B
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1411
gc3.gc3libs: DEBUG: Reading resource file for pid 1411
gc3.gc3libs: DEBUG: Recovered resource information from files in
/home/gc3-user/.gc3/shellcmd.d: Available memory: 7.15593e+08B, used memory:
1.4e+10B
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1405
gc3.gc3libs: DEBUG: Reading resource file for pid 1405
gc3.gc3libs: DEBUG: Recovered resource information from files in
/home/gc3-user/.gc3/shellcmd.d: Available memory: 7.15593e+08B, used memory:
1.4e+10B
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1408
gc3.gc3libs: DEBUG: Reading resource file for pid 1408
gc3.gc3libs: DEBUG: Recovered resource information from files in
/home/gc3-user/.gc3/shellcmd.d: Available memory: 7.15593e+08B, used memory:
1.4e+10B
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1412
gc3.gc3libs: DEBUG: Reading resource file for pid 1412
gc3.gc3libs: DEBUG: Recovered resource information from files in
/home/gc3-user/.gc3/shellcmd.d: Available memory: 7.15593e+08B, used memory:
1.4e+10B
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1401
gc3.gc3libs: DEBUG: Reading resource file for pid 1401
gc3.gc3libs: DEBUG: Ignoring error while submit to resource
130.60.24.78@hobbes: Resource 130.60.24.78@hobbes does not have enough
available memory: 7.15593e+08B < 14GB..
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1409
gc3.gc3libs: DEBUG: Reading resource file for pid 1409
gc3.gc3libs: DEBUG: Ignoring error while submit to resource
130.60.24.77@hobbes: Resource 130.60.24.77@hobbes does not have enough
available memory: 7.15593e+08B < 14GB..
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1404
gc3.gc3libs: DEBUG: Reading resource file for pid 1404
gc3.gc3libs: DEBUG: Ignoring error while submit to resource
130.60.24.66@hobbes: Resource 130.60.24.66@hobbes does not have enough
available memory: 7.15593e+08B < 14GB..
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1411
gc3.gc3libs: DEBUG: Reading resource file for pid 1411
gc3.gc3libs: DEBUG: Ignoring error while submit to resource
130.60.24.46@hobbes: Resource 130.60.24.46@hobbes does not have enough
available memory: 7.15593e+08B < 14GB..
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1405
gc3.gc3libs: DEBUG: Reading resource file for pid 1405
gc3.gc3libs: DEBUG: Ignoring error while submit to resource
130.60.24.44@hobbes: Resource 130.60.24.44@hobbes does not have enough
available memory: 7.15593e+08B < 14GB..
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1408
gc3.gc3libs: DEBUG: Reading resource file for pid 1408
gc3.gc3libs: DEBUG: Ignoring error while submit to resource
130.60.24.75@hobbes: Resource 130.60.24.75@hobbes does not have enough
available memory: 7.15593e+08B < 14GB..
gc3.gc3libs: DEBUG: Checking status of the following PIDs: 1412
gc3.gc3libs: DEBUG: Reading resource file for pid 1412
gc3.gc3libs: DEBUG: Ignoring error while submit to resource
130.60.24.73@hobbes: Resource 130.60.24.73@hobbes does not have enough
available memory: 7.15593e+08B < 14GB..
gc3.gc3libs: DEBUG: Checking if keypair is registered in SSH agent...
gc3.gc3libs: WARNING: Option `public_key` in configuration file should contain
the path to a public key file (with `.pub` ending), but
'/home/joel/.ssh/joel.pem' was found instead. Continuing anyway.
gc3.gc3libs: DEBUG: Trying to load key file `/home/joel/.ssh/joel.pem` as DSS
key...
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 842, in emit
msg = self.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 719, in format
return fmt.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
record.message = record.getMessage()
File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file ec2.py, line 541
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 842, in emit
msg = self.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 719, in format
return fmt.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
record.message = record.getMessage()
File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
msg = msg % self.args
TypeError: not all arguments converted during string formatting
Logged from file ec2.py, line 541
gc3.gc3libs: DEBUG: Trying to load key file `/home/joel/.ssh/joel.pem` as RSA
key...
gc3.gc3libs: DEBUG: Create new VM using image id `ami-00000085`
gc3.gc3libs: INFO: Error in submitting job to resource 'hobbes':
UnrecoverableError: Error starting instance: EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-138b26dc-5a15-4952-a918-3efb51fdc29
5</RequestID></Response>
Traceback (most recent call last):
File "/home/joel/gc3pie/src/gc3libs/core.py", line 242, in __submit_application
lrms.submit_job(app)
File "/home/joel/gc3pie/src/gc3libs/backends/ec2.py", line 870, in submit_job
user_data=user_data)
File "/home/joel/gc3pie/src/gc3libs/backends/ec2.py", line 452, in _create_instance
raise UnrecoverableError("Error starting instance: %s" % str(ex))
UnrecoverableError: Error starting instance: EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-138b26dc-5a15-4952-a918-3efb51fdc29
5</RequestID></Response>
gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.834': UnrecoverableError: Error starting instance:
EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-138b26dc-5a15-4952-a918-3efb51fdc29
5</RequestID></Response>
gc3.gc3libs: DEBUG: Ignored error in submitting task
'GTSubControllApplication.834': UnrecoverableError: Error starting instance:
EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-138b26dc-5a15-4952-a918-3efb51fdc29
5</RequestID></Response>
Traceback (most recent call last):
File "/home/joel/gc3pie/src/gc3libs/core.py", line 969, in progress
self._core.submit(task)
File "/home/joel/gc3pie/src/gc3libs/core.py", line 168, in submit
return self.__submit_application(app, resubmit, **extra_args)
File "/home/joel/gc3pie/src/gc3libs/core.py", line 271, in __submit_application
raise ex
UnrecoverableError: Error starting instance: EC2ResponseError: 400 Bad Request
<?xml version="1.0"?>
<Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota exceeded
for ram: Requested 14336, but already used 509760 of 512000
ram</Message></Error></Errors><RequestID>req-138b26dc-5a15-4952-a918-3efb51fdc29
5</RequestID></Response>
Status of jobs in the 's10' session: (at 12:28:31, 08/13/13)
NEW 8/12 (66.7%)
RUNNING 4/12 (33.3%)
STOPPED 0/12 (0.0%)
SUBMITTED 0/12 (0.0%)
TERMINATED 0/12 (0.0%)
TERMINATING 0/12 (0.0%)
UNKNOWN 0/12 (0.0%)
total 12/12 (100.0%)
gc3.gc3utils: INFO: sleeping for 10 seconds...
Original comment by joelfiddes
on 13 Aug 2013 at 10:34
Hi Joel,
We think that this is actually not a gc3pie bug but it is related to
some issue with the quota reporting system of the Hobbes cloud.
We also think we "fixed" the issue by forcing an update of the actual
resource usage on the system, so please re-try now and let us know if
the issues is actually solved.
.a.
Original comment by antonio....@gmail.com
on 13 Aug 2013 at 12:35
Hi Joel,
| gc3.gc3libs: ERROR: Ignored error in submitting task
| 'GTSubControllApplication.831': UnrecoverableError: Error starting
instance: EC2ResponseError: 400 Bad Request
| <?xml version="1.0"?>
| <Response><Errors><Error><Code>TooManyInstances</Code><Message>Quota
| exceeded for ram: Requested 14336, but already used 509760 of 512000
|
ram</Message></Error></Errors><RequestID>req-550b26b0-785f-46cc-902b-20eedb57249
1</RequestID></Response>
Can you please update to the latest version of GC3Pie? I fixed the
reporting to get (hopefully) more readable log lines.
Original comment by riccardo.murri@gmail.com
on 13 Aug 2013 at 1:08
Hi Riccardo,
will that affect current running simulations? Whats the best method to
upgrade (follow documentation? ie uninstall/reinstall?)
Currently got 18VMs running with ip error stated again.
gc3.gc3libs: ERROR: Ignored error in submitting task
'GTSubControllApplication.874': ValueError: _make_resource: `remote_ip`
must be a valid IP or hostname.
Cheers
Joel
Original comment by joelfiddes
on 13 Aug 2013 at 1:30
Original comment by antonio....@gmail.com
on 28 Nov 2013 at 2:43
Original issue reported on code.google.com by
joelfiddes
on 23 Jul 2013 at 1:12