search

abhilekhsingh / gc3pie

Automatically exported from code.google.com/p/gc3pie
0 stars 0 forks source link

EC2 backend keeps spawning VMs if flavor and application requests have a mismatch #429

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?

1. Select a small VM flavor, e.g., `m1.small`
2. Try to submit an application with larger memory requirements from a 
session-based script, e.g., `-m 4GB`

What is the expected output? What do you see instead?

I would expect the EC2 backend to detect the mismatch after starting
the first VM, and giving up with an `UnrecoverableError`.

Instead, it keeps spawning VMs, none of which can host the job(s)
because they're too small:

    gc3.gc3libs: DEBUG: Ignoring error while submit to resource 130.60.24.103@hobbes: Resource 130.60.24.103@hobbes does not have enough available memory: 4000MB requested, but only 2050MB available..
    gc3.gc3libs: DEBUG: Checking status of the following PIDs:
    gc3.gc3libs: DEBUG: Ignoring error while submit to resource 130.60.24.87@hobbes: Resource 130.60.24.87@hobbes does not have enough available memory: 4000MB requested, but only 2050MB available..
    gc3.gc3libs: DEBUG: Checking status of the following PIDs:
    gc3.gc3libs: DEBUG: Ignoring error while submit to resource 130.60.24.107@hobbes: Resource 130.60.24.107@hobbes does not have enough available memory: 4000MB requested, but only 2050MB available..
    gc3.gc3libs: DEBUG: Checking status of the following PIDs:
    gc3.gc3libs: DEBUG: Ignoring error while submit to resource 130.60.24.105@hobbes: Resource 130.60.24.105@hobbes does not have enough available memory: 4000MB requested, but only 2050MB available..
    gc3.gc3libs: DEBUG: Checking if keypair is registered in SSH agent...
    gc3.gc3libs: DEBUG: Found remote key fingerprint in SSH agent.
    gc3.gc3libs: DEBUG: Create new VM using image id `ami-000000ef`
    gc3.gc3libs: INFO: VM with id `i-00000f59` has been created and is in pending state.

Original issue reported on code.google.com by riccardo.murri@gmail.com on 21 Mar 2014 at 3:53

GoogleCodeExporter commented 9 years ago 
In addition, I would expect that unused VMs are terminated after some
period of inactivity.  This does not seem to happen: after like 10
minutes, the first-spawned VM is still there though no jobs are
running on it.

GC3Pie seems to be aware of the issue though:

    gc3.gc3libs: INFO: Closing connection to cloud 'hobbes'...
    gc3.gc3libs: DEBUG: Checking status of the following PIDs:
    gc3.gc3libs: DEBUG: Recovered resource information from files in /home/gc3-user/.gc3/shellcmd.d: available memory: 2050.65MB, used memory: 0MB
    gc3.gc3libs: DEBUG: Checking status of the following PIDs:
    gc3.gc3libs: DEBUG: Recovered resource information from files in /home/gc3-user/.gc3/shellcmd.d: available memory: 2050.65MB, used memory: 0MB
    gc3.gc3libs: DEBUG: Checking status of the following PIDs:
    gc3.gc3libs: DEBUG: Recovered resource information from files in /home/gc3-user/.gc3/shellcmd.d: available memory: 2050.65MB, used memory: 0MB
    gc3.gc3libs: DEBUG: Checking status of the following PIDs:
    gc3.gc3libs: DEBUG: Recovered resource information from files in /home/gc3-user/.gc3/shellcmd.d: available memory: 2050.65MB, used memory: 0MB
    gc3.gc3libs: WARNING: VM instance i-00000f55 at 130.60.24.103 is no longer needed. You may need to terminate it manually.
    gc3.gc3libs: WARNING: VM instance i-00000f54 at 130.60.24.87 is no longer needed. You may need to terminate it manually.
    gc3.gc3libs: WARNING: VM instance i-00000f57 at 130.60.24.107 is no longer needed. You may need to terminate it manually.
    gc3.gc3libs: WARNING: VM instance i-00000f56 at 130.60.24.105 is no longer needed. You may need to terminate it manually.

Original comment by riccardo.murri@gmail.com on 21 Mar 2014 at 3:56