Template: VM template

[muhamadazmy]

Part of: #1

[katia-e]

Data model

Model is based on the old ays template excluding redundant fields.

@0xd37ff48ad935931f;
struct Schema {
    # Description for the VM
    description @0 :Text;

    # Name of the VM
    name @1 :Text;

    # OS Image
    osImage @2 :Text = "Ubuntu 16.04";

    # Memory available for the vm in GB
    memory @4 :Int64 = 10;

    # Type of VM: defines the number of CPU and memory available for the vm
    sizeID @5 :Int64 = -1;

    # List of port forwards to create
    ports @6 :List(Text);

    # ID of the VM
    machineId @7 :Int64 = 0;

    # Public ip of the VM
    ipPublic @8 :Text;

    # Private ip of the VM
    ipPrivate @9 :Text;

    # Credentials to create ssh connection to the VM
    sshLogin @10 :Text;
    sshPassword @10 :Text;  

    # Virtual Data Center id
    vdc @11 :Text;

    # List of disk instances to be attached to the VM
    disk @12 :List(Text);

    # stores VM history which includes the actions performed on this machine and the time these actions were performed
    vmHistory @13 :Text;

    # List of vdc users that have access to the vm
    uservdc @14 :List(UserVdcEntry);

    # The name of the machine that will be created when executing the clone action
    cloneName @15 :Text;\

    # The list of snapshots of the node
    snapshots @16 :List(Text);

    # The epoch of a snapshot to rollback or delete.
    snapshotEpoch @17 :Text;

    # Callback url to send vm info after installing if succeeded or report the error to during installation
    vmInfoCallback @18 :Text;

    struct UserVdcEntry {
        name @0 :Text;
        accesstype @1 :Text = "R";
    }
}

[zaibon]

Memory available for the vm in GB memory @4 :Int64 = 10;

That can be right default value. 10GB of ram ?? Should a value in Mib and with a reasonable default value (512 ? )

Type of VM: defines the number of CPU and memory available for the vm

doesnt this conflcit with memory ?

List of port forwards to create ports @6 :List(Text);

what is the execpect format here ?

vmHistory @13 :Text;

why do we need this ?

The name of the machine that will be created when executing the clone action cloneName @15 :Text;\

No need for this to be in the schema, the name of the cloned machine will be an argument of an action

The epoch of a snapshot to rollback or delete. snapshotEpoch @17 :Text;

same here will be an argument of the action

[katia-e]

That can be right default value. 10GB of ram ?? Should a value in Mib and with a reasonable default value (512 ? )

In the old ays template memory is SSD disc size, while sizeID defines the type of the vm, for example type 1: 512MB memory, 1 core; type 2: 1GB memory, 1 core etc. For clarity I can rename memory to diskSize, for example

what is the execpect format here ? (ports @6 :List(Text))

list of strings containing addr:port? Should they better be given as List(Int64) and contain only port numbers?

[zaibon]

For clarity I can rename memory to diskSize, for example

yes please

Should they better be given as List(Int64) and contain only port numbers?

2 solution, or a list of Text, and expecte format is [src:dst] of a list of PortForward with PortForward beeing a struct that has source and destination attribute

[katia-e]

New version

@0xd37ff48ad935931f;
struct Schema {
    # Description for the VM
    description @0 :Text;

    # Name of the VM
    name @1 :Text;

    # OS Image
    osImage @2 :Text = "Ubuntu 16.04";

    # Memory available for the vm in GB
    bootDiskSize @4 :Int64 = 10;

    # Type of VM: defines the number of CPU and memory available for the vm
    sizeID @5 :Int64 = -1;

    # List of port forwards to create
    ports @6 :List(Port);

    struct Port{
        source @0: string;
        destination @1 :string;
    }

    # ID of the VM
    machineId @7 :Int64 = 0;

    # Public ip of the VM
    ipPublic @8 :Text;

    # Private ip of the VM
    ipPrivate @9 :Text;

    # Credentials to create ssh connection to the VM
    sshLogin @10 :Text;
    sshPassword @10 :Text;  

    # Virtual Data Center id
    vdc @11 :Text;

    # List of disk instances to be attached to the VM
    disk @12 :List(Disk);

    struct Disk{
        size @0 :Int64 ;
        iops @1 :Int64;
    }

    # List of vdc users that have access to the vm
    uservdc @14 :List(UserVdcEntry);

    struct UserVdcEntry {
        name @0 :Text;
        accesstype @1 :Text = "R";
    }
}

[zaibon]

LGTM

[yveskerwyn]

What I didn't like about the original schema is disk @12 :List(Text);

Can we change this to disks?

[yveskerwyn]

And for consistency: sizeID -> sizeId

Or: machineId -> machineID

[yveskerwyn]

I remember @grimpy telling me that we will remove the notion of size from the OVC cloudAPI, if that is true, we might get rid of it here too/already, and allow people to specify memory + number of virtual CPU cores explicitly, @grimpy ?

[yveskerwyn]

Not so sure about changing memory to diskSize, this is really not the same, @grimpy ?

[zaibon]

Memory really doesn't make sense if this is supposed to represent the size of the boot disk.

[yveskerwyn]

the way it works now, is that size indeed overrules whatever you specified for memory, if you do not specify a size and only memory, the size (and thus the number of virtual CPU cores) is "calculated", quite confusing, but once you know how it works, you kind of accept it, personally I'd love to see just memory, and an additional attribute to specify the number of virtual CPU cores, @grimpy is best positioned to help us decide here

[zaibon]

There is confusion here. I know that in ovc, you can't choose the combinaison of cpu and memory. It's just predefined sizes. But this should not impact the size of the boot disk. So I'm a bit confused with what you say regarding auto calculations of cpu and ram.

[yveskerwyn]

@zaibon check this: https://github.com/Jumpscale/lib9/blob/9.2.1/JumpScale9Lib/clients/openvcloud/OpenvCloudClientFactory.py#L700

[yveskerwyn]

machine_create() takes vcpus as optional, but currently the node.ovc template doesn't use this argument, see: https://github.com/openvcloud/ays_templates/blob/master/templates/node.ovc/actions.py#25

My suggestion would be to add vcpu to the blueprint template too, and (optionally) drop the size

[zaibon]

Both the lib and the ays templates looks weird to me. @grimpy can you give input here. What exactly does OVC in term of VM configuration nowadays ?

[yveskerwyn]

Also see this: https://github.com/0-complexity/openvcloud/issues/1329

[grimpy]

So from the ticket @yveskerwyn just mentioned we will make it possible in OVC to optionally pass vcpus and memory instead of a sizeId this is scheduled for 2.6 (don't hold your breath).

In the meanwhile there seems to a bit of confusion about the size of the bootdisk. The bootdisk size is limited to what is allowed by the size Example: Sizes:

[
  {
    "description": "",
    "disks": [
      10,      20,      50,      100,      250,      500,      1000,      2000
    ],
    "vcpus": 2,
    "memory": 2048,
    "id": 3,
    "name": "10GB at SSD Speed, Unlimited Transfer - 18 USD/month"
  },
  {
    "description": "",
    "disks": [
      10,      20,       50,      100,      250,      500,      1000,      2000
    ],
    "vcpus": 4,
    "memory": 8192,
    "id": 5,
    "name": "10GB at SSD Speed, Unlimited Transfer - 70 USD/month"
  }
]

Image:

 {
    "username": null,
    "status": "CREATED",
    "description": "",
    "name": "Ubuntu 14.04 x64",
    "size": 10,
    "type": "Linux",
    "id": 3,
    "accountId": 0
  }

So bootdisk size needs to be within the slected size. Also it needs to be minimum the same size or bigger then the image size

[katia-e]

In code for OVC client, in API of machine_create for creating a vm I see such fields as sshkeypath and sshkeyname(required). With new config manager, isn't it supposed to fetch ssh-key automatically?

    def machine_create(
            self,
            name,
            sshkeyname=None,
            memsize=2,
            vcpus=None,
            disksize=10,
            datadisks=[],
            image="Ubuntu 16.04 x64",
            sizeId=None,
            stackId=None,
            sshkeypath=None,
            description=None,
    ):

[katia-e]

I tried to create a VM giving the name of the ssh-key loaded to the agent. So my call looks like this:

        space.machine_create(name=self.data['name'],
                             sshkeyname='id_rsa',
                            )

The machine is created successfully (as well as if no keys are given), but then I observe an attempt of the ssh connection and zrobot stops responding and start loading CPU:

Hanging screen is this:

[katia-e]

Currently I'm trying to get ssh connection to a VM. I use j.clients.ssh.ssh_keys_list_from_agent() to get a path to the ssh-key uploaded to the ssh-agent. Is this a correct way?

Initially I was expecting that I can fetch key from sshkey service, but it only loads key with given path and doesn't store neither path nor name of the key.

[zaibon]

using ssh-agent is not the right way to work. we've been having with ssh-agent already when it loads more then a few keys usually openssh server stop accepting ssh connection cause too many login attemps have failed due to too many sshley tried.

Right way to do this is to always specify what key to use when creating a connection. I think js ssh client supports it, if not we need to update it so it can be used like that.

[katia-e]

Right way to do this is to always specify what key to use when creating a connection.

@zaibon , how to specify the key? to additionally provide a path or keyname in the blueprint?

I think js ssh client supports it

I thought .js ssh client just calls ssh-agent, what is the difference then?

[zaibon]

I think js ssh client supports it, if not we need to update it so it can be used like that.

which mean I don't know how or if is expose on the ssh client in jumspcale. But if not, this has to be done

@rkhamis maybe you can comment on this ?

[katia-e]

I see the templates now use this data structure service.model.data, service.producers and so on. I thought they are from old ays and we have to store evetything in self.data instead, no?

[zaibon]

indeed, these are all old implementation from AYS. this needs to be changed

[katia-e]

A VM can never be created with the same name even if the previous machine was deleted? Is that correct?

[yveskerwyn]

it is a bug that only exists on be-g8-4: https://github.com/0-complexity/openvcloud/issues/1346

on be-gen-1 and ch-gen-1 it works as expected

[katia-e]

Data model for diskovc template

struct Schema {
    # Size of the disk in GB
    size @0 :Int64 = 1;

    # Type of the disk (B=Boot; D=Data)
    type @1 :Text = "D";

    # description of disk
    description @2 :Text = "disk";

    # name of the disk
    devicename @3 :Text;

    # id of the disk
    diskId @4: Int64;

    # OpenvCloud
    openccloud @5 :Text;

    # location of the disk
    location @6 :Text;

    # Limmits
    maxIOPS @7 :Int64 = 2000;

    totalBytesSec @8 :Int64;

    readBytesSec @9 :Int64;

    writeBytesSec @10 :Int64;

    totalIopsSec @11 :Int64;

    readIopsSec @12 :Int64;

    writeIopsSec @13 :Int64;

    totalBytesSecMax @14 :Int64;

    readBytesSecMax @15 :Int64;

    writeBytesSecMax @16 :Int64;

    totalIopsSecMax @17 :Int64;

    readIopsSecMax @18 :Int64;

    writeIopsSecMax @19 :Int64;

    sizeIopsSec @20 :Int64;
}

[katia-e]

Was implemented in #19