search

TritonDataCenter / rfd

Requests for Discussion
Mozilla Public License 2.0
261 stars 75 forks source link

RFD 107: Discussion #49

Open papertigers opened 7 years ago

papertigers commented 7 years ago

This issue represents an opportunity for discussion of RFD 107 Self assigned IP's and reservations while it remains in a pre-published state.

rmustacc commented 7 years ago

When comparing the flexibility we have/want on the VMAPI end, particularly with IPv6 coming in the future, is there a reason that we're not mirroring some of flexibility of the 'xxx_count' and 'xxx_ips' options?

Particularly, how do you expect this to change when we want to allow CloudAPI to specify both an IPv4 and IPv6 network on a single NIC?

danmcd commented 7 years ago

@rmustacc any WIP or completed RFDs @papertigers should cite? (I'm curious about the have vs. want space in VMAPI.)

rmustacc commented 7 years ago

@danmcd This was all done before RFDs existed, but @papertigers is already citing the docs in VMAPI which talk about this.

papertigers commented 7 years ago

After giving it some thought I agree with you @rmustacc.

You are talking about a scenario where one interface can have one or many IPv4 addresses as well as one or many IPv6 addresses.

For example the value passed to VMAPI would be:

[
  { "ipv4_uuid": "72a9cd7d-2a0d-4f45-8fa5-f092a3654ce2", 
    "ipv4_ips": ["192.168.1.234"],
    "ipv6_uuid": "d1516824-ece0-46d0-bbe1-87a120268d16",
    "ipv6_count": 1
  }
]

My thoughts were that we would know the type of network passed into VMAPI so specifying ipvX_network was not necessary. However, in that scenario it would be impossible to provision with the above scenario. I think the api interface should be updated to more closely reflect VMAPI's interface.

rmustacc commented 7 years ago

@papertigers You make a good point though. There's no good reason to really overburden the user and figure out if it's an IPv4 or IPv6 network. The main reason I wanted to mention it was to make sure that we could specify multiple networks / nic and specify the array of IPs and count. It may make sense to just have the fact that it's IPv4 / IPv6 be figured out inside of Triton.

At least, on the surface, it seems like an unnecessary piece of information to force a user to know, especially when the simple case of only one network is going to be the primary thing.

rmustacc commented 7 years ago

Some notes from discussion today:

askfongjojo commented 7 years ago

It'll be worthwhile to have a section on the key use cases to help us to visualize the workflow and supporting API calls involved. Here are some examples I can think of:

  1. Pre-allocated IP ranges for different applications/environments/teams Sometimes a subnet is shared between Dev/Test. The way to maintain a soft quota is to assign a range of IP addresses to each of them. In this case, there is no need to create and reserve the nic objects ahead of time. A specific IP address will be passed during provisioning requests. When instances are destroyed, the nics are destroyed along with them. What users in this case will also need are:

    • ability to see which IP addresses in the subnet are in use / free, and which instances are using them
    • validation against the request for an IP address that is already in use

  2. Designated IP addresses for applications that are "bound" to them Some application licenses are linked to IP addresses. Some have IP-based firewall rules or DNS mappings. There is a practical need to keep their IP addresses unchanged to avoid the overhead of maintaining these dependencies. As such, these fixed IP addresses need to be reserved at all times. When the applications using them are de-provisioned and provisioned again, the IP addresses should not be grabbed by another instance. Users in this case will need:

    • ability to reserve an IP address
    • ability to keep an IP address reserved after an instance is destroyed or the nic is removed from the instance

  3. Similar to 2, there are also cases when fixed mac addresses are required NIC objects with a specific mac and optionally a specific IP may be created and reserved ahead of time. End user in this case will need to be able to pass the mac address in the NIC request.

papertigers commented 7 years ago

Working on pushing out an updated version addressing the above feedback.

I think that if we do end up allowing users to specify mac addresses it will only be the last three octets so that our OUI stays the same. I will leave that out of the first update I am going to push and we can discuss that after.

papertigers commented 7 years ago

Changes pushed.

Thanks @melloc and @askfongjojo for the feedback on vmapi changes.

This should maybe even work for public networks for the IPs that one has allocated (ListIPs / Get IP)

Does napi currently let you query a public network by account uuid, and is there a way to make sure the GetIP call is being called only on an IP you own?

How does setting the primary NIC fit in here?

Still unclear. For now I have added it to AddNic, but that means will also probably want an UpdateNic.

It'll be worthwhile to have a section on the key use cases to help us to visualize the workflow and supporting API calls involved.

Let me get that section written up.

The question about being able to specify the mac address is still an open issue. Still not totally convinced we should allow this, and if we do it should only be the last three octets.

danmcd commented 7 years ago

For AddNIC, if we get to a world with multiple IPs on a NIC, how does a "boolean" for primary help? Shouldn't it indicate a specific IP? Or is just knowing the NIC is primary sufficent?

papertigers commented 7 years ago

Theres some background on primaryIP and primary NIC in this ticket: https://smartos.org/bugview/PUBAPI-1291

Perhaps @marsell has more input on what that should actually look like since I believe he is currently making some changes in that area.

danmcd commented 7 years ago

Relevant code-change in the above ticket: https://github.com/joyent/sdc-cloudapi/commit/2a296b0 (They acknowledge a future where multiple IPs (v4 or v6) can be on a single NIC.)

rmustacc commented 7 years ago

Thanks for the updates. Based on what's here I have a few follow up questions:

papertigers commented 7 years ago

In the AddNic section, it sounds like the main thing that we're allowing now is to specify the ability to add an 'ip' entry to specify the specific IP, is that right?

Thats right. What I don't know is if this endpoint should allow you to specify that this ip is also the primaryIp. Anyone have thoughts on this?

Should we define what kinds of errors we're going to get back if we try to provision an IP on a network that we don't own, but can provision. For example, on the public networks? We may want to define the set of CloudAPI level errors for all the modified / new entry points.

Agreed. I will start working on this. Also, in the 'Future' section of AddNic I provide an example of what adding multiple IP's to a nic looks like. Do we want to define what happens in that case? For example one of the specified ips is not available etc. In that situation its probably best we validate that the entire request can be completed without issue so that we do not end up with a scenario where only part of the request is successful.

Can we more explicitly state which Networks one can list? Can I perform a ListNetworkIPs of a newtork I have something provisioned on like a shared public network, if so how are we going to limit the provisionability?

I think you should be able to list any private network you are an owner of, including one another user has given you access to.

I also think listing IPs on public network owned by your account would be ideal. However, the napi docs say we can filter a networks ips by belongs_to_uuid but not something like owner_uuid. However looking at the source the moray bucket has the info. So the validation schema can probably be updated to allow filtering by owner. @melloc do you have any concerns with this?

Can we talk through a few of the lifecycle things here. For example, say I have an IP that's provisioned. Can I mark it reserved? If so, what happens to it when I delete the instance that's provisioned it. Does it stay reserved?

This is how it worked in my head when I wrote the rfd. The napi docs document similar lifecycle things. I will add it to be explicit.

Should we have fabric aliases for the IP listing capabilities? In other words, should GET /:login/fabrics/default/vlans/:vlan_id/networks/:network/ips/:ip_address be an alias for GET /:login/networks/:id/ips/:ip_address.

This was my intention.

trentm commented 7 years ago

Types of NAPI IP objects:

  // 1. I believe, this is an IP that was allocated (e.g. perhaps because of
  //    earlier usage), but is now free.
  {
    "ip": "10.99.99.52",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": false,
    "free": true
  },

  // 2. An IP assigned (belonging) to a given server.
  {
    "ip": "10.99.99.45",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": false,
    "free": false,
    "belongs_to_type": "server",
    "belongs_to_uuid": "564dc847-1949-276a-8eb3-8d3a1df000ca",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  },

  // 3. An IP assigned (belonging) to a given instance.
  {
    "ip": "10.99.99.46",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": false,
    "free": false,
    "belongs_to_type": "zone",
    "belongs_to_uuid": "2f612f25-792d-4d13-9360-44a862cd1cb0",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  },

  // 4. A *reserved* IP. (It also happens to be assigned to an instance.)
  {
    "ip": "10.99.99.11",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": true,
    "free": false,
    "belongs_to_type": "zone",
    "belongs_to_uuid": "7ce84909-8ebf-4072-963f-64f603c94f34",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  },

  // 5. A *specially reserved* IP (belongs_to_type=other).
  //
  //    NAPI currently does this (belongs_to_type=other) for
  //    (a) network.gateway, (b) network.resolvers, and (c) IPv4 broadcast
  //    addresses. In these cases the `belongs_to_uuid` is set to either the
  //    UFDS admin user UUID (i.e. owned by the system) or (for a non-NATed
  //    fabric gateway) to '00000000-0000-0000-0000-000000000000'.
  // 
  //    AFAIK, the NAPI API can be used by operators to set
  //    belongs_to_uuid=other for other purposes.
  {
    "ip": "10.99.99.255",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": true,
    "free": false,
    "belongs_to_type": "other",
    "belongs_to_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  }

Re ListNetworkIPs and GetNetworkIP:

papertigers commented 7 years ago

Just wanted to dump some of my comments on this from discussion over chat.

It may be worth while to expose all of the raw napi output with exception to the free property, which is defined by the napi docs as follows:

Setting the free property on an IP removes all other properties from the IP (including reserved,
belongs_to_uuid, and belongs_to_type). This therefore makes the IP available for automatic
provisioning again.

Having a belongs_to_uuid=other would be much better than having a made up cloudapi concept aka triton_reserved=true. Especially because having a property either set to true or undefined like @trentm pointed out is a little awkward. We just have to make sure its documented in cloudapi that type other means you can't update that specific ip aka the goal of triton_reserved

I also now realize there is value in including the owner_uuid property. For example when a user is listing ip's on a private network that has multiple owners, it would be nice to know which account owns that instance on that network.

So, in addition to Trent's questions, is the napi interface considered stable? Do we have an issue with returning raw napi output?

cc @rmustacc @melloc @danmcd

trentm commented 7 years ago

My notes from the call. There are some "Q" questions below.

Example NAPI IP object:

  {
    "ip": "10.99.99.255",
    "network_uuid": "afafc2ed-a173-4de6-914f-fddda47a2a57",
    "reserved": true,
    "free": false,
    "belongs_to_type": "other",
    "belongs_to_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  }

CloudAPI IP objects:

Naming "managed": Something that says "this IP is managed by something other than you". "You can't release this IP." Ideas:

Q: What does a NAPI IP object for a volume storage zone nic look like? What should the cloudapi IP object look like for it?

Q: What does a NAPI IP object for a NAT zone NIC look like?

trentm commented 7 years ago

Q: What does a NAPI IP object for a NAT zone NIC look like?

A nat zone in nightly-1 has a vm.nics[1] like this:

    {
      "interface": "net1",
      "mac": "90:b8:d0:1e:ef:b3",
      "vlan_id": 2,
      "nic_tag": "sdc_overlay/8227494",
      "gateway": "192.168.128.1",
      "gateways": [
        "192.168.128.1"
      ],
      "netmask": "255.255.252.0",
      "ip": "192.168.128.1",
      "ips": [
        "192.168.128.1/22"
      ],
      "network_uuid": "b0427f14-89aa-419b-b84a-c3de86b88e0c",
      "allow_ip_spoofing": true,
      "mtu": 8500
    }

and its NAPI IP object is the first one here:

[root@headnode (nightly-1) ~]# sdc-napi /networks/b0427f14-89aa-419b-b84a-c3de86b88e0c/ips
[
  {
    "ip": "192.168.128.1",
    "network_uuid": "b0427f14-89aa-419b-b84a-c3de86b88e0c",
    "reserved": true,
    "free": false,
    "belongs_to_type": "zone",
    "belongs_to_uuid": "18329628-2b5a-4809-affa-458f7ce4fe7a",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  },
...
  {
    "ip": "192.168.131.255",
    "network_uuid": "b0427f14-89aa-419b-b84a-c3de86b88e0c",
    "reserved": true,
    "free": false,
    "belongs_to_type": "other",
    "belongs_to_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853"
  }
]

Following the rules mentioned above, the CloudAPI IP object for that NAT zone's IP would be:

  {
    "ip": "192.168.128.1",
    "reserved": true,
    "owner_uuid": "930896af-bf8c-48d4-885c-6573a94b1853",
    "managed": false
  },

which isn't quite right yet because we'd want "managed": true I think.

Q: So perhaps the managed rule needs to be updated to be true if belongs_to_type === "other" or the owner_uuid is the "admin" account?

ip.managed = (napiIp.belongs_to_type==="other" || napiIp.owner_uuid === adminUuid);
papertigers commented 7 years ago

The NAT zone's napi payload is what I was trying to get at on the call, so thanks for grabbing that output.

field owner_uuid: always show (allows seeing IP is owned by another user that is sharing a private network)

When an IP is reserved by a user or operator napi will show it as follows:

  {
    "ip": "192.168.128.4",
    "network_uuid": "080db52e-8ccf-4a6a-9176-56b746345db3",
    "reserved": true,
    "free": false
  }

In this case there is no owner_uuid field to relay back to cloudapi but users still need to be able to see reserved IPs.

Q: In the case above do you want "owner_uuid": "" or would you rather the field be omitted?

trentm commented 7 years ago

@papertigers

When an IP is reserved by a user or operator napi will show it as follows:

Should that example show reserved=true?

papertigers commented 7 years ago

@trentm it should have, you might have seen github before I edited the comment with the right paste.

papertigers commented 7 years ago

We plan on exposing owner_uuid in all ip's returned when its set. However, things like a users nat zone is owned by the admin user.

Q. Do we have concerns over exposing the admin UUID? We could always go with a uuid of all zero's or something like the string "triton", although thats not a valid uuid.

jussisallinen commented 7 years ago

Q. Do we have concerns over exposing the admin UUID? We could always go with a uuid of all zero's or something like the string "triton", although thats not a valid uuid.

As an operator I would very much prefer all zero’s or anything else over exposing the admin UUID if possible, just to be on the safe side.

papertigers commented 7 years ago

Q. Do we have concerns over exposing the admin UUID? We could always go with a uuid of all zero's or something like the string "triton", although thats not a valid uuid.

The decision from chat was that we should omit owner_uuid in this case rather than lie about it.

trentm commented 7 years ago

FWIW we already expose the admin UUID as the "owner" field on public and admin-owned images. That could theoretically also be changed in future versions of the APIs. I'm still fine with us starting by omitting owner_uuid if it is the admin UUID.

Q: In the case above do you want "owner_uuid": "" or would you rather the field be omitted?

@papertigers I'd rather owner_uuid be omitted if the field is not present on the NAPI IP object.