Only allocate SNAT IP when there is no ephemeral or floating IP

bnaecker commented 1 year ago

In the instance-creation saga, we unconditionally allocate an SNAT IP for every instance. This is regardless of whether the request explicitly asked for ephemeral addresses. In that case, OPTE always uses the ephemeral address anyway, and no network traffic will ever flow through the SNAT IP. We ought to make such allocation optional, and only provide one if no ephemeral addresses were requested.

bnaecker commented 1 year ago

Related to https://github.com/oxidecomputer/omicron/issues/2715. We should ideally also move the SNAT address allocation and release to the instance start saga and (maybe a new?) stop saga.

david-crespo commented 7 months ago

Wonder if the fact that you can now detach and attach ephemeral IPs changes things here. If you start with an ephemeral IP and no SNAT IP, then you detach the ephemeral IP, you have nothing.

david-crespo commented 7 months ago

Based on the discussion in #5043, we should:

At instance create, only allocate a SNAT IP (from default pool) if there is no ephemeral or floating IP
When the user has detached all ephemeral and floating IPs, allocate a SNAT IP (from default pool)
When an ephemeral or floating IP is attached, remove SNAT IP if there is one

None of that sounds too hard. One wrinkle is that we'd be introducing a new way that detaching a floating or ephemeral IP can fail, because if there is no default pool or the default pool is full, then allocating the SNAT IP at detach time will fail and we probably shouldn't allow that. Fortunately the lack of a default pool means it's very hard to do anything, so we should consider that just a Bad Configuration among many possible bad configurations operators must endeavor to avoid. Default pool being full is also generally a bad situation — you can't create new instances either — so it's not that weird if a few other operations are hamstrung by that.

david-crespo commented 5 months ago

We're not getting this into v9.

askfongjojo commented 2 months ago

Would this still be required given the internet gateway feature? (https://github.com/oxidecomputer/omicron/issues/2154)

rcgoodfellow commented 2 months ago

Would this still be required given the internet gateway feature? (https://github.com/oxidecomputer/omicron/issues/2154)

Yes, I think the steps that @david-crespo outlined above should be taken independently of the internet gateway feature.

oxidecomputer / omicron

Only allocate SNAT IP when there is no ephemeral or floating IP #4317