IPv4: Need to standardize how public IPv4 networking behaves by default

garloff commented 1 year ago

As a IaaS user (DevOps person), I need to make sure the way to get a public IPv4 address to a VM or Loadbalancer is standardized across SCS clouds.

Definition of Ready:

[ ] User Story is small enough to be finished within one sprint
[ ] User Story is clear and understood by the whole team
[ ] Acceptance criteria are defined
[ ] Acceptance criteria are clear and understood by the whole team

Definition of Done:

[ ] All acceptance criteria are met
[ ] Changes have been reviewed
[ ] CI tests have run successfully
[ ] Documentation has been updated
[ ] Release Notes have been updated

fkr commented 1 year ago

This is important, since we don't want to fall into this category:

https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/cloud/meta.py#L160

garloff commented 1 year ago

90+% of existing OpenStack clouds have an external virtual network (called external or ext01 or ext-net or internet or public or so) from which floating IP addresses can be allocated and which can be attached to fixed IP addresses from internal virtual networks.

garloff commented 1 year ago

So I favor the

* Cloud only has private tenant network provided by neutron and requires
   floating-ip for external routing (dreamhost, hp)

approach. (Maybe this is not used by 90+% of all OpenStack clouds?)

garloff commented 1 year ago

Plan to standardize IPv4 networking:

Created networks, subnets are private
Networks can be connected to other networks with a router
Router can have an external network (gateway) configured (opt-in!), then
- SNAT is enabled, so outgoing IPv4 works as soon as router has external gateway set (disabling connectivity in such setups via security groups, disabling the router's SNAT is unfortunately a privileged operation on most clouds)
- Floating IPs can be attached to existing private ports in subnets connected to a such connected router
- Outgoing connections are now using this floating IP (instead of a shared SNAT IP)
- Incoming connections to this floating IP reach the VM (subject to security groups of course) or the Loadbalancer
OPTIONAL: Additional networks (provider networks), which may provide access to provider-specific networks and possibly as well to the internet

garloff commented 1 year ago

Open question: Should the name of the external network be standardized? external (wave)? ext01 (pco)? internet (sxperts)? ext-net (cleura)? admin_external_net (otc) Ext-Net (ovh), extern (teuto)? Or maybe at least a recommendation (SHOULD)? And make it transparent in the self-description? (Note: It is trivially discoverable for those with access: openstack network list --external. However, still adds complexity for terraform, ansible, heat automation.)

garloff commented 1 year ago

Sidenote: IPv6 connectivity is dealt with in a separate issue (#166).

fkr commented 1 year ago

@garloff @berendt is this something that should be done as part of VP04 (and the work of VP04 taken into consideration for that)?

garloff commented 10 months ago

I don't think that any work in VP04 will change the user-visible approach to networking in OpenStack.

berendt commented 10 months ago

I don't think that any work in VP04 will change the user-visible approach to networking in OpenStack.

I would agree with that

fkr commented 9 months ago

Open question: Should the name of the external network be standardized? external (wave)? ext01 (pco)? internet (sxperts)? ext-net (cleura)? admin_external_net (otc) Ext-Net (ovh), extern (teuto)? Or maybe at least a recommendation (SHOULD)? And make it transparent in the self-description? (Note: It is trivially discoverable for those with access: openstack network list --external. However, still adds complexity for terraform, ansible, heat automation.)

Not really sure wether agreeing on a standard name is helpful (especially since there are also clouds with more than one (1) external network. Furthermore this is - yet another topic - that will cause grief with existing clouds. Much rather I'd like to see that we standardize the functional default behaviour - so that things behave the same way on scs compliant clouds.

fkr commented 9 months ago

Router can have an external network (gateway) configured (opt-in!), then

SNAT is enabled, so outgoing IPv4 works as soon as router has external gateway set (disabling connectivity in such setups via security groups, disabling the router's SNAT is unfortunately a privileged operation on most clouds)

Floating IPs can be attached to existing private ports in subnets connected to a such connected router

Outgoing connections are now using this floating IP (instead of a shared SNAT IP)

Incoming connections to this floating IP reach the VM (subject to security groups of course) or the Loadbalancer

OPTIONAL: Additional networks (provider networks), which may provide access to provider-specific networks and possibly as well to the internet

This seems to be a good starting point and could be drafted.

josephineSei commented 5 months ago

After looking through this issue and discussing with @markus-hentsch , I have identified more than one thing to standardize:

standardized network architecture / topology:
- this means, we want to have a picture about how network topology should look like
- e.g. always use private networks to attach VMs to, and always use a neutron router to gain access from the private network to an external network.
- open questions: is the "external" flag enough to mark provider networks?
- do we allow shared networks (between projects)? or do we just not care?
- how many external networks are okay? (e.g. one for IPv4, one for IPv6, and maybe one for a special VPN case?... )
- how to choose the fitting network.
if we have a coarse picture of the topology: standardized workflow / guideline to work with Floating IPs:
- necessary because of some strange behavior of Floating IPs: it is possible to steal a floating ip from a VM in the same project and that should be avoided.

I would add a separate issue for the second part, with a detailed description, of the behavior, if you want @fkr

josephineSei commented 5 months ago

I described the behavior in here: https://input.scs.community/gRSKVYboQRebi62Lx_eSMA?both

josephineSei commented 5 months ago

I described the behavior in here: https://input.scs.community/gRSKVYboQRebi62Lx_eSMA?both

I edited some parts of the etherpad in preperation of the IaaS call on Wednesday 20th of March

josephineSei commented 4 months ago

We (@cah-patrickthiem , @anjastrunk , @josephineSei , @markus-hentsch , @kgube , @martinmo , @tonifinger , @mbuechse ) had a lengthy discussion about this issue and possible splitting some of the open questions mentioned in the IaaS-call today into different issues. That would mainly concern the loadbalancer. Some other issues for security groups, floating IPs, DNS, etc... already exist.

josephineSei commented 4 months ago

I wanted to look into the naming, when I stumbled upon this:

stack@devstack:~/devstack$ openstack network show public
+---------------------------+----------------------------------------------------------------------------+
| Field                     | Value                                                                      |
+---------------------------+----------------------------------------------------------------------------+
| admin_state_up            | UP                                                                         |
| availability_zone_hints   |                                                                            |
| availability_zones        |                                                                            |
| created_at                | 2024-01-24T16:12:31Z                                                       |
| description               |                                                                            |
| dns_domain                | None                                                                       |
| id                        | 73edb86b-d7ab-4db3-82b7-25fa8b012e40                                       |
| ipv4_address_scope        | None                                                                       |
| ipv6_address_scope        | None                                                                       |
| is_default                | True                                                                       |
| is_vlan_transparent       | None                                                                       |
| mtu                       | 1500                                                                       |
| name                      | public                                                                     |
| port_security_enabled     | True                                                                       |
| project_id                | 15f2ab0eaa5b4372b759bde609e86224                                           |
| provider:network_type     | flat                                                                       |
| provider:physical_network | public                                                                     |
| provider:segmentation_id  | None                                                                       |
| qos_policy_id             | None                                                                       |
| revision_number           | 3                                                                          |
| router:external           | External                                                                   |
| segments                  | None                                                                       |
| shared                    | False                                                                      |
| status                    | ACTIVE                                                                     |
| subnets                   | 3e0206bc-53c8-44ca-a0f1-2c2548bba766, 84dffd43-6d7f-4c2f-9180-8f0f0b83c9d4 |
| tags                      |                                                                            |
| tenant_id                 | 15f2ab0eaa5b4372b759bde609e86224                                           |
| updated_at                | 2024-01-24T16:12:52Z                                                       |
+---------------------------+----------------------------------------------------------------------------+
stack@devstack:~/devstack$ openstack subnet show 3e0206bc-53c8-44ca-a0f1-2c2548bba766
+----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| allocation_pools     | 172.24.4.2-172.24.4.254              |
| cidr                 | 172.24.4.0/24                        |
| created_at           | 2024-01-24T16:12:38Z                 |
| description          |                                      |
| dns_nameservers      |                                      |
| dns_publish_fixed_ip | None                                 |
| enable_dhcp          | False                                |
| gateway_ip           | 172.24.4.1                           |
| host_routes          |                                      |
| id                   | 3e0206bc-53c8-44ca-a0f1-2c2548bba766 |
| ip_version           | 4                                    |
| ipv6_address_mode    | None                                 |
| ipv6_ra_mode         | None                                 |
| name                 | public-subnet                        |
| network_id           | 73edb86b-d7ab-4db3-82b7-25fa8b012e40 |
| project_id           | 15f2ab0eaa5b4372b759bde609e86224     |
| revision_number      | 0                                    |
| segment_id           | None                                 |
| service_types        |                                      |
| subnetpool_id        | None                                 |
| tags                 |                                      |
| updated_at           | 2024-01-24T16:12:38Z                 |
+----------------------+--------------------------------------+
stack@devstack:~/devstack$ openstack subnet show 84dffd43-6d7f-4c2f-9180-8f0f0b83c9d4
+----------------------+-------------------------------------------------------------------+
| Field                | Value                                                             |
+----------------------+-------------------------------------------------------------------+
| allocation_pools     | 2001:db8::1-2001:db8::1,2001:db8::3-2001:db8::ffff:ffff:ffff:ffff |
| cidr                 | 2001:db8::/64                                                     |
| created_at           | 2024-01-24T16:12:52Z                                              |
| description          |                                                                   |
| dns_nameservers      |                                                                   |
| dns_publish_fixed_ip | None                                                              |
| enable_dhcp          | False                                                             |
| gateway_ip           | 2001:db8::2                                                       |
| host_routes          |                                                                   |
| id                   | 84dffd43-6d7f-4c2f-9180-8f0f0b83c9d4                              |
| ip_version           | 6                                                                 |
| ipv6_address_mode    | None                                                              |
| ipv6_ra_mode         | None                                                              |
| name                 | ipv6-public-subnet                                                |
| network_id           | 73edb86b-d7ab-4db3-82b7-25fa8b012e40                              |
| project_id           | 15f2ab0eaa5b4372b759bde609e86224                                  |
| revision_number      | 0                                                                 |
| segment_id           | None                                                              |
| service_types        |                                                                   |
| subnetpool_id        | None                                                              |
| tags                 |                                                                   |
| updated_at           | 2024-01-24T16:12:52Z                                              |
+----------------------+-------------------------------------------------------------------+
stack@devstack:~/devstack$ openstack floating ip create --subnet 84dffd43-6d7f-4c2f-9180-8f0f0b83c9d4 public
Error while executing command: BadRequestException: 400, Unable to find any IP address on external network 73edb86b-d7ab-4db3-82b7-25fa8b012e40.
stack@devstack:~/devstack$ openstack floating ip create public
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2024-03-27T13:35:59Z                 |
| description         |                                      |
| dns_domain          |                                      |
| dns_name            |                                      |
| fixed_ip_address    | None                                 |
| floating_ip_address | 172.24.4.107                         |
| floating_network_id | 73edb86b-d7ab-4db3-82b7-25fa8b012e40 |
| id                  | 9e193d33-17f9-400b-b639-b51750d41bc0 |
| name                | 172.24.4.107                         |
| port_details        | None                                 |
| port_id             | None                                 |
| project_id          | 15f2ab0eaa5b4372b759bde609e86224     |
| qos_policy_id       | None                                 |
| revision_number     | 0                                    |
| router_id           | None                                 |
| status              | DOWN                                 |
| subnet_id           | None                                 |
| tags                | []                                   |
| updated_at          | 2024-03-27T13:35:59Z                 |
+---------------------+--------------------------------------+

The error with the floating ip for ipv6 can be a local issue.

A single (external) network can have multiple subnets. This is the same as physical behavior: on the same Layer 2 network there can be multiple layer three subnets. But within the scope of this ticket this leads to the following question:

Do we allow multiple / different subnets for external networks (e.g. combining IPv4 and IPv6)?
- this would lead to users having to specify the correct subnet when asking for a floating IP
How do we let user know what to expect from a network (e.g. will it grant IPv4 or IPv6 floating IP at default? Which subnet is used defaultly?)

markus-hentsch commented 4 months ago

Port Security and Shared Networks

@cah-patrickthiem, @josephineSei in the context of our discussion today about port security and security groups I had a closer look at this on a DevStack and found some interesting things.

Context recap: we consider disabled port security on ports within a network shared by multiple tenants a threat because then malicious tenants can spoof their addresses to be that of other tenants in the network. With port security enabled, OpenStack makes sure that only traffic from/to known assigned addresses is allowed. Also port security controls whether security group rules are applied or not (enabled port security = security groups enabled).

Changing port security on admin side:

an admin can create a shared or public/external[^1] network with --disable-port-security which disables port security for all ports created in this network per default
- if tenants create VMs within such network while not specifying security groups, no default security group will be assigned to the VM
- if tenants create VMs within such network while specifying at least one security group, the server will reach ERROR state but the message about it not being able to assign the security group because of the network enforcing disabled port security seems to be only visible in Nova's logs (see the bottom of this post), not in the server status

We should add "Port security MUST NOT be disabled for networks created as shared (--share on CLI) or with router:external (--external on CLI)" or something similar to the standard. Note: how we phrase this depends on how the glossary turns out, i.e. whether we state the API attributes here (shared, router:external) or if we reference well-defined network classification terminology established in the glossary.

[^1]: public/external as in --external on CLI or router:external in the API.

Changing port security on user side:

if a tenant uses openstack port set --disable-port-security ... on a port in a shared or public/external[^1] network, the default Neutron API policies will deny this request:
```
ForbiddenException: 403: Client Error for url: $NEUTRON:9696/networking/v2.0/ports/b4a5ea84-6084-43dd-a5bc-8b8bf40cba0f, (rule:update_port and rule:update_port:port_security_enabled) is disallowed by policy
```
- the default policy^2 restricts this to admins or users of the same project the network was created in

Since in the default policy creation of shared or external networks is an admin-only action, we should be safe here in most cases. However, since admins can create networks in specific projects (e.g. openstack network create --project ...) in conjunction with the external or shared flag, this might introduce loopholes.

Therefore, I think we should also add "Networks marked as shared or external (router:external attribute) MUST NOT be created within a tenant's project" or something. Again, phrasing should be in line with the terminology established.

Here is the nova-compute log entry when a tenant tries to create a VM with a security group assigned in a network where the admin has disabled port security:

nova.exception.SecurityGroupCannotBeApplied: Network requires port_security_enabled and subnet associated in order to apply security groups

... and this is what the user sees in openstack server show:

| fault                               | {'code': 500, 'created': '2024-04-09T11:45:40Z', 'message':        |
|                                     | 'Exceeded maximum number of retries. Exhausted all hosts available |
|                                     | for retrying build failures for instance                           |
|                                     | 4dd38355-7bdf-4c84-b07f-1516c3ae0925.'}                            |
| status                              | ERROR                                                              |

SovereignCloudStack / issues