Open josephineSei opened 6 months ago
Going a little bit back in time in the OpenStack project, Networking was also done by Nova. It was eventually taken out and brought to a new project: Neutron (first Name: Quantum) but remnants remained in Nova. So it was for quite some time thath Nova and Neutron shared the responsibility for creating and attaching a Floating IP to the correct port. Not too long ago that was also excluded from Nova and implemented in Neutron alone.
And that is a problem.
openstack floating ip create
gives me a (randomly assigned) IP from the pool, that I can freely use in my project.openstack server add floating ip
attaches a free floating IP to the given VMopenstack server remove floating ip
detaches the given floating IP from the given server.openstack floating ip delete
"deletes" an IP == i can and will no use this IP anymoreThese steps explain what a user is expecting to do with floating ips, even though the create
and delete
may be a bit confusing.
This was also the behavior, when Nova was still responsible for Floating IPs
using openstack floating ip create
allocates an IP from the provider networks pool for the project.
using openstack server add floating ip
attaches a ANY floating IP allocated for the project to the given VM
$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server | ACTIVE | client-internal=192.168.10.79 | | S |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107, 10.54.8.58 | | S |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
$ openstack floating ip list
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
| 08600d00-cd66-48e4-9e69-9e6a5c932ef1 | 10.54.8.206 | None | None | 0d71b88b-049c-4cbc-b9b6-863aef0ccac4 | 02542d0fcee94e99a825ba98f0804296 |
| 4a89a886-a47b-46d3-95cd-fb38820f56b3 | 10.54.8.171 | None | None | 0d71b88b-049c-4cbc-b9b6-863aef0ccac4 | 02542d0fcee94e99a825ba98f0804296 |
| d8e827d0-4446-4217-9f3a-d30d10c74653 | 10.54.8.220 | None | None | 0d71b88b-049c-4cbc-b9b6-863aef0ccac4 | 02542d0fcee94e99a825ba98f0804296 |
| f282badb-a1b6-4269-9783-5df9ce2fccc9 | 10.54.8.58 | 192.168.10.107 | 85921f34-8821-409c-80e8-81093e1e805d | 0d71b88b-049c-4cbc-b9b6-863aef0ccac4 | 02542d0fcee94e99a825ba98f0804296 |
+--------------------------------------+---------------------+------------------+--------------------------------------+--------------------------------------+----------------------------------+
$ openstack server add floating ip test-server 10.54.8.58
$ openstack server list
+--------------------------------------+-----------------+--------+-------------------------------------------+-------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------+--------+-------------------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server | ACTIVE | client-internal=192.168.10.79, 10.54.8.58 | | S |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107 | | S |
+--------------------------------------+-----------------+--------+-------------------------------------------+-------+--------+
Due to Neutron taking over the complete process the check whether a floating IP is already associated to a VM was skipped. This leaves us with this "stealing" behavior.
openstack server remove floating ip
detaches the given floating IP from the $ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server | ACTIVE | client-internal=192.168.10.79, 10.54.8.206 | | S |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107, 10.54.8.58 | | S |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
$ openstack server remove floating ip test-server 10.54.8.58
$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server | ACTIVE | client-internal=192.168.10.79, 10.54.8.206 | | S |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107 | | S |
+--------------------------------------+-----------------+--------+--------------------------------------------+-------+--------+
$ openstack server remove floating ip this-server-does-not-exist 10.54.8.206
$ openstack server list
+--------------------------------------+-----------------+--------+--------------------------------+-------+--------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------------+--------+--------------------------------+-------+--------+
| 8858fd79-2878-4003-b442-0b52a4f0413a | test-server | ACTIVE | client-internal=192.168.10.79 | | S |
| bfcfbb8f-9714-43ad-bdb9-d2671cd3dc85 | client-training | ACTIVE | client-internal=192.168.10.107 | | S |
+--------------------------------------+-----------------+--------+--------------------------------+-------+--------+
A combination from removing a floating Ip from a VM (step 3) and deleting the floating IP from the project (step 4) would lead in the worst case to a DOS attack on a VM. Because after deleting a floating IP, it cannot be ensured for a normal user to get the same IP address
$ source openrc admin admin
$ openstack floating ip list
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
| ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project |
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
| 1875754d-7b9f-47c2-9c0d- | 172.24.4.210 | 10.0.0.45 | d8387e3b-3b19-444a-9983- | 73edb86b-d7ab-4db3-82b7- | 15f2ab0eaa5b4372b759bde609e86224 |
| 83eafd1a0a76 | | | 42b61b3d19c1 | 25fa8b012e40 | |
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
$ openstack floating ip create --floating-ip-address 172.24.4.222 public
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| created_at | 2024-03-13T08:48:54Z |
| description | |
| dns_domain | |
| dns_name | |
| fixed_ip_address | None |
| floating_ip_address | 172.24.4.222 |
| floating_network_id | 73edb86b-d7ab-4db3-82b7-25fa8b012e40 |
| id | 3978a1f6-3af8-432f-978a-c7feafd88057 |
| name | 172.24.4.222 |
| port_details | None |
| port_id | None |
| project_id | 15f2ab0eaa5b4372b759bde609e86224 |
| qos_policy_id | None |
| revision_number | 0 |
| router_id | None |
| status | DOWN |
| subnet_id | None |
| tags | [] |
| updated_at | 2024-03-13T08:48:54Z |
+---------------------+--------------------------------------+
$ openstack floating ip list
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
| ID | Floating IP Address | Fixed IP Address | Port | Floating Network | Project |
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
| 1875754d-7b9f-47c2-9c0d- | 172.24.4.210 | 10.0.0.45 | d8387e3b-3b19-444a-9983- | 73edb86b-d7ab-4db3-82b7- | 15f2ab0eaa5b4372b759bde609e86224 |
| 83eafd1a0a76 | | | 42b61b3d19c1 | 25fa8b012e40 | |
| 3978a1f6-3af8-432f-978a- | 172.24.4.222 | None | None | 73edb86b-d7ab-4db3-82b7- | 15f2ab0eaa5b4372b759bde609e86224 |
| c7feafd88057 | | | | 25fa8b012e40 | |
+------------------------------------+---------------------+------------------+------------------------------------+------------------------------------+----------------------------------+
$ source openrc demo demo
$ openstack floating ip list
$ openstack floating ip create --floating-ip-address 172.24.4.222 public
Error while executing command: ForbiddenException: 403, (rule:create_floatingip and rule:create_floatingip:floating_ip_address) is disallowed by policy
$ openstack floating ip create public
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| created_at | 2024-03-13T08:49:51Z |
| description | |
| dns_domain | |
| dns_name | |
| fixed_ip_address | None |
| floating_ip_address | 172.24.4.155 |
| floating_network_id | 73edb86b-d7ab-4db3-82b7-25fa8b012e40 |
| id | 0f340eb1-74c7-4cc0-8495-8f648ff7bc61 |
| name | 172.24.4.155 |
| port_details | None |
| port_id | None |
| project_id | f58edaee60ad484facd2436d31d9caff |
| qos_policy_id | None |
| revision_number | 0 |
| router_id | None |
| status | DOWN |
| subnet_id | None |
| tags | [] |
| updated_at | 2024-03-13T08:49:51Z |
+---------------------+--------------------------------------+
$ source openrc admin admin
$ openstack floating ip create --floating-ip-address 172.24.4.155 public
Error while executing command: ConflictException: 409, IP address 172.24.4.155 already allocated in subnet 3e0206bc-53c8-44ca-a0f1-2c2548bba766
openstack floating ip delete
"deletes" an IP == i can and will no use this IP anymore
I looked a bit through nova code, wrote a gist and put this whole topic on the PTG plan for Neutron. I think it should be discussed there how to handle this whole issue.
After discussing this with Neutron I was asked to file 2 Bugs. One for each workflow:
Including detailed Requests from the CLI. So I am currently reproducing both cases and will describe them with the whole debug output.
After that I should ask the Nova people to look over this and they should decide (from Neutrons perspective) how to proceed, and which behavior they would like.
Here are the new bug reports: https://bugs.launchpad.net/neutron/+bug/2060808 https://bugs.launchpad.net/nova/+bug/2060812
After looking through the reported bugs:
There were no discussion about these bugs in the last weeks team meetings. The IRC Nova meeting is held on Tuesday 16 UTC - unfortunately I will not be able to attend this, but I will try to reach out to the Nova team to discuss the "new" behavior.
I attended the Nova IRC meeting and asked about the bugs: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting They wanted to have some time looking into it and a more detailed discussion about this. So I will try to contact them in their IRC channel at a different time (UTC afternoon would work)
The current Floating Ip behavior has two downsides:
There are many ways to deal with this behavior: