search

osism / issues

This repository is used for bug reports that are cross-project or not bound to a specific repository (or to an unknown repository).
https://www.osism.tech
1 stars 1 forks source link

7.0.0[bc]: octavia (ovn) does not find existing subnet #890

Closed garloff closed 7 months ago

garloff commented 8 months ago

Testing on a CiaB system with 7.0.0b and openstack-health-monitor:

New JHSUBNET: aeefbfec-fb5c-4b91-8fee-6cd99562f4bb
New SUBNET: dcf3b761-b3c8-44b7-b9d7-26b1186c1762  33e70699-87ae-4b21-8907-772497a8d2a9
[...]
New LBAAS:  on ciab/APIMonitor_1709379467/85 on deb12: 
[...]
ERROR: openstack loadbalancer create --vip-subnet-id aeefbfec-fb5c-4b91-8fee-6cd99562f4bb --name APIMonitor_1709379467_LB_0 --provider ovn => 1 Subnet aeefbfec-fb5c-4b91-8fee-6cd99562f4bb not found. (HTTP 400) (Request-ID: req-0dd11ee4-6ae2-44c7-9540-d069712d3e49)
ERROR: LBAAS creation failed

Clearly the subnet exists and belong to the project. Somehow, the OVN loadbalancer does not seem to accept tenant subnets. Which it obviously should. And did before.

garloff commented 8 months ago

This might be specific to the CiaB setup, where amphorae are disabled, but maybe the removal -octavia_network_type: tenant was too much to achieve this?

garloff commented 8 months ago

The upgrade to 7.0.0c did not help.

OK, apparently, octavia does not accept the TLS certiificate from neutron:

2024-03-08 17:33:05.683 733 ERROR octavia.network.drivers.neutron.base     raise exceptions.SSLError(msg)
2024-03-08 17:33:05.683 733 ERROR octavia.network.drivers.neutron.base keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://api.in-a-box.cloud:9696/v2.0/subnets/01b868dc-92be-4e77-8daa-71cabc423347: HTTPSConnectionPool(host='api.in-a-box.cloud', port=9696): Max retries exceeded with url: /v2.0/subnets/01b868dc-92be-4e77-8daa-71cabc423347 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
2024-03-08 17:33:05.683 733 ERROR octavia.network.drivers.neutron.base
garloff commented 8 months ago

This is really strange. Inside the octavia_api container, the cert validation with openssl s_client succeeds:

echo | openssl s_client -connect api.in-a-box.cloud:9696  2>/dev/null
CONNECTED(00000003)                                                            
---                                                                            
Certificate chain                                                              
 0 s:                                                                          
   i:CN = OSISM Cloud-in-a-Box CA                                              
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256                       
   v:NotBefore: Nov 20 15:21:35 2022 GMT; NotAfter: Nov 17 16:21:35 2032 GMT   
 1 s:CN = OSISM Cloud-in-a-Box CA                                              
   i:CN = OSISM Cloud-in-a-Box CA                                              
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256                       
   v:NotBefore: Nov 20 16:21:20 2022 GMT; NotAfter: Nov 17 16:21:20 2032 GMT   
---                                                                            
[...]
subject=
issuer=CN = OSISM Cloud-in-a-Box CA
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3424 bytes and written 400 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
[...]

The /etc/octavia/octavia.conf file mentions ca_certificates_file = /etc/ssl/certs/ca-certificates.crt numerous times. That file exists and does have the trusted CA cert for in-a-box.cloud in there. So for whatever reason does octavia fail to consider it when talking to neutron? But why?

[neutron]
region_name = RegionOne
endpoint_type = internal
ca_certificates_file = /etc/ssl/certs/ca-certificates.crt
garloff commented 8 months ago

Still puzzled. octavia-api.log gives me:

2024-03-08 19:06:23.133 734 WARNING keystonemiddleware.auth_token [-] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.                                                                                                                                                                                                                                         
2024-03-08 19:06:35.522 733 WARNING openstack [None req-e9128b28-dc8a-4fbf-a8bc-566037af8969 - 26c88a620f834253ba7dc478af6ce12d - - default default] Disabling service 'block-storage': Encountered an exception attempting to process config for project 'cinder' (service type 'block-storage'): no such option valid_interfaces in group [cinder]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [cinder]
2024-03-08 19:06:35.523 733 WARNING openstack [None req-e9128b28-dc8a-4fbf-a8bc-566037af8969 - 26c88a620f834253ba7dc478af6ce12d - - default default] Disabling  service 'compute': Encountered an exception attempting to process config for project 'nova' (service type 'compute'): no such option valid_interfaces in group [nova]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [nova]
2024-03-08 19:06:35.523 733 WARNING openstack [None req-e9128b28-dc8a-4fbf-a8bc-566037af8969 - 26c88a620f834253ba7dc478af6ce12d - - default default] Disabling service 'image': Encountered an exception attempting to process config for project 'glance' (service type 'image'): no such option valid_interfaces in group [glance]: oslo_config.cfg.NoSuchOptError: no such option valid_interfaces in group [glance]
2024-03-08 19:06:35.561 733 ERROR octavia.network.drivers.neutron.base [None req-e9128b28-dc8a-4fbfa8bc-566037af8969 - 26c88a620f834253ba7dc478af6ce12d - - default default] Error retrieving subnet (subnet id: 98bb455b-387f-4d7c-a1e3-bdaf50b382ad.: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://api.in-a-box.cloud:9696/v2.0/subnets/98bb455b-387f-4d7c-a1e3-bdaf50b382ad: HTTPSConnectionPool(host='api.in-a-box.cloud', port=9696): Max retries exceeded with url: /v2.0/subnets/98bb455b-387f-4d7c-a1e3-bdaf50b382ad (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))

So I was reading through https://docs.openstack.org/octavia/latest/configuration/configref.html#neutron to also understand the warning. Adding a few cafile = /etc/ssl/.... settings and valid_interfaces = internal settings and restarting the octavia_api container does not change anything. (I can see in the config dump that the variables are read ... but the error messages don't go away.)

2024-03-08 19:10:20.284 733 INFO octavia.api.app [-] neutron.ca_certificates_file   = /etc/ssl/certs/ca-certificates.crt
2024-03-08 19:10:20.284 733 INFO octavia.api.app [-] neutron.cafile                 = /etc/ssl/certs/ca-certificates.crt
2024-03-08 19:10:20.286 733 INFO octavia.api.app [-] neutron.valid_interfaces       = ['internal']

Nova on the other hand ignores cafile and valid_interfaces. Looks like we are in an evil intermittent phase of a change? The errors still occur:

2024-03-08 19:10:20.913 734 ERROR octavia.network.drivers.neutron.base [None req-d2aa63e8-b05c-469c-9bbe-e629a0921b4e - 26c88a620f834253ba7dc478af6ce12d - - default default] Error retrieving subnet (subnet id: 98bb455b-387f-4d7c-a1e3-bdaf50b382ad.: keystoneauth1.exceptions.connection.SSLError: SSL exception connecting to https://api.in-a-box.cloud:9696/v2.0/subnets/98bb455b-387f-4d7c-a1e3-bdaf50b382ad: HTTPSConnectionPool(host='api.in-a-box.cloud', port=9696): Max retries exceeded with url: /v2.0/subnets/98bb455b-387f-4d7c-a1e3-bdaf50b382ad (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
garloff commented 8 months ago

And

[neutron]
insecure = True

in /etc/kolla/octavia-api/octavia.conf does not help either. This is getting more and more obscure.

garloff commented 8 months ago

So, when octavia-api talks to neutron, it does not seem to have its own configuration in mind. Except when I use endpoint_override = https://api.in-a-box.cloud:9696. Then it suddenly stops throwing those SSL errors. The error with the missing subnets remains :-( Similarly, setting an endpoint = https://api.in-a-box.cloud:8774/v2.1 for nova in the octavia.conf does make the nova, cinder, and glance (!!!) warnings go away. This is definitely strange behavior.

garloff commented 8 months ago

According to other testers of v7.0.0[bc], this does not affect full installations, so maybe the issue is limited to CiaB. (I have not looked at testbed yet.)

garloff commented 8 months ago

Hi Christian, OK, so we reenable the octavia amphora provider in CiaB deployments. I'd still like to have a working octavia ovn provider, obviously. Maybe, as a side-effect, we'll get this as well?

garloff commented 7 months ago

I upgraded to 7.0.0d. The not found subnet still happens, with both ovn and amphora provider. Also the disturbing log messages remain. (I can still fix them with cafile and endpoint_override, but that does not resolve the issue with not-found subnets.)

garloff commented 7 months ago

Upgrade to 7.0.0 did not make a difference: Still no access to the existing subnet (neither ovn nor amphora):

dragon@manager:/opt/configuration/environments/openstack$ openstack subnet list
+--------------------------------------+-----------------------------------+--------------------------------------+-----------------+
| ID                                   | Name                              | Network                              | Subnet          |
+--------------------------------------+-----------------------------------+--------------------------------------+-----------------+
| 98bb455b-387f-4d7c-a1e3-bdaf50b382ad | test-subnet                       | 7c7d9e93-5770-48c5-b29d-25d338d2831b | 192.168.64.0/24 |
| f9243dd5-59a0-4964-9d69-fefaa04820c7 | APIMonitor_1710528148_SUBNET_VM_0 | 14bba778-c773-4319-b797-2e44ba49cdf0 | 10.250.0.0/22   |
+--------------------------------------+-----------------------------------+--------------------------------------+-----------------+
dragon@manager:/opt/configuration/environments/openstack$ openstack loadbalancer create --vip-subnet-id 98bb455b-387f-4d7c-a1e3-bdaf50b382ad --name testlb --provider ovn
Subnet 98bb455b-387f-4d7c-a1e3-bdaf50b382ad not found. (HTTP 400) (Request-ID: req-bd00a62c-dbcc-465d-b06b-ab1018431e50)
dragon@manager:/opt/configuration/environments/openstack$ openstack loadbalancer create --vip-subnet-id 98bb455b-387f-4d7c-a1e3-bdaf50b382ad --name testlb --provider amphora
Subnet 98bb455b-387f-4d7c-a1e3-bdaf50b382ad not found. (HTTP 400) (Request-ID: req-e7ff6ada-ddbd-411a-8a7d-da66d388b81b)
dragon@manager:/opt/configuration/environments/openstack$

Same SSL exceptions in /var/log/kolla/octavia/octavia-api.log.

berendt commented 7 months ago

Test of latest images including the bugfix:

dragon@testbed-manager:~$ openstack --os-cloud test loadbalancer create --provider ovn --vip-subnet-id subnet-test --name test
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| availability_zone   | None                                 |
| created_at          | 2024-03-25T12:02:07                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 03f7df5f-4ba5-4eaf-b0f0-44e119ea1107 |
| listeners           |                                      |
| name                | test                                 |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 2b8f799708a44710855f5b6e1bdc99ed     |
| provider            | ovn                                  |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 192.168.200.251                      |
| vip_network_id      | 19eaf344-e5e7-42db-a1c8-b66aba552182 |
| vip_port_id         | a3ccaab9-53f7-476e-8314-54b51ef7da85 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 19d0fd77-da96-42f3-b4c9-b5b4ed42a9fa |
| tags                |                                      |
| additional_vips     | []                                   |
+---------------------+--------------------------------------+
dragon@testbed-manager:~$ openstack --os-cloud test loadbalancer list
+--------------------------------------+------+----------------------------------+-----------------+---------------------+------------------+----------+
| id                                   | name | project_id                       | vip_address     | provisioning_status | operating_status | provider |
+--------------------------------------+------+----------------------------------+-----------------+---------------------+------------------+----------+
| 03f7df5f-4ba5-4eaf-b0f0-44e119ea1107 | test | 2b8f799708a44710855f5b6e1bdc99ed | 192.168.200.251 | ACTIVE              | ONLINE           | ovn      |
+--------------------------------------+------+----------------------------------+-----------------+---------------------+------------------+----------+
berendt commented 7 months ago

Included in 7.0.1. Also tested in the cloud-in-a-box, works there as well.

dragon@manager:/opt/configuration/environments/openstack$ openstack --os-cloud test loadbalancer create --provider ovn --vip-subnet-id subnet-test --name test
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| availability_zone   | None                                 |
| created_at          | 2024-03-28T09:45:10                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 8c37ba47-9a23-4496-ab37-611c9f15f901 |
| listeners           |                                      |
| name                | test                                 |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | b31385a573fc44e489c8afb6109ee865     |
| provider            | ovn                                  |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 192.168.200.221                      |
| vip_network_id      | b8684359-35ca-4b08-9e3d-0001fd69bb78 |
| vip_port_id         | c33275fe-e2dd-4719-bce3-6f6b6ff189cb |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | e51bec68-5d9e-4179-b301-69ab7c247ff4 |
| vip_vnic_type       | normal                               |
| tags                |                                      |
| additional_vips     | []                                   |
+---------------------+--------------------------------------+
dragon@manager:/opt/configuration/environments/openstack$ openstack --os-cloud test loadbalancer list
+---------------------------+------+---------------------------+-----------------+---------------------+------------------+----------+
| id                        | name | project_id                | vip_address     | provisioning_status | operating_status | provider |
+---------------------------+------+---------------------------+-----------------+---------------------+------------------+----------+
| 8c37ba47-9a23-4496-ab37-  | test | b31385a573fc44e489c8afb61 | 192.168.200.221 | ACTIVE              | ONLINE           | ovn      |
| 611c9f15f901              |      | 09ee865                   |                 |                     |                  |          |
+---------------------------+------+---------------------------+-----------------+---------------------+------------------+----------+