[occm] Simple external LB fails on OVH, `not a valid external network` error

aasseman commented 9 months ago

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: Tried this example: https://github.com/kubernetes/cloud-provider-openstack/blob/8a156e543ca44924a5f26aaf001fb86bcbd100f9/examples/loadbalancers/external-http-nginx.yaml. The load balancer gets created, but gets stuck before allocating a floating IP with error:

E1208 01:19:17.756124      11 controller.go:298] error processing service default/external-http-nginx-service (retrying with exponential backoff): failed to ensure load balancer: error creating LB floatingip: Bad request with: [POST https://network.us-east-va-1.cloud.ovh.us/v2.0/floatingips], error message: {"NeutronError": {"type": "BadRequest", "message": "Bad floatingip request: Network 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 is not a valid external network.", "detail": ""}}
I1208 01:19:17.756223      11 event.go:307] "Event occurred" object="default/external-http-nginx-service" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error creating LB floatingip: Bad request with: [POST https://network.us-east-va-1.cloud.ovh.us/v2.0/floatingips], error message: {\"NeutronError\": {\"type\": \"BadRequest\", \"message\": \"Bad floatingip request: Network 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 is not a valid external network.\", \"detail\": \"\"}}"

Created load balancer:

(openstack) loadbalancer show 67ba6c0b-802a-40a8-8ace-edf61342fa07
+---------------------+-----------------------------------------------------------------------------------------+
| Field               | Value                                                                                   |
+---------------------+-----------------------------------------------------------------------------------------+
| admin_state_up      | True                                                                                    |
| availability_zone   | None                                                                                    |
| created_at          | 2023-12-08T00:42:17                                                                     |
| description         | Kubernetes external service default/external-http-nginx-service from cluster kubernetes |
| flavor_id           | cfb937f6-ffe9-490f-8e7a-65a4dd1fcf00                                                    |
| id                  | 67ba6c0b-802a-40a8-8ace-edf61342fa07                                                    |
| listeners           | d0facb60-fd8b-4482-b8e4-dd43eccc9e9f                                                    |
| name                | kube_service_kubernetes_default_external-http-nginx-service                             |
| operating_status    | ONLINE                                                                                  |
| pools               | 26586b4b-29c3-4605-8137-8a9e2718d0be                                                    |
| project_id          | <redacted>                                                                              |
| provider            | amphora                                                                                 |
| provisioning_status | ACTIVE                                                                                  |
| updated_at          | 2023-12-08T00:43:43                                                                     |
| vip_address         | 192.168.168.230                                                                         |
| vip_network_id      | 0b428775-e708-4c2f-a652-e192f756bb42                                                    |
| vip_port_id         | 10967e37-22e1-4c2c-8785-524c3fc492c8                                                    |
| vip_qos_policy_id   | None                                                                                    |
| vip_subnet_id       | 37982fe5-2126-453d-8ca7-54dd3c8383a3                                                    |
| tags                | kube_service_kubernetes_default_external-http-nginx-service                             |
| additional_vips     | []                                                                                      |
+---------------------+-----------------------------------------------------------------------------------------+

What you expected to happen: I expected the external load balancer example to work, since I can also create an external LB manually using the OpenStack CLI without issues.

How to reproduce it: Create a k8s cluster in OVH OpenStack, install [occm] using the instructions in https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md, then try https://github.com/kubernetes/cloud-provider-openstack/blob/8a156e543ca44924a5f26aaf001fb86bcbd100f9/examples/loadbalancers/external-http-nginx.yaml.

Anything else we need to know?: Things I've tried and work:

The internal load balancer example
Setting up an external load balancer on OVH using the OpenStack CLI, using the simple example here: https://help.ovhcloud.com/csm/en-ie-public-cloud-network-octavia-use-lbaas-openstack?id=kb_article_view&sysparm_article=KB0050296 (but can't use it in a k8s service through [occm])
Creating an internal load balancer using [occm], then attaching a FIP to it using the openstack CLI

I unsuccessfully tried creating an LB service through [occm]:

With an existing FIP through loadBalancerIP
With an existing external LB created with the CLI, through loadbalancer.openstack.org/load-balancer-id

Environment:

openstack-cloud-controller-manager(or other related binary) version: 1.28.0

OpenStack version:

(openstack) configuration show
+---------------------------------------------+----------------------------------+
| Field                                       | Value                            |
+---------------------------------------------+----------------------------------+
| additional_user_agent                       | [('osc-lib', '2.7.0')]           |
| api_timeout                                 | None                             |
| auth.application_credential_id              |  <redacted>                      |
| auth.application_credential_secret          | <redacted>                       |
| auth.auth_url                               | https://auth.cloud.ovh.us/       |
| auth_type                                   | v3applicationcredential          |
| baremetal_introspection_status_code_retries | 5                                |
| baremetal_status_code_retries               | 5                                |
| beta_command                                | False                            |
| cacert                                      | None                             |
| cert                                        | None                             |
| cloud                                       | openstack                        |
| default_domain                              | default                          |
| deferred_help                               | False                            |
| disable_vendor_agent                        | {}                               |
| floating_ip_source                          | neutron                          |
| identity_api_version                        | 3                                |
| image_api_use_tasks                         | False                            |
| image_format                                | qcow2                            |
| image_status_code_retries                   | 5                                |
| interface                                   | public                           |
| key                                         | None                             |
| loadbalancer_api_version                    | 2.0                              |
| message                                     |                                  |
| network_api_version                         | 2                                |
| networks                                    | []                               |
| object_store_api_version                    | 1                                |
| region_name                                 | US-WEST-OR-1                     |
| secgroup_source                             | neutron                          |
| status                                      | active                           |
| timing                                      | False                            |
| verbose_level                               | 1                                |
| verify                                      | True                             |
+---------------------------------------------+----------------------------------+

OpenStack CLI client version: 6.2.0

jichenjc commented 9 months ago

@dulek can you help provide some insight here ?

I tried this locally with devstack env and it works fine for the LB case IIRC so not sure it's due to openstack different settings or configurations..

kayrus commented 9 months ago

@aasseman it looks like your private network router is associated with a different FIP network/subnet. In order to confirm this, try to run the commands below:

export NET_ID="$(openstack loadbalancer show 67ba6c0b-802a-40a8-8ace-edf61342fa07 -f value -c vip_network_id)"
export SUBNET_ID="$(openstack loadbalancer show 67ba6c0b-802a-40a8-8ace-edf61342fa07 -f value -c vip_subnet_id)"
export PORT_ID="$(openstack port list --network $NET_ID --device-owner network:router_interface --fixed-ip subnet=$SUBNET_ID -f value -c id)"
export ROUTER_ID="$(openstack port show $PORT_ID -f value -c device_id)"
export VIP_SUBNET_ID="$(openstack router show $ROUTER_ID -f json -c external_gateway_info | jq -r '.external_gateway_info.external_fixed_ips[]|.subnet_id')"
openstack subnet show $VIP_SUBNET_ID -f value -c cidr

and compare this value with the FIP you're trying to address.

For additional API debug logs try to set the OCCM verbosity to 10 with --v=10.

aasseman commented 9 months ago

Yes the subnets are different:

The router's VIP_SUBNET_IP is indeed external (with a public CIDR)
The LB's SUBNET_ID is internal. It corresponds to the subnet-id setting I supplied in the cloud.conf secret.

If I try setting subnet-id to an external/public one, I get:

1213 01:55:43.885576      11 loadbalancer.go:1925] "Creating loadbalancer" lbName="kube_service_kubernetes_default_external-http-nginx-service" service="default/external-http-nginx-service"
E1213 01:55:46.175817      11 controller.go:298] error processing service default/external-http-nginx-service (retrying with exponential backoff): failed to ensure load balancer: error creating loadbalancer kube_service_kubernetes_default_external-http-nginx-service: error creating loadbalancer {"name":"kube_service_kubernetes_default_external-http-nginx-service","description":"Kubernetes external service default/external-http-nginx-service from cluster kubernetes","vip_subnet_id":"8bb34217-736f-4c97-b58d-dcd7b2b2a4fd","provider":"amphora","listeners":[{"protocol":"TCP","protocol_port":80,"name":"listener_0_kube_service_kubernetes_default_external-http-nginx-service","default_pool":{"lb_algorithm":"ROUND_ROBIN","protocol":"TCP","name":"pool_0_kube_service_kubernetes_default_external-http-nginx-service","members":[{"address":"192.168.1.20","protocol_port":31538,"name":"k8s-worker-0","subnet_id":"8bb34217-736f-4c97-b58d-dcd7b2b2a4fd"}]},"connection_limit":-1,"timeout_client_data":50000,"timeout_member_data":50000,"timeout_member_connect":5000,"timeout_tcp_inspect":0,"allowed_cidrs":["0.0.0.0/0"],"tags":["kube_service_kubernetes_default_external-http-nginx-service"]}],"tags":["kube_service_kubernetes_default_external-http-nginx-service"]}: Bad request with: [POST https://load-balancer.us-west-or-1.cloud.ovh.us/v2.0/lbaas/loadbalancers], error message: {"faultcode": "Client", "faultstring": "Validation failure: Supplied VIP network_id is not allowed by the configuration of this deployment. Only ['vrack'] are allowed.", "debuginfo": null}
I1213 01:55:46.176128      11 event.go:307] "Event occurred" object="default/external-http-nginx-service" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: error creating loadbalancer kube_service_kubernetes_default_external-http-nginx-service: error creating loadbalancer {\"name\":\"kube_service_kubernetes_default_external-http-nginx-service\",\"description\":\"Kubernetes external service default/external-http-nginx-service from cluster kubernetes\",\"vip_subnet_id\":\"8bb34217-736f-4c97-b58d-dcd7b2b2a4fd\",\"provider\":\"amphora\",\"listeners\":[{\"protocol\":\"TCP\",\"protocol_port\":80,\"name\":\"listener_0_kube_service_kubernetes_default_external-http-nginx-service\",\"default_pool\":{\"lb_algorithm\":\"ROUND_ROBIN\",\"protocol\":\"TCP\",\"name\":\"pool_0_kube_service_kubernetes_default_external-http-nginx-service\",\"members\":[{\"address\":\"192.168.1.20\",\"protocol_port\":31538,\"name\":\"k8s-worker-0\",\"subnet_id\":\"8bb34217-736f-4c97-b58d-dcd7b2b2a4fd\"}]},\"connection_limit\":-1,\"timeout_client_data\":50000,\"timeout_member_data\":50000,\"timeout_member_connect\":5000,\"timeout_tcp_inspect\":0,\"allowed_cidrs\":[\"0.0.0.0/0\"],\"tags\":[\"kube_service_kubernetes_default_external-http-nginx-service\"]}],\"tags\":[\"kube_service_kubernetes_default_external-http-nginx-service\"]}: Bad request with: [POST https://load-balancer.us-west-or-1.cloud.ovh.us/v2.0/lbaas/loadbalancers], error message: {\"faultcode\": \"Client\", \"faultstring\": \"Validation failure: Supplied VIP network_id is not allowed by the configuration of this deployment. Only ['vrack'] are allowed.\", \"debuginfo\": null}"

In short, OVH says that only an internal subnet is allowed.

Nevertheless, using a private subnet for the LB seems to be a correct way to do things, and does correspond to OVH's docs I linked. Indeed I can still attach a public FIP to it (manually), and get public connectivity to my services.

aasseman commented 9 months ago

PS: 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 from the error messages I posted in the first message is an external floating network, contrary to the what the error message implies. I can use it to allocate public FIPs no problem:

$ openstack floating ip create 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| created_at          | 2023-12-13T02:15:19Z                 |
| description         |                                      |
| dns_domain          | None                                 |
| dns_name            | None                                 |
| fixed_ip_address    | None                                 |
| floating_ip_address | <Redacted>                           |
| floating_network_id | 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 |
| id                  | 280790e6-172d-4849-b1b1-8717d86293e4 |
| name                | <Redacted>                           |
| port_details        | None                                 |
| port_id             | None                                 |
| project_id          | 86910a0db88a45bf91bb7bc2dd17c7d8     |
| qos_policy_id       | None                                 |
| revision_number     | 0                                    |
| router_id           | None                                 |
| status              | DOWN                                 |
| subnet_id           | None                                 |
| tags                | []                                   |
| updated_at          | 2023-12-13T02:15:19Z                 |
+---------------------+--------------------------------------+

aasseman commented 9 months ago

Sorry, I'm dumping even more information.

I ran [occm] with --v 10, in the 1st scenario where the LBs subnet-id is private:

I1213 02:27:45.717699       9 client.go:128] X-Auth-Token: ***
I1213 02:27:45.717916       9 client.go:128] OpenStack Request Body: {
I1213 02:27:45.717950       9 client.go:128]   "floatingip": {
I1213 02:27:45.717991       9 client.go:128]     "description": "Floating IP for Kubernetes external service default/external-http-nginx-service from cluster kubernetes",
I1213 02:27:45.718027       9 client.go:128]     "floating_network_id": "9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14",
I1213 02:27:45.718047       9 client.go:128]     "port_id": "adb3cb47-d614-49c8-927b-b7443357ba66"
I1213 02:27:45.718094       9 client.go:128]   }
I1213 02:27:45.718128       9 client.go:128] }
I1213 02:27:45.888336       9 client.go:128] OpenStack Response Code: 400
I1213 02:27:45.888465       9 client.go:128] OpenStack Response Headers:
I1213 02:27:45.888487       9 client.go:128] Content-Length: 170
I1213 02:27:45.888492       9 client.go:128] Content-Type: application/json
I1213 02:27:45.888496       9 client.go:128] Date: Wed, 13 Dec 2023 02:27:45 GMT
I1213 02:27:45.888500       9 client.go:128] Server: Apache
I1213 02:27:45.888503       9 client.go:128] Strict-Transport-Security: max-age=15768000
I1213 02:27:45.888506       9 client.go:128] X-Iplb-Instance: 547
I1213 02:27:45.888509       9 client.go:128] X-Iplb-Request-Id: 0FCC1EB8:8406_0FCC92F5:01BB_657916A1_3F7524:15B90
I1213 02:27:45.888513       9 client.go:128] X-Openstack-Request-Id: req-ccd97347-f171-4757-91ac-4ca6d72f1cde
I1213 02:27:45.888703       9 client.go:128] OpenStack Response Body: {
I1213 02:27:45.888774       9 client.go:128]   "NeutronError": {
I1213 02:27:45.888813       9 client.go:128]     "detail": "",
I1213 02:27:45.888872       9 client.go:128]     "message": "Bad floatingip request: Network 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 is not a valid external network.",
I1213 02:27:45.888909       9 client.go:128]     "type": "BadRequest"
I1213 02:27:45.888929       9 client.go:128]   }
I1213 02:27:45.888967       9 client.go:128] }
I1213 02:27:45.889067       9 controller.go:839] Finished syncing service "default/external-http-nginx-service" (700.026842ms)
E1213 02:27:45.889127       9 controller.go:298] error processing service default/external-http-nginx-service (retrying with exponential backoff): failed to ensure load balancer: error creating LB floatingip: Bad request with: [POST https://network.us-east-va-1.cloud.ovh.us/v2.0/floatingips], error message: {"NeutronError": {"type": "BadRequest", "message": "Bad floatingip request: Network 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 is not a valid external network.", "detail": ""}}

I ran the equivalent request in the CLI, and it completed successfully:

$ openstack floating ip create 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 --port adb3cb47-d614-49c8-927b-b7443357ba66
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field               | Value                                                                                                                                                                                                                                                                               |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at          | 2023-12-13T02:29:44Z                                                                                                                                                                                                                                                                |
| description         |                                                                                                                                                                                                                                                                                     |
| dns_domain          | None                                                                                                                                                                                                                                                                                |
| dns_name            | None                                                                                                                                                                                                                                                                                |
| fixed_ip_address    | 192.168.1.102                                                                                                                                                                                                                                                                       |
| floating_ip_address | <redacted public IP>                                                                                                                                                                                                                                                                        |
| floating_network_id | 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14                                                                                                                                                                                                                                                |
| id                  | 1c2fb1dc-29a2-4845-be38-6db9f3bf3652                                                                                                                                                                                                                                                |
| name                | <redacted public IP>                                                                                                                                                                                                                                                                        |
| port_details        | {'name': 'octavia-lb-a8969a3b-a41c-476f-bc7d-ac533ecac1f0', 'network_id': '9da4bc6a-13cf-49b8-8caa-62ef0745d7a4', 'mac_address': 'fa:16:3e:46:3e:80', 'admin_state_up': False, 'status': 'DOWN', 'device_id': 'lb-a8969a3b-a41c-476f-bc7d-ac533ecac1f0', 'device_owner': 'Octavia'} |
| port_id             | adb3cb47-d614-49c8-927b-b7443357ba66                                                                                                                                                                                                                                                |
| project_id          | 86910a0db88a45bf91bb7bc2dd17c7d8                                                                                                                                                                                                                                                    |
| qos_policy_id       | None                                                                                                                                                                                                                                                                                |
| revision_number     | 0                                                                                                                                                                                                                                                                                   |
| router_id           | 4f5eb77b-d840-46b6-9683-5d75334d6d11                                                                                                                                                                                                                                                |
| status              | DOWN                                                                                                                                                                                                                                                                                |
| subnet_id           | None                                                                                                                                                                                                                                                                                |
| tags                | []                                                                                                                                                                                                                                                                                  |
| updated_at          | 2023-12-13T02:29:44Z                                                                                                                                                                                                                                                                |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

kayrus commented 9 months ago

@aasseman can you compare with the API call submitted by openstack CLI? Use the --debug flag.

UPD: I checked the --debug locally, the API calls are the same (except the description field, can you run CLI with a description?). And the only clue is that the credentials used by OCCM are different to credentials used by CLI.

UPD2: https://github.com/openstack/neutron/blob/dcccd7cabe9fa959c88b477e55ad47b815d52518/neutron/db/l3_db.py#L1486-L1488 +1 to the different credentials clue

aasseman commented 9 months ago

Trying again using the same credentials on both sides: still fails, then I proceed to successfully do it from the CLI while also giving it a description:

I1213 19:13:44.392353      10 loadbalancer.go:976] Creating floating IP  for loadbalancer 01b9443d-6184-42c1-8d2f-430f02f287cf
I1213 19:13:44.392367      10 loadbalancer.go:861] Creating floating ip with opts {Description:Floating IP for Kubernetes external service default/external-http-nginx-service from cluster kubernetes FloatingNetworkID:9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 FloatingIP: PortID:6d22c458-ae4c-4ebc-b041-e8be2eaf6bd3 FixedIP: SubnetID: TenantID: ProjectID:}
I1213 19:13:44.392615      10 client.go:128] OpenStack Request URL: POST https://network.us-east-va-1.cloud.ovh.us/v2.0/floatingips
I1213 19:13:44.392748      10 client.go:128] OpenStack Request Headers:
I1213 19:13:44.392767      10 client.go:128] Accept: application/json
I1213 19:13:44.392773      10 client.go:128] Content-Type: application/json
I1213 19:13:44.392778      10 client.go:128] User-Agent: openstack-cloud-controller-manager/v1.28.0 gophercloud/v1.4.0
I1213 19:13:44.392784      10 client.go:128] X-Auth-Token: ***
I1213 19:13:44.392958      10 client.go:128] OpenStack Request Body: {
I1213 19:13:44.392967      10 client.go:128]   "floatingip": {
I1213 19:13:44.392985      10 client.go:128]     "description": "Floating IP for Kubernetes external service default/external-http-nginx-service from cluster kubernetes",
I1213 19:13:44.392991      10 client.go:128]     "floating_network_id": "9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14",
I1213 19:13:44.392996      10 client.go:128]     "port_id": "6d22c458-ae4c-4ebc-b041-e8be2eaf6bd3"
I1213 19:13:44.393001      10 client.go:128]   }
I1213 19:13:44.393006      10 client.go:128] }
I1213 19:13:44.566877      10 client.go:128] OpenStack Response Code: 400
I1213 19:13:44.567121      10 client.go:128] OpenStack Response Headers:
I1213 19:13:44.567169      10 client.go:128] Content-Length: 170
I1213 19:13:44.567200      10 client.go:128] Content-Type: application/json
I1213 19:13:44.567220      10 client.go:128] Date: Wed, 13 Dec 2023 19:13:44 GMT
I1213 19:13:44.567251      10 client.go:128] Server: Apache
I1213 19:13:44.567272      10 client.go:128] Strict-Transport-Security: max-age=15768000
I1213 19:13:44.567305      10 client.go:128] X-Iplb-Instance: 546
I1213 19:13:44.567326      10 client.go:128] X-Iplb-Request-Id: 0FCC1EB8:9F16_0FCC92F5:01BB_657A0268_3C5DAF:2C3CE
I1213 19:13:44.567348      10 client.go:128] X-Openstack-Request-Id: req-3ecf4618-5f85-4a6a-bad9-f165cb0547c9
I1213 19:13:44.567517      10 client.go:128] OpenStack Response Body: {
I1213 19:13:44.567593      10 client.go:128]   "NeutronError": {
I1213 19:13:44.567639      10 client.go:128]     "detail": "",
I1213 19:13:44.567662      10 client.go:128]     "message": "Bad floatingip request: Network 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14 is not a valid external network.",
I1213 19:13:44.567684      10 client.go:128]     "type": "BadRequest"
I1213 19:13:44.567705      10 client.go:128]   }
I1213 19:13:44.567726      10 client.go:128] }
I1213 19:13:44.567818      10 controller.go:839] Finished syncing service "default/external-http-nginx-service" (992.626932ms)
E1213 19:13:44.567872      10 controller.go:298] error processing service default/external-http-nginx-service (retrying with exponential backoff): failed to ensure load balancer: error creating LB floatingip: Bad request with: [POST https://network.us-east-va-1.cloud.ovh.us/v2.0/floatingips], error message: {"NeutronError": {"type": "BadRequest", "message": "Bad floatingip request: Network 9fb20ca2-e0ff-41f5-a33c-eb399

$ openstack --debug floating ip create --port 6d22c458-ae4c-4ebc-b041-e8be2eaf6bd3 --description "Floating IP for Kubernetes external service default/external-http-nginx-service from cluster kubernetes" 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14
[...]
REQ: curl -g -i -X POST https://network.us-west-or-1.cloud.ovh.us/v2.0/floatingips -H "Content-Type: application/json" -H "User-Agent: openstacksdk/1.0.1 keystoneauth1/5.1.2 python-requests/2.28.2 CPython/3.12.0" -H "X-Auth-Token: <REDACTED>" -d '{"floatingip": {"port_id": "6d22c458-ae4c-4ebc-b041-e8be2eaf6bd3", "floating_network_id": "9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14", "description": "Floating IP for Kubernetes external service default/external-http-nginx-service from cluster kubernetes"}}'
https://network.us-west-or-1.cloud.ovh.us:443 "POST /v2.0/floatingips HTTP/1.1" 201 968
RESP: [201] Content-Length: 968 Content-Type: application/json Date: Wed, 13 Dec 2023 19:14:43 GMT Server: Apache Strict-Transport-Security: max-age=15768000 X-IPLB-Instance: 291 X-IPLB-Request-ID: 49CACE82:B94C_335189D0:01BB_657A02A3_3066372:168FA x-openstack-request-id: req-54c81d82-05c8-478a-afaa-c9771557f842
RESP BODY: {"floatingip": {"id": "44c6d786-6f08-48d7-b7b5-3a1aa54198dc", "tenant_id": "86910a0db88a45bf91bb7bc2dd17c7d8", "floating_ip_address": "15.204.30.3", "floating_network_id": "9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14", "router_id": "4f5eb77b-d840-46b6-9683-5d75334d6d11", "port_id": "6d22c458-ae4c-4ebc-b041-e8be2eaf6bd3", "fixed_ip_address": "192.168.1.91", "status": "DOWN", "project_id": "86910a0db88a45bf91bb7bc2dd17c7d8", "description": "Floating IP for Kubernetes external service default/external-http-nginx-service from cluster kubernetes", "qos_policy_id": null, "port_details": {"name": "octavia-lb-01b9443d-6184-42c1-8d2f-430f02f287cf", "network_id": "9da4bc6a-13cf-49b8-8caa-62ef0745d7a4", "mac_address": "fa:16:3e:c5:c6:f9", "admin_state_up": false, "status": "DOWN", "device_id": "lb-01b9443d-6184-42c1-8d2f-430f02f287cf", "device_owner": "Octavia"}, "tags": [], "created_at": "2023-12-13T19:14:44Z", "updated_at": "2023-12-13T19:14:44Z", "revision_number": 0}}
POST call to network for https://network.us-west-or-1.cloud.ovh.us/v2.0/floatingips used request id req-54c81d82-05c8-478a-afaa-c9771557f842
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field               | Value                                                                                                                                                                                                                                                                               |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at          | 2023-12-13T19:14:44Z                                                                                                                                                                                                                                                                |
| description         | Floating IP for Kubernetes external service default/external-http-nginx-service from cluster kubernetes                                                                                                                                                                             |
| dns_domain          | None                                                                                                                                                                                                                                                                                |
| dns_name            | None                                                                                                                                                                                                                                                                                |
| fixed_ip_address    | 192.168.1.91                                                                                                                                                                                                                                                                        |
| floating_ip_address | <redacted public IP>                                                                                                                                                                                                                                                                         |
| floating_network_id | 9fb20ca2-e0ff-41f5-a33c-eb3993f2bc14                                                                                                                                                                                                                                                |
| id                  | 44c6d786-6f08-48d7-b7b5-3a1aa54198dc                                                                                                                                                                                                                                                |
| name                | <redacted public ip>                                                                                                                                                                                                                                                                         |
| port_details        | {'name': 'octavia-lb-01b9443d-6184-42c1-8d2f-430f02f287cf', 'network_id': '9da4bc6a-13cf-49b8-8caa-62ef0745d7a4', 'mac_address': 'fa:16:3e:c5:c6:f9', 'admin_state_up': False, 'status': 'DOWN', 'device_id': 'lb-01b9443d-6184-42c1-8d2f-430f02f287cf', 'device_owner': 'Octavia'} |
| port_id             | 6d22c458-ae4c-4ebc-b041-e8be2eaf6bd3                                                                                                                                                                                                                                                |
| project_id          | 86910a0db88a45bf91bb7bc2dd17c7d8                                                                                                                                                                                                                                                    |
| qos_policy_id       | None                                                                                                                                                                                                                                                                                |
| revision_number     | 0                                                                                                                                                                                                                                                                                   |
| router_id           | 4f5eb77b-d840-46b6-9683-5d75334d6d11                                                                                                                                                                                                                                                |
| status              | DOWN                                                                                                                                                                                                                                                                                |
| subnet_id           | None                                                                                                                                                                                                                                                                                |
| tags                | []                                                                                                                                                                                                                                                                                  |
| updated_at          | 2023-12-13T19:14:44Z                                                                                                                                                                                                                                                                |
+---------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
clean_up CreateFloatingIP: 
END return value: 0

kayrus commented 9 months ago

@aasseman credentials are wrong. in OCCM you're using us-east-va-1 region. In CLI you're using us-west-or-1 region.

aasseman commented 9 months ago

:man_facepalming: yes indeed.... The symptoms were slightly misleading as I'd have expected a total failure instead of a partial one.

I indeed omitted the region param for cloud.conf. It'd be nice to have [occm] crash in that case, since it's a required param.

Anyway, sorry for wasting your time on this, this is a really dumb mistake...

kayrus commented 9 months ago

hm. the region is not a required parameter, it's omitted quite frequently. usually the keystone URL is the primary one, but not in case when keystone is federated across multiple regions (OVH case). however adding a region as a required parameter may cause other consequences.

kubernetes / cloud-provider-openstack

[occm] Simple external LB fails on OVH, `not a valid external network` error #2493