search

kubernetes / cloud-provider-openstack

Apache License 2.0
619 stars 610 forks source link

`cinder-csi-plugin@v1.23.3` fails with "x509: certificate signed by unknown authority" #1948

Closed ialidzhikov closed 2 years ago

ialidzhikov commented 2 years ago

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: I am trying to update out cinder-csi-plugin version from v1.23.0 to v1.23.3 (for K8s 1.23 clusters). We are running docker.io/k8scloudprovider/cinder-csi-plugin@v1.23.0 without issues more than a year. When I tried v1.23.3, it fails with:

Flag --nodeid has been deprecated, This flag would be removed in future. Currently, the value is ignored by the driver
I0722 07:40:09.034911       1 driver.go:75] Driver: cinder.csi.openstack.org
I0722 07:40:09.034930       1 driver.go:76] Driver version: 2.0.0@v1.23.3
I0722 07:40:09.034933       1 driver.go:77] CSI Spec version: 1.3.0
I0722 07:40:09.034938       1 driver.go:107] Enabling controller service capability: LIST_VOLUMES
I0722 07:40:09.034941       1 driver.go:107] Enabling controller service capability: CREATE_DELETE_VOLUME
I0722 07:40:09.034943       1 driver.go:107] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0722 07:40:09.034945       1 driver.go:107] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0722 07:40:09.034947       1 driver.go:107] Enabling controller service capability: LIST_SNAPSHOTS
I0722 07:40:09.034949       1 driver.go:107] Enabling controller service capability: EXPAND_VOLUME
I0722 07:40:09.034951       1 driver.go:107] Enabling controller service capability: CLONE_VOLUME
I0722 07:40:09.034953       1 driver.go:107] Enabling controller service capability: LIST_VOLUMES_PUBLISHED_NODES
I0722 07:40:09.034955       1 driver.go:107] Enabling controller service capability: GET_VOLUME
I0722 07:40:09.034958       1 driver.go:119] Enabling volume access mode: SINGLE_NODE_WRITER
I0722 07:40:09.034960       1 driver.go:129] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0722 07:40:09.034963       1 driver.go:129] Enabling node service capability: EXPAND_VOLUME
I0722 07:40:09.034965       1 driver.go:129] Enabling node service capability: GET_VOLUME_STATS
I0722 07:40:09.034970       1 openstack.go:137] InitOpenStackProvider configFiles: [/etc/kubernetes/cloudprovider/cloudprovider.conf]
I0722 07:40:09.035170       1 openstack.go:90] Block storage opts: {0 true false}
I0722 07:40:09.035194       1 client.go:252] Using user-agent shoot--foo--bar foo-team cp cinder-csi-plugin/v1.23.3 gophercloud/2.0.0
W0722 07:40:09.041370       1 main.go:100] Failed to GetOpenStackProvider: Post "<auth-url>": x509: certificate signed by unknown authority

I checked the diff between v1.23.0..v1.23.3, but I cannot spot a change that would cause such failure. I don't have any change to the configuration - the same v1.23.0 configuration is used.

The more interesting thing is that I don't face such issue when I build v1.23.3 cinder-csi-plugin image from source.

$ git checkout v1.23.3
$ make image-csi-plugin GOOS=linux

An example image that works without issues - innoweek/cinder-csi-plugin-amd64:v1.23.3.

I assume there is something wrong with docker.io/k8scloudprovider/cinder-csi-plugin:v1.23.3. Can it be the case the binary in this image does not correspond to the v1.23.3 tag? Otherwise I cannot explain why it works for my custom built image and does not work for the upstream one.

What you expected to happen: docker.io/k8scloudprovider/cinder-csi-plugin:v1.23.3 to work without issues ("x509: certificate signed by unknown authority").

How to reproduce it: See above.

Anything else we need to know?:

Environment:

ialidzhikov commented 2 years ago

/sig storage

jichenjc commented 2 years ago

I also found related issue https://github.com/kubernetes/cloud-provider-openstack/issues/1938 and seem also suffer an issue that I don't know why .. maybe I didn't follow the exact build process (sorry about that)

do you might I create a new rc release and you give a try first before we upgrade to 1.23.4 ? @ialidzhikov ?

ialidzhikov commented 2 years ago

do you might I create a new rc release and you give a try first before we upgrade to 1.23.4 ? @ialidzhikov ?

Sounds good. Sure, I will test the new rc on our side and provide feedback whether it fixes the issue or not.

jichenjc commented 2 years ago

@ialidzhikov please try 1.23.4-rc1 and see whether it can fix your problem .. didn't do anything other than just do the release procedure again... at least previous issue on #1938 seems ok now on my local env

ialidzhikov commented 2 years ago

I can confirm that 1.23.4-rc1 works fine and this issue is not reproducible with it.

xmudrii commented 2 years ago

I can also confirm that 1.23.4-rc1 works as expected and that the issue is not reproducible with it.

jichenjc commented 2 years ago

https://github.com/kubernetes/cloud-provider-openstack/releases/tag/v1.23.4 is created and I ran docker run -it k8scloudprovider/cinder-csi-plugin:v1.23.4 which blkid prove to contains blkid output, so hopefully this is better than 1.23.3

root@jjtest1:~/go/src/github.com/cloud-provider-openstack# docker run -it k8scloudprovider/cinder-csi-plugin:v1.23.4 which blkid /sbin/blkid

ialidzhikov commented 2 years ago

v1.23.4 image works fine for me. Thank you @jichenjc!

/close

k8s-ci-robot commented 2 years ago

@ialidzhikov: Closing this issue.

In response to [this](https://github.com/kubernetes/cloud-provider-openstack/issues/1948#issuecomment-1196316217): >`v1.23.4` image works fine for me. Thank you @jichenjc! > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.