vultr / vultr-csi

Container Storage Interface (CSI) Driver for Vultr Block Storage
Apache License 2.0
40 stars 17 forks source link

[BUG] - Provisioning fails when IPv6 is enabled on the host #97

Closed dvcrn closed 1 year ago

dvcrn commented 2 years ago

Describe the bug Block storage provisioning is not working when the host is commuting with IPv6 to the block API

Warning  ProvisioningFailed    10s (x3 over 22s)      block.csi.vultr.com_csi-vultr-controller-0_5df7f339-784b-49f2-9bda-5f1894bbe607  failed to provision volume with StorageClass "vultr-block-storage-hdd-retain": rpc error: code = Internal desc = {"error":"Unauthorized IP address: 2001:19f0:7002:xxx","status":401}

Where 2001:xxx is the main GUA of the node

To provision a volume, IPv6 has to get disabled first, then provision, then re-enable IPv6

To Reproduce Steps to reproduce the behavior:

  1. Enable IPv6 on the host
  2. Try to provision block storage

Expected behavior

gooooer commented 1 year ago

I have the same issue with ipv4 too:

Warning ProvisioningFailed 2s (x5 over 17s) block.csi.vultr.com_csi-vultr-controller-0_68116136-a5f1-4cba-8423-e68ab92910e4 failed to provision volume with StorageClass "vultr-block-storage-hdd": rpc error: code = Internal desc = {"error":"Unauthorized IP address: 207.246.105.116","status":401}

gooooer commented 1 year ago

@dvcrn I found a solution. Go to your account API settings and add relevant subnets in Access Control section on that page.

dvcrn commented 1 year ago

Glad you figured your case out!

My issue is specifically related to IPv6, so IPv4 subnets aren't working. Forcing calls to use IPv4 instead works as a workaround, so in /etc/hosts:

66.55.134.142 api.vultr.com

Alternatively, use sysctl to disable ipv6 on the interface from the host, wait until the volume is provisioned, then re-enable ipv6

NeilHanlon commented 1 year ago

Heyo all,

This is a pretty significant problem given that VKE 1.25.7+1 and VKE 1.26.2+1 were both released on 2023-03-14 without this bug being fixed.

I have added my nodes' IPv6 subnets (as /64s, /128s, etc) to my account settings, but still this does not permit IPv6 traffic to the Block Storage API. I even tried allowing ALL IP6 traffic, and this still does not permit the traffic in.

Disabling IPv6 is not an acceptable answer.

What can we do here?

happytreees commented 1 year ago

@NeilHanlon

Hello there, this bug is not related to the CSI itself and is related to how API keys work within Vultr. This is simply an issue with the permissions of the API key being used. If you use the CSI on accounts without any API rules it works without issue.

With that being said, I will have the appropriate teams review the behavior of the API key rule functionality to identify what is happening. I will continue this conversation in the ticket you have open as well.

Closing this as it is not related to the CSI itself.

NeilHanlon commented 1 year ago

Thank you @happytreees - I also opened #GHK-86EGG under my business account with Vultr.