Storage IP addresses are not released correctly when there are overlapping IP addresses in different ranges

waza-ari commented 6 years ago

ISSUE TYPE

Bug Report

COMPONENT NAME

Cloudstack Core, Storage Networking

CLOUDSTACK VERSION

I think all current versions are affected

CONFIGURATION

Cloudstack with Basic networking (KVM Hosts) and NFS storage

OS / ENVIRONMENT

CentOS/RHEL 6, but not relevant

SUMMARY

It looks like the release of storage IP addresses is not working proberly, as the releaseIpAddress method of StorageNetworkIpAddressDaoImpl does not take the range id into account

STEPS TO REPRODUCE

Create a basic zone and assign an IP address pool for storage traffic. This should create an entry in cloud.dc_storage_network_ip_range and one entry per IP address in range in cloud.op_dc_storage_network_ip_address
Delete that zone again
Create another zone and assign the same IP address pool for storage traffic. This will create a new range and create the IP addresses again with the new range
No add/delete some SSVMs or KVM hosts taking a storage IP address

EXPECTED RESULTS

In the cloud.op_dc_storage_network_ip_address table, only the IP addresses of the second range should be used both when adding or deleting SSVMs (obtaining and releasing IPs)

ACTUAL RESULTS

When obtaining IP addresses, the correct pool is queried for free IP addresses and the taken field is set correctly. When releasing an IP address however, the range_id is not taken into account, only the IP address itself is used. In our setup this leads to the problem that only the IP address in the first (unused) pool is released (taken is set to null), such that after a few iterations now new VMs can be spawned, even though enough IP addresses are available.

We suspect the problem in https://github.com/apache/cloudstack/blob/893a88d225276e45f12f9490e6af2c94a81c2965/engine/schema/src/main/java/com/cloud/dc/dao/StorageNetworkIpAddressDaoImpl.java#L99-L105 as it is not taking the range_id into consideration.

DaanHoogland commented 3 years ago

@wido @GabrielBrascher is this known to you guys? As described it seems like a simple fix. on the other hand we are phasing out basic networks in favour of shared nets. so we might hold out on this. @waza-ari are you still involved? do you have code (as you so precisely know where the crux is)?

waza-ari commented 3 years ago

Hi there, I’m not with my former company anymore, I don’t know what the status of this issue is. I also do not have the code fragment anymore, I barely remember what it was about :)

I pointed a colleague from my former company to this thread though, maybe he knows if the problem still occurs.

GabrielBrascher commented 3 years ago

@DaanHoogland (un)fortunately we haven't seen it; I don't have much info to help here. Maybe @kiwiflyer knows something about it?

I would need to set a test environment only focused to reproduce such an issue and then debug it.

DaanHoogland commented 3 years ago

ok, @GabrielBrascher , leaving this for now unless @kiwiflyer or @nathanejohnson or @waza-ari 's former colleague comes back. Also looking at the suggested code it does not seem to be specific to basic or any type of networking as i'd expect the releaseIpAddress(String ip) call t get a rangeId as well for any kind of ip being released.

nvazquez commented 3 years ago

Hi @GabrielBrascher @kiwiflyer @waza-ari @nathanejohnson is this still a valid issue?

GabrielBrascher commented 3 years ago

@nvazquez I still have not seen nor heard of such an issue. So I am not able to provide any insights regarding this issue.

It seems that @waza-ari did not get back. Maybe we should close this and if someone reproduces it, then we reopen it.

What do you think @DaanHoogland @kiwiflyer @nathanejohnson @waza-ari?

waza-ari commented 3 years ago

Hi, I honestly can’t tell what is best to do. I pointed my former colleagues to this issue, but some of them have changed the company as well, others apparently didn’t come back to this issue. Without knowing for sure I guess they found some method around it.

I still remember that it was quite easy to reproduce back then and if @DaanHoogland is right and it does not only affect basic networking, it might deserve a check. All you need is two overlapping IP ranges and the issue should become apparent quite quickly.

Personally though I’m not involved anymore and haven’t used CloudStack for years, so I won’t be able to drive this anymore, even though I opened it back then. Sorry about that.

DaanHoogland commented 3 years ago

thanks @waza-ari , I'll close this and we'll re-open or open a new one when needed.

apache / cloudstack