pool overlap issue on `gefyra up`

[tschale]

When running gefyra up it sometimes fails with a pool overlap error:

[...]
[INFO] Creating Docker network
[ERROR] 403 Client Error for http+docker://localhost/v1.41/networks/create: Forbidden ("Pool overlaps with other one on this address space")
[INFO] Removing running bridges
[INFO] Uninstalling Operator
[INFO] Removing Cargo
[INFO] Stopping remainder container from Gefyra network
[INFO] Removing Docker network
Traceback (most recent call last):
  File "docker.api.client", line 268, in _raise_for_status
  File "requests.models", line 960, in raise_for_status
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http+docker://localhost/v1.41/networks/create

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gefyra.api.up", line 48, in up
  File "gefyra.local.networking", line 24, in handle_create_network
  File "docker.models.networks", line 156, in create
  File "docker.api.network", line 153, in create_network
  File "docker.api.client", line 274, in _result
  File "docker.api.client", line 270, in _raise_for_status
  File "docker.errors", line 31, in create_api_error_from_http_exception
docker.errors.APIError: 403 Client Error for http+docker://localhost/v1.41/networks/create: Forbidden ("Pool overlaps with other one on this address space")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docker.api.client", line 268, in _raise_for_status
  File "requests.models", line 960, in raise_for_status
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http+docker://localhost/v1.41/networks/ee2727b13b1219a4678a81d0f5e303b1c55b0ead527e92cfd03e0c15b8c2f8cc

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "gefyra.__main__", line 143, in main
  File "gefyra.api.up", line 57, in up
  File "gefyra.api.utils", line 10, in wrapper
  File "gefyra.api.down", line 29, in down
  File "gefyra.local.networking", line 37, in handle_remove_network
  File "docker.models.networks", line 91, in remove
  File "docker.utils.decorators", line 19, in wrapped
  File "docker.api.network", line 187, in remove_network
  File "docker.api.client", line 270, in _raise_for_status
  File "docker.errors", line 31, in create_api_error_from_http_exception
docker.errors.APIError: 403 Client Error for http+docker://localhost/v1.41/networks/ee2727b13b1219a4678a81d0f5e303b1c55b0ead527e92cfd03e0c15b8c2f8cc: Forbidden ("error while removing network: network k3d-gefyra-demos id ee2727b13b1219a4678a81d0f5e303b1c55b0ead527e92cfd03e0c15b8c2f8cc has active endpoints")

I've seen this quite regularly. It seems to occur randomly, if I run gefyra up multiple times, it usually works eventually.

I am somewhat confused about the error while removing network: network k3d-gefyra-demos in the stacktrace. I also can see that error when running gefyra down. However that is the docker network of the k3d cluster and not of gefyra itself. Does this mean gefyra is trying to remove that network? Or is this a confusing error output?

[Schille]

Fix is in version 0.6.16. Please close this issue if you can confirm it working.

[tschale]

It doesn't seem to be completely resolved. If I run gefyra down and then gefyra up again, I can see the error for about one minute. After that it works. When running gefyra up when it didn't run before (i.e. no gefyra down immediately before) it doesn't seem to occur anymore, at least I haven't observed it so far. Which is an improvement, this was quite regular before.

[Schille]

It is with great pleasure to inform you, that in version 0.7.2 there is another fix for this. A retry logic now tries to create the Docker network. Hopefully this helps. Please check and confirm it working.

[tschale]

I don't have a huge sample size, but it seems that it doesn't occur anymore.