Minikube container not using docker network specified in the --network option

paularah commented 2 years ago

What Happened?

I have a couple of docker containers running locally in the same network on my machine. I want to be able to reach the pods in the cluster started with the same docker network from any of the containers running outside the cluster on my machine. I assume since the cluster was started with the same docker network, this should be somehow possible.

Attach the log file

N/A

Operating System

Ubuntu

Driver

Docker

afbjorklund commented 2 years ago

You should be able to reach the node IP with Docker on Ubuntu (on the private network), unlike the situation on Mac/Win.

It shouldn't be very different from multi-node. You can reach the services but not the pods, but you seem to be aware of this...

This assumes that they are running in the "minikube" network ?

Otherwise you need to set up a bridge between the Docker networks.

paularah commented 2 years ago

I think the issue is that minikube doesn't use the user-defined network given. In my case, I have a network created with a subnet. When I start minikube with this network, minikube uses the default docker bridge network instead. I pasted a timeline

` ❯ minikube start --network=compose_pt-net --driver=docker

😄 minikube v1.25.2 on Ubuntu 20.04 ✨ Using the docker driver based on user configuration 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image ... 🔥 Creating docker container (CPUs=2, Memory=3900MB) ...- 🐳 Preparing Kubernetes v1.23.3 on Docker 20.10.12 ... ▪ kubelet.housekeeping-interval=5m ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

purpleteam-orchestrator on  main [✘] is 📦 v4.0.0-alpha.3 via  v16.14.2 on ☁️ (us-east-1) took 2m13s ❯ docker inspect --format='{{json .NetworkSettings.Networks}}' minikube

{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"NetworkID":"3a476095b22fcc4819a8b22f0d8b62bf61899098985b8a3bf5b3070d97eba588","EndpointID":"7258e3b3003fa561973b1e452977c8fb60a09f6729c80760f05e2aa6ea0810a8","Gateway":"172.17.0.1","IPAddress":"172.17.0.2","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:ac:11:00:02","DriverOpts":null}}

`

paularah commented 2 years ago

@afbjorklund there are a couple of other unusual cases I have identified, I will try to reproduce. But essentially you're not guaranteed that the network provided is what minikube would start up the cluster with. In one the cases, I created a bridge network and started minikube with it. This failed because a subnet wasn't defined in the network. Without any changes to the network, I tried the second time and surprisingly, minikube starts the cluster with the same network.

paularah commented 2 years ago

Steps to reproduce

Create a docker network with a subnet docker network create -d bridge test-net --subnet=172.25.0.0/24
Start minikube with created network minikube start --network=test-net --driver=docker
Inspect the minikube container networking settings docker inspect --format='{{json .NetworkSettings.Networks}}' minikube

This minikube container is started with the default bridge network instead of the user-defined network test-net

klaases commented 1 year ago

Hi @paularah, if you are still experiencing this issue an need community support, please check out our resources on:

https://minikube.sigs.k8s.io/community/

Are you still experiencing this issue?

/triage needs-information

paularah commented 1 year ago

Hi @klaases, This was due to not specifying a gateway in the docker network. I think the documentation needs to reflect that you need both a subnet and gateway to be able to use a user-defined bridge network.

klaases commented 1 year ago

Hi @paularah, thank you for the followup and confirmation.

Would you be interested in creating a PR to help address this documentation? I would be happy to review and approve such a change to help improve the minikube experience.

paularah commented 1 year ago

Hi @klaases, Sure, I'll tag you to the PR.

klaases commented 1 year ago

Hi @paularah, thank you for your PR, and look forward to review.

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/minikube/issues/14663#issuecomment-1465319135): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

kubernetes / minikube