[Bug]: 2 VMs are getting same ip address

berelton commented 4 months ago

What happened?

Hello netmaker team, I got the issue when 2 VMs got the same ip address.

So we have 3 VMs: main -> has netmaker instance deployed vm1 -> connecting to meshnet vm2 -> connecting to meshnet

On main VM we are creating 2 subnets:

$ nmctl network create --name test1 --ipv4_addr 10.10.0.0/16
$ nmctl network create --name test2 --ipv4_addr 192.168.10.0/24
$ nmctl enrollment_key create --tags test1 --unlimited --networks test1
$ nmctl enrollment_key create --tags test2 --unlimited --networks test2
$ nmctl network list
+-----------+----------------------+----------------------+-----------------------+----------------------+
|   NETID   | ADDRESS RANGE (IPV4) | ADDRESS RANGE (IPV6) | NETWORK LAST MODIFIED | NODES LAST MODIFIED  |
+-----------+----------------------+----------------------+-----------------------+----------------------+
| test1     | 10.10.0.0/16         |                      | 2024-05-06T14:43:22Z  | 2024-05-06T14:43:22Z |
| test2     | 192.168.10.0/24      |                      | 2024-05-09T18:00:11Z  | 2024-05-09T18:00:11Z |
+-----------+----------------------+----------------------+-----------------------+----------------------+

On vm1:

$ wget -O netclient https://fileserver.netmaker.io/releases/download/v0.24.0/netclient-linux-amd64 && chmod +x ./netclient && ./netclient install
$ netclient join -t <token_for_test1>
$ netclient join -t <token_for_test2>
$ hostname -I
10.10.0.1 192.168.10.1

On vm2:

$ wget -O netclient https://fileserver.netmaker.io/releases/download/v0.24.0/netclient-linux-amd64 && chmod +x ./netclient && ./netclient install
$ netclient join -t <token_for_test1>
$ netclient join -t <token_for_test2>
$ hostname -I
10.10.0.2 192.168.10.1

So as you see both the VMs are getting 192.168.10.1 ip, but they should have different ip addresses.

Version

v0.24.0

What OS are you using?

No response

Relevant log output

No response

Contributing guidelines

[X] Yes, I did.

NEETweeb commented 4 months ago

Hello @berelton ! Thanks for reporting this issue. Tried to reproduce it on my end but wasn't able to verify the issue.

But this doesn't mean that this is a non-issue. It would help if you could provide the nodes list via nmctl node list.

berelton commented 4 months ago

hello @NEETweeb !

Appreciate the swift reply!

Sure, here is the response:

$ nmctl node list
+--------------------------------------+-----------------+-----------+--------+-----------------------+-------+
|                  ID                  |    ADDRESSES    |  NETWORK  | EGRESS | REMOTE ACCESS GATEWAY | RELAY |
+--------------------------------------+-----------------+-----------+--------+-----------------------+-------+
| 02a36777-62b2-4ff9-8952-75ff3796a03f | 10.10.0.2/16    | test1     | false  | false                 | false |
| 74e5e2f6-f5c5-4716-82a2-5371c2bef3ea | 192.168.10.1/24 | test2     | false  | false                 | false |
| cd9a6669-8dd9-47a0-aff9-53324df69273 | 10.10.0.1/16    | test1     | false  | false                 | false |
+--------------------------------------+-----------------+-----------+--------+-----------------------+-------+

If you could share your email and ssh key, I can share share you the access to the VMs to investigate, or we can have a quick call to debug together.

aizhvaly commented 4 months ago

hello @NEETweeb !

I'd like adjust more context about @berelton netmaker setup.

This situation happen when deploying netmaker via helm to a Kubernetes cluster with 2 replicas as indicated in the documentation examples. Most likely, 2 requests to connect to the network came on different replicas due to the Robin Round balancing on gateway. It seems that the problem may be related to synchronization between netmaker servers. I hope this information will help in studying the problem.

berelton commented 4 months ago

hello @NEETweeb , @afeiszli, any updates on that?

abhishek9686 commented 4 months ago

hello @NEETweeb , @afeiszli, any updates on that?

While using HA setup make sure caching is set to false CACHING_ENABLED=false

berelton commented 4 months ago

Hello @abhishek9686 , it is set to false.

$ nmctl server config
{
  "CoreDNSAddr": "SERVER_PUBLIC_IP",
  "APIConnString": "api.id0.mydomain.com:443",
  "APIHost": "api.id0.mydomain.com",
  "APIPort": "8081",
  "Broker": "",
  "ServerBrokerEndpoint": "",
  "BrokerType": "mosquitto",
  "EmqxRestEndpoint": "",
  "NetclientAutoUpdate": "enabled",
  "NetclientEndpointDetection": "",
  "MasterKey": "(hidden)",
  "DNSKey": "(hidden)",
  "AllowedOrigin": "*",
  "NodeID": "netmaker-a77-0",
  "RestBackend": "on",
  "MessageQueueBackend": "",
  "DNSMode": "on",
  "DisableRemoteIPCheck": "off",
  "Version": "v0.21.2",
  "SQLConn": "",
  "Platform": "linux",
  "Database": "postgres",
  "Verbosity": 1,
  "AuthProvider": "",
  "OIDCIssuer": "",
  "ClientID": "",
  "ClientSecret": "",
  "FrontendURL": "",
  "DisplayKeys": "on",
  "AzureTenant": "",
  "Telemetry": "on",
  "HostNetwork": "",
  "Server": "id0.mydomain.com",
  "PublicIPService": "",
  "MQPassword": "",
  "MQUserName": "",
  "MetricsExporter": "",
  "BasicAuth": "",
  "LicenseValue": "",
  "NetmakerTenantID": "",
  "IsEE": "no",
  "StunPort": 3478,
  "StunList": "",
  "TurnServer": "",
  "TurnApiServer": "",
  "TurnPort": 0,
  "TurnUserName": "",
  "TurnPassword": "",
  "UseTurn": false,
  "UsersLimit": 0,
  "NetworksLimit": 0,
  "MachinesLimit": 0,
  "IngressesLimit": 0,
  "EgressesLimit": 0,
  "DeployedByOperator": false,
  "Environment": "",
  "JwtValidityDuration": 43200000000000,
  "RacAutoDisable": true,
  "CacheEnabled": "",
  "endpoint_detection": false,
  "AllowedEmailDomains": ""
}

berelton commented 4 months ago

hello @NEETweeb , @afeiszli, any updates on that?

abhishek9686 commented 4 months ago

@berelton what is the rate at which you are joining clients to the network? are multiple clients joining the network at same time?

berelton commented 3 months ago

@abhishek9686 it is like 5 seconds difference, yes.

Is there a way to handle this?

abhishek9686 commented 3 months ago

Hello @abhishek9686 , it is set to false.

$ nmctl server config
{
  "CoreDNSAddr": "SERVER_PUBLIC_IP",
  "APIConnString": "api.id0.mydomain.com:443",
  "APIHost": "api.id0.mydomain.com",
  "APIPort": "8081",
  "Broker": "",
  "ServerBrokerEndpoint": "",
  "BrokerType": "mosquitto",
  "EmqxRestEndpoint": "",
  "NetclientAutoUpdate": "enabled",
  "NetclientEndpointDetection": "",
  "MasterKey": "(hidden)",
  "DNSKey": "(hidden)",
  "AllowedOrigin": "*",
  "NodeID": "netmaker-a77-0",
  "RestBackend": "on",
  "MessageQueueBackend": "",
  "DNSMode": "on",
  "DisableRemoteIPCheck": "off",
  "Version": "v0.21.2",
  "SQLConn": "",
  "Platform": "linux",
  "Database": "postgres",
  "Verbosity": 1,
  "AuthProvider": "",
  "OIDCIssuer": "",
  "ClientID": "",
  "ClientSecret": "",
  "FrontendURL": "",
  "DisplayKeys": "on",
  "AzureTenant": "",
  "Telemetry": "on",
  "HostNetwork": "",
  "Server": "id0.mydomain.com",
  "PublicIPService": "",
  "MQPassword": "",
  "MQUserName": "",
  "MetricsExporter": "",
  "BasicAuth": "",
  "LicenseValue": "",
  "NetmakerTenantID": "",
  "IsEE": "no",
  "StunPort": 3478,
  "StunList": "",
  "TurnServer": "",
  "TurnApiServer": "",
  "TurnPort": 0,
  "TurnUserName": "",
  "TurnPassword": "",
  "UseTurn": false,
  "UsersLimit": 0,
  "NetworksLimit": 0,
  "MachinesLimit": 0,
  "IngressesLimit": 0,
  "EgressesLimit": 0,
  "DeployedByOperator": false,
  "Environment": "",
  "JwtValidityDuration": 43200000000000,
  "RacAutoDisable": true,
  "CacheEnabled": "",
  "endpoint_detection": false,
  "AllowedEmailDomains": ""
}

can you check the CacheEnabled config in the configmap deployed?

sk3pp3r commented 2 months ago

can you check if both vm's have the same mac address?

To verify if both virtual machines have identical MAC addresses, you would need to check the network configuration settings of each VM. Typically, MAC addresses should be unique to each device to avoid network conflicts.

gravitl / netmaker