openark / orchestrator

MySQL replication topology management and HA
Apache License 2.0
5.61k stars 927 forks source link

does DiscoveryIgnoreMasterHostnameFilters work for master with same ip subnet as slaves? #1310

Open larry7xia80 opened 3 years ago

larry7xia80 commented 3 years ago

hi shlomi-noach, we have the replication chain as follows. [top master] 10.40.0.19 (3306) <------- [slave] 10.40.0.9 (53306) <------- [slave] 10.40.0.199 (53306). we put "DiscoveryIgnoreMasterHostnameFilters": ["10.40.0.19"] in orchestrator.conf.json, but 10.40.0.19 still show up in orchestrator webUI, and update consul kv with 10.40.0.19 as master.

does DiscoveryIgnoreMasterHostnameFilters work for master with same ip subnet as slaves?

we used this paramter DiscoveryIgnoreMasterHostnameFilters when we replicated from external source e.g google cloud sql. in other enviroment. it worked and external source will not show up in orchestrator.

thanks , Best regards,

shlomi-noach commented 3 years ago

does DiscoveryIgnoreMasterHostnameFilters work for master with same ip subnet as slaves?

There's nothing in the code to consider subnets. It should work the same way inside/outside the subnet. Can you paste output of orchestrator-client -c topology -aliss your_cluster?

larry7xia80 commented 3 years ago

thanks shlomi-noach,

[PRD]root@prd-npci-uscentral1a-v15-#####-db01•17:18:12>` orchestrator-client --auth dbadmin:$pw --api http://prd-adm-v10--###11b-orchestrator02:3000/api -c topology -i prd-npci--###1a-v15--#####--db01:53306 cli `

prd-npci-###1a--#####--db01:3306          [0s,ok,5.7.20-19-log,rw,ROW,>>,GTID]
+ prd-npci--###1b-v15--#####--db02:53306   [0s,ok,5.7.20-19-log,rw,ROW,>>,GTID]
  + prd-npci--###1a-v15--#####--db01:53306 [0s,ok,5.7.20-19-log,ro,ROW,>>,GTID]
orchestrator-client --auth dbadmin:$pw --api http://prd-adm-v10-uscentral11b-orchestrator02:3000/api -c topology -alias #####
prd-npci-###1a--#####--db01:3306          [0s,ok,5.7.20-19-log,rw,ROW,>>,GTID]
+ prd-npci--###1b-v15--#####--db02:53306   [0s,ok,5.7.20-19-log,rw,ROW,>>,GTID]
   + prd-npci--###1a-v15--#####--db01:53306 [0s,ok,5.7.20-19-log,ro,ROW,>>,GTID]
[PRD]root@prd-npci-uscentral1a-v15-#####-db01•17:18:12> ping prd-npci-###1a--#####--db01
PING prd-npci-###1a--#####--db01 (10.40.0.19) 56(84) bytes of data.
64 bytes from prd-npci-###1a--#####--db01 (10.40.0.19): icmp_seq=1 ttl=64 time=1.67 ms

please see above. i replaced part of server name with #.

larry7xia80 commented 3 years ago

thanks

shlomi-noach commented 3 years ago

Try adding the resolved hostname, prd-npci-###1a--#####--db01, to DiscoveryIgnoreMasterHostnameFilters?

larry7xia80 commented 3 years ago

thanks shlomi-noach, will try to add both ip and server_name to DiscoveryIgnoreMasterHostnameFilters. feedback here later.....

larry7xia80 commented 3 years ago

hi shlomi-noach, after adding both the ip and hostname to orchestrator.conf.json and restart the services.
still no luck. top master "prd-npci-###1a--#####--db01" still shows up in orchestrator GUI. DiscoveryIgnoreMasterHostnameFilters": ["10.40.0.19", "prd-npci-###1a--#####--db01"] we tried nslookup all the hosts of that cluster on raft leader, it could resolve the hostname to IP .

[xia@prd-adm-v10-###11b-orchestrator02 ~]$ nslookup prd-npci-###1a--#####--db01
Server:         169.254.169.254
Address:        169.254.169.254#53

Non-authoritative answer:
Name:   prd-npci-###1a--#####--db01.c.platinum-fortress-817.internal
Address: 10.40.0.19

is there a way for me to check if DiscoveryIgnoreMasterHostnameFilterswas effective, or other debug information to track ? thanks

shlomi-noach commented 3 years ago
  1. Try hitting /api/hostname-resolve-cache -- do you see that server in there? What's the entry?
  2. Try hitting /api/reset-hostname-resolve-cache, wait a minute -- does that change anything?
  3. What's your configuration for HostnameResolveMethod and MySQLHostnameResolveMethod?
yangeagle commented 3 years ago

Sorry,I didn't write clearly. I mean, try these steps: 1.config DiscoveryIgnoreMasterHostnameFilters as above 2.restart orchestrator 3.forget the cluster

  1. just discover the instance 10.40.0.9
shlomi-noach commented 3 years ago

I'm not sure what the comment above means to say?