search

ZigzagAK / ngx_dynamic_upstream

Dynamic upstream for nginx
Other
34 stars 11 forks source link

dns_update not works #7

Closed zdm closed 3 years ago

zdm commented 3 years ago

Hi, I am using:

dns_update 10s pool;

But it resolves server name and adds peers ONLY when nginx starts. When peers ip addresses are changed in dns response - it doesn't refresh peers list in nginx upstream zone (backends become unavailable and curl "http://127.0.0.1/dynamic-upstream?upstream=test&verbose=" shows old list). Is this by design or i am doing something wrong?

ZigzagAK commented 3 years ago

May be you have nscd daemon installed? This module uses getaddrinfo, and this function use nscd cache.

zdm commented 3 years ago

I am running under docker container. Don't think, that it installs any additional modules. So, It should work and should refresh peer list on timeout? Correct? Problems on my side?

On 07.03.2021 22:29, Aleksey Konovkin wrote:

May be you have nscd daemon installed? This module uses getaddrinfo, and this function use nscd cache.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ZigzagAK/ngx_dynamic_upstream/issues/7#issuecomment-792346846, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH2MSD7VUSDQWO7TMJXS5DTCPO3LANCNFSM4YYG56UA.

zdm commented 3 years ago

Does nginx resolver setting make any effect for your module or it is not used in dns_update?

On 07.03.2021 22:29, Aleksey Konovkin wrote:

May be you have nscd daemon installed? This module uses getaddrinfo, and this function use nscd cache.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ZigzagAK/ngx_dynamic_upstream/issues/7#issuecomment-792346846, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH2MSD7VUSDQWO7TMJXS5DTCPO3LANCNFSM4YYG56UA.

zdm commented 3 years ago

Is it possible, that it uses TTL, that returned by dns server?

ZigzagAK commented 3 years ago

No, resolver is not work with this module. Only dns_update. Yes, dns_update 10s should refresh peers.

Please, use dns_update without pool keyword.

dns_update 10s;

zdm commented 3 years ago

Without threads pool option - also not works.

On 07.03.2021 22:55, Aleksey Konovkin wrote:

No, resolver is not work with this module. Only dns_update. Yes, dns_update 10s should refresh peers.

Please, use dns_update without pool keyword.

dns_update 10s;

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ZigzagAK/ngx_dynamic_upstream/issues/7#issuecomment-792351336, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH2MSFA5OMJTXGPJRTIC53TCPR37ANCNFSM4YYG56UA.

ZigzagAK commented 3 years ago

I didn't check this feature under docker. Please check updates without docker.

ZigzagAK commented 3 years ago

I will check this feature unde docker in a week.

zdm commented 3 years ago

Thank you. I am trying to make it work. Will write if will have any results.

Tell me if you will need some help.

On 07.03.2021 23:19, Aleksey Konovkin wrote:

I will check this feature unde docker in a week.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ZigzagAK/ngx_dynamic_upstream/issues/7#issuecomment-792355073, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH2MSFOCVSQLNOEZ4PCHOTTCPUVBANCNFSM4YYG56UA.

ZigzagAK commented 3 years ago

Under regular OS updates works correctly. I checked it with google.com server in upstream. From time to time list of peers were changed. Nginx.conf in repository represents config, that works correctly.

May be under docker we have some problem. I will check it. And you may check it without docker (directly under main os).

zdm commented 3 years ago

What version of nginx you are using? 1.19.7?

On 07.03.2021 23:35, Aleksey Konovkin wrote:

Under regular OS updates works correctly. I checked it with google.com server in upstream. From time to time list of peers were changed. Nginx.conf in repository represents config, that works correctly.

May be under docker we have some problem. I will check it. And you may check it without docker (directly under main os).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ZigzagAK/ngx_dynamic_upstream/issues/7#issuecomment-792357513, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH2MSEEAXVUTCWMAFFATW3TCPWT5ANCNFSM4YYG56UA.

zdm commented 3 years ago

so, for me upstream server is resolved on startup and cached forever.

most probable, that the problem is not in the docker env, but in the nginx version.

could you check it with 1.19.7 pls?

On 07.03.2021 23:35, Aleksey Konovkin wrote:

Under regular OS updates works correctly. I checked it with google.com server in upstream. From time to time list of peers were changed. Nginx.conf in repository represents config, that works correctly.

May be under docker we have some problem. I will check it. And you may check it without docker (directly under main os).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ZigzagAK/ngx_dynamic_upstream/issues/7#issuecomment-792357513, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH2MSEEAXVUTCWMAFFATW3TCPWT5ANCNFSM4YYG56UA.

zdm commented 3 years ago

With nginx v1.18.0 also not works. So the problem is definitely in docker environment.

zdm commented 3 years ago

I can debug it under my infrastructure if you will give me instructions. For example, at first we can check, if dns requests are performed at all? Add some console logs and run.

ZigzagAK commented 3 years ago

Resolve is performed with standard C function getaddrinfo (nginx on start also use this function). This function is implemented in standard linux library. If you want debug network requests, you should use tcpdump or wireshark with standard 53 DNS service port. Docker as I know may cache ip addresses and this may be a cause. Sorry, but I haven't free time at this week to check it. I will try it when I get free time.

zdm commented 3 years ago

Docker has own resolver, but problem is in other thing. Nginx doesn't send dns requests, it resolves domain name only at startup, and don't refresh, despite that dns returns new set of ip addresses.

I am unable to find getaddr call in your source code.

On 12.03.2021 22:55, Aleksey Konovkin wrote:

Resolve is performed with standard C function getaddrinfo (nginx on start also use this function). This function is implemented in standard linux library. If you want debug network requests, you should use tcpdump or wireshark with standard 53 DNS service port. Docker as I know may cache ip addresses and this may be a cause. Sorry, but I haven't free time at this week to check it. I will try it when I get free time.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ZigzagAK/ngx_dynamic_upstream/issues/7#issuecomment-797749781, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH2MSE3DY4F4B2N7ZDL53TTDJ5VDANCNFSM4YYG56UA.

ZigzagAK commented 3 years ago

https://github.com/ZigzagAK/ngx_dynamic_upstream/blob/master/src/ngx_dynamic_upstream_op.cpp#L132 -> https://github.com/nginx/nginx/blob/master/src/core/ngx_inet.c#L1117

ZigzagAK commented 3 years ago

Really, problem exists and it is in my code. Please, add to https://github.com/ZigzagAK/ngx_dynamic_upstream/blob/master/src/ngx_http_dynamic_upstream_module.cpp#L961 op.hash = 0;

zdm commented 3 years ago

trying...

On 12.03.2021 23:55, Aleksey Konovkin wrote:

Really, problem exists and it is in my code. Please, add to https://github.com/ZigzagAK/ngx_dynamic_upstream/blob/master/src/ngx_http_dynamic_upstream_module.cpp#L961 https://github.com/ZigzagAK/ngx_dynamic_upstream/blob/master/src/ngx_http_dynamic_upstream_module.cpp#L961 op.hash = 0;

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ZigzagAK/ngx_dynamic_upstream/issues/7#issuecomment-797777862, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAH2MSBA3XTSZTQKNQX4YL3TDKETPANCNFSM4YYG56UA.

ZigzagAK commented 3 years ago

021/03/13 01:03:09 [notice] 15936#16046: google: added server google.com peer 64.233.164.138:80 2021/03/13 01:03:09 [notice] 15936#16046: google: added server google.com peer 64.233.164.102:80 2021/03/13 01:03:09 [notice] 15936#16046: google: added server google.com peer 64.233.164.113:80 2021/03/13 01:03:09 [notice] 15936#16046: google: added server google.com peer 64.233.164.100:80 2021/03/13 01:03:09 [notice] 15936#16046: google: added server google.com peer 64.233.164.139:80 2021/03/13 01:03:09 [notice] 15936#16046: google: added server google.com peer 64.233.164.101:80 2021/03/13 01:03:09 [notice] 15936#16046: google: removed server google.com peer 173.194.220.101:80 2021/03/13 01:03:09 [notice] 15936#16046: google: removed server google.com peer 173.194.220.100:80 2021/03/13 01:03:09 [notice] 15936#16046: google: removed server google.com peer 173.194.220.138:80 2021/03/13 01:03:09 [notice] 15936#16046: google: removed server google.com peer 173.194.220.102:80 2021/03/13 01:03:09 [notice] 15936#16046: google: removed server google.com peer 173.194.220.113:80 2021/03/13 01:03:09 [notice] 15936#16046: google: removed server google.com peer 173.194.220.139:80 2021/03/13 01:03:09 [info] 15936#16046: google: dns synced 2021/03/13 01:04:29 [notice] 15936#16046: google: added server google.com peer 64.233.165.139:80 2021/03/13 01:04:29 [notice] 15936#16046: google: added server google.com peer 64.233.165.138:80 2021/03/13 01:04:29 [notice] 15936#16046: google: added server google.com peer 64.233.165.102:80 2021/03/13 01:04:29 [notice] 15936#16046: google: added server google.com peer 64.233.165.101:80 2021/03/13 01:04:29 [notice] 15936#16046: google: added server google.com peer 64.233.165.100:80 2021/03/13 01:04:29 [notice] 15936#16046: google: added server google.com peer 64.233.165.113:80 2021/03/13 01:04:29 [notice] 15936#16046: google: removed server google.com peer 64.233.164.101:80 2021/03/13 01:04:29 [notice] 15936#16046: google: removed server google.com peer 64.233.164.139:80 2021/03/13 01:04:29 [notice] 15936#16046: google: removed server google.com peer 64.233.164.100:80 2021/03/13 01:04:29 [notice] 15936#16046: google: removed server google.com peer 64.233.164.113:80 2021/03/13 01:04:29 [notice] 15936#16046: google: removed server google.com peer 64.233.164.102:80 2021/03/13 01:04:29 [notice] 15936#16046: google: removed server google.com peer 64.233.164.138:80 2021/03/13 01:04:29 [info] 15936#16046: google: dns synced

ZigzagAK commented 3 years ago

fix has been pushed.

zdm commented 3 years ago

i found following issues:

1.for example currently domain has 3 ip and i shutdown all peers, so domain name can't be resolved, but nginx will not update peers list, it still contains 3 old ip addresses. correct behavior is to remove all peers and replace with 0.0.0.1 down. this is importand, because these old ip addresses can be re-assigned to completely other service by dhcp.

  1. nginx can;t start, if domain name can;t be resolved.
ZigzagAK commented 3 years ago
  1. May be you are right. I will think about it.
  2. This is standard nginx behavior. This is not related to this module.
ZigzagAK commented 3 years ago

Resolved.