search

nigoroll / libvmod-dynamic

The Varnish dns/named director continued
BSD 2-Clause "Simplified" License
95 stars 34 forks source link

Assert error in getdns_result(): Condition((rr) != 0) not true. #116

Closed delthas closed 3 months ago

delthas commented 3 months ago

Stressing Varnish + libvmod-dynamic with many dynamic backends (with probes), I'm sometimes getting the following panic:

Assert error in getdns_result(), dyn_resolver_getdns.c line 259:
  Condition((rr) != 0) not true.
Backtrace:
  0x55bf1d92da92: /usr/sbin/varnishd(+0x5ba92) [0x55bf1d92da92]
  0x55bf1d9aac45: /usr/sbin/varnishd(VAS_Fail+0x45) [0x55bf1d9aac45]
  0x7f845e54e5ec: ./vmod_cache/_vmod_dynamic.29b766e616262ab0ef9252419367ce48fbf642168ba34fc382b1742a77dac5c4(+0xf5ec) [0x7f845e54e5ec]
  0x7f845e544b65: ./vmod_cache/_vmod_dynamic.29b766e616262ab0ef9252419367ce48fbf642168ba34fc382b1742a77dac5c4(+0x5b65) [0x7f845e544b65]
  0x7f84731cd064: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8064) [0x7f84731cd064]
  0x7f8472f0262d: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8472f0262d]

This refers to the assert here: https://github.com/nigoroll/libvmod-dynamic/blob/master/src/dyn_resolver_getdns.c#L259

I can't reproduce this on my machine to debug locally, I only got this on a remote server. I think the DNS server there might be having issues; but if this is caused by some DNS server issue we should probably not panic (and instead just add no backend).

From my understanding of the function, this can happen if getdns_common_more_answers(state) != 0 from the beginning (or if getdns_common_more_answers(state) == 0 and getdns_list_get_dict(state->answers, state->answer++, &rr) gives a NULL rr, not sure if that's relevant).

nigoroll commented 3 months ago

Nice find, thank you @delthas !