Exception: Send error: 101: Network is unreachable

[marcosfrm]

The Client is experiencing recurring DNS resolution failures when downloading work units and submitting results. It typically takes 3-6 tries for the DNS to resolve with my ISP servers.

After changing the DNS server to Cloudflare, I encountered the following exception:

18:30:56:D4:NS:1.1.1.1:53:Exception: Send error: 101: Network is unreachable
18:30:56:D4:NS:1.1.1.1:53:       At: src/cbang/net/Socket.cpp:339:cb::Socket::write()
18:30:56:D4:NS:1.1.1.1:53:  #1 0x0094d4fa in cb::Socket::write(unsigned char const*, long, unsigned int, cb::SockAddr const*) at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #2 0x00a228f3 in cb::DNS::Nameserver::transmit(cb::Enumeration<cb::DNS::TypeEnumeration>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /usr/bin/fah-client1.1:53:  #3 0x00a17219 in cb::DNS::Base::pump() at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #4 0x00a1d56c in void std::__invoke_impl<void, void (cb::DNS::Base::*&)(), cb::DNS::Base*&>(std::__invoke_memfun_deref, void (cb::DNS::Base::*&)(), cb::DNS::Base*&) at /usr/bin/fah-client56:D4:NS:1.1.1.1:53:  #5 0x00a1d36d in std::__invoke_result<void (cb::DNS::Base::*&)(), cb::DNS::Base*&>::type std::__invoke<void (cb::DNS::Base::*&)(), cb::DNS::Base*&>(void (cb::DNS::Base::*&)(), cb::D
NS::Base*&) at /usr/bin/fah-client0a1cf32 in void std::_Bind<void (cb::DNS::Base::*(cb::DNS::Base*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #7 0x00a1ca10 in void std::_Bind<void (cb::DNS::Base::*(cb::DNS::Base*))()>::operator()<, void>() at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #8 0x00a1bfa8 in std::_Function_handler<void (), std::_Bind<void (cb::DNS::Base::*(cb::DNS::Base*))()> >::_M_invoke(std::_Any_data const&) at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #9 0x0078c950 in std::function<void ()>::operator()() const at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #10 0x0078b608 in cb::Event::EventFactory::bind(std::function<void ()>)::{lambda(cb::Event::Event&, int, unsigned int)#1}::operator()(cb::Event::Event&, int, unsigned int) const at /usr/bin/fah-client1.1:53:  #11 0x0078ea97 in std::_Function_handler<void (cb::Event::Event&, int, unsigned int), cb::Event::EventFactory::bind(std::function<void ()>)::{lambda(cb::Event::Event&, int, unsigned
 int)#1}>::_M_invoke(std::_Any_data const&, cb::Event::Event&, int&&, unsigned int&&) at /usr/bin/fah-clienterator()(cb::Event::Event&, int, unsigned int) const at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #13 0x00892f01 in cb::Event::Event::call(int, short) at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #14 0x00891fc4 at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #15 0x00acc93f at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #16 0x00accec5 at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #17 0x00acd625 in event_base_loop at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #18 0x00accf4d in event_base_dispatch at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #19 0x0088949c in cb::Event::Base::dispatch() at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #20 0x007fb4d6 in FAH::Client::OS::dispatch() at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #21 0x007a519d in FAH::Client::App::run() at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #22 0x00780c9d at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #23 0x00780c2e in main at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:  #24 0x7f49225977e5 in __libc_start_main at /lib64/libc.so.6
18:30:56:D4:NS:1.1.1.1:53:  #25 0x00780a8e in _start at /usr/bin/fah-client
18:30:56:D4:NS:1.1.1.1:53:Caught at: src/cbang/dns/Nameserver.cpp:208:cb::DNS::Nameserver::transmit()
18:30:58:D3:Default:Unit -DDu5C8uJxxGrLs5XGfAe1EVPMZj9VXLvIhr-eyHwpE enabled
18:30:58:D1:Default:Remaining CPUs: 0, Remaining GPUs: 0, Active WUs: 1
18:31:12:W :CON1:DNS lookup failed for assign1.foldingathome.org
18:31:12:D3:CON1:Connection failed
18:31:12:E :OUT1:Failed response: CONNECT

On the next attempt, it worked. The network was up when this exception occurred.

AlmaLinux 8 cbang git bb6841b234fcbbad74272d081b31eb3bb78bdde1 fah-client git 88692f3db014831bb4d226b4fc2efa8066b5118d

[jcoffland]

I'm not seeing this error. It looks like you may have a networking error. Send error: 101: Network is unreachable means address 1.1.1.1 is unreachable. Is it possible your /etc/resolv.conf contains other server addresses? Can you ping 1.1.1.1?

[marcosfrm]

Indeed, this exception is due to a lack of connectivity, likely caused by having debugging enabled. Without debugging and with Cloudflare's DNS, name resolution is working fine. With my ISP's servers, it fails consistently, recovering after a few attempts. I suspect they have subpar servers.

[marcosfrm]

With my ISP's servers:

18:09:46:D5:DNS: response TIMEOUT to a:assign1.foldingathome.org with 1 waiting requests, 1 attempts
18:09:51:D5:DNS: response TIMEOUT to a:assign1.foldingathome.org with 1 waiting requests, 2 attempts
18:09:56:D5:DNS: response TIMEOUT to a:assign1.foldingathome.org with 1 waiting requests, 3 attempts
18:09:56:W :CON2:DNS lookup failed for assign1.foldingathome.org
18:09:56:D3:CON2:Connection failed
18:09:56:E :OUT2:Failed response: CONNECT

...until it succeeds...

[jcoffland]

This is strange behavior. Debugging shouldn't have a significant impact on DNS lookups. I also don't see this issue with debugging enabled. Are you having other networking problems?

[marcosfrm]

I didn't see the problem again today. Looks like it was something specific to this VM. I'm going to close this bug. If it happens on other machines, I'll open a new one. Sorry for the trouble.