Not able to connect: err(9)

[onlynishant]

I am trying to use aerospike-c client but not able to connect to aerospike server (running in docker) Here is my code:

#include <aerospike/aerospike.h>
#include <aerospike/as_error.h>

int main(){
  as_config config;
  as_config_init(&config);
  config.conn_timeout_ms = 10000;

  as_config_add_host(&config, "192.168.99.100", 3000);

  aerospike as;
  aerospike_init(&as, &config);
  printf( "%s\n", "I am here");
  as_error err;
  if (aerospike_connect(&as, &err) != AEROSPIKE_OK) {
    printf("err(%d) %s at [%s:%d]\n", err.code, err.message, err.file, err.line);
    aerospike_destroy(&as);
    return -1;
  }
  aerospike_destroy(&as);
  return 0;
}

I am not able to connect:

$] gcc  -o aerospike_c_test aerospike_new_test.c -laerospike -lev -lssl -lcrypto -lpthread -lm -lz -lrt
$] ./aerospike_c_test
I am here
err(9)  at [(null):0]

I tried the similar code in Golang and it worked. Can someone please help me to understand what's going wrong? I have installed aerospike-c-client it with source code:

./install_libev
git submodule update --init
make EVENT_LIB=libev
sudo make install

[BrianNichols]

What is the error message?

[onlynishant]

 if (aerospike_connect(&as, &err) != AEROSPIKE_OK) {
    printf("err(%d) %s at [%s:%d]\n", err.code, err.message, err.file, err.line);

err(9) at [(null):0]

That's all I got.

[BrianNichols]

That is the low level socket timeout error. I don't think your server is reachable. Can you ping 192.168.99.100 from the client side?

[onlynishant]

Yes, I am able to ping.

$] ping 192.168.99.100
PING 192.168.99.100 (192.168.99.100) 56(84) bytes of data.
64 bytes from 192.168.99.100: icmp_seq=1 ttl=63 time=0.403 ms
64 bytes from 192.168.99.100: icmp_seq=2 ttl=63 time=0.277 ms
64 bytes from 192.168.99.100: icmp_seq=3 ttl=63 time=0.805 ms
64 bytes from 192.168.99.100: icmp_seq=4 ttl=63 time=0.593 ms
64 bytes from 192.168.99.100: icmp_seq=5 ttl=63 time=0.731 ms
64 bytes from 192.168.99.100: icmp_seq=6 ttl=63 time=0.575 ms
64 bytes from 192.168.99.100: icmp_seq=7 ttl=63 time=0.609 ms
64 bytes from 192.168.99.100: icmp_seq=8 ttl=63 time=0.571 ms
64 bytes from 192.168.99.100: icmp_seq=9 ttl=63 time=0.457 ms
64 bytes from 192.168.99.100: icmp_seq=10 ttl=63 time=0.563 ms
64 bytes from 192.168.99.100: icmp_seq=11 ttl=63 time=0.537 ms
^C
--- 192.168.99.100 ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 10498ms
rtt min/avg/max/mdev = 0.277/0.556/0.805/0.139 ms

It works with Go-client without any issue but doesn't work with C-client. Could it be related to C-client installation? If Yes, how can i debug it?

[BrianNichols]

First, subscribe to the client log:

static bool
example_log_callback(as_log_level level, const char * func, const char * file, uint32_t line, const char * fmt, ...)
{
    va_list ap;
    va_start(ap, fmt);
    vprintf(fmt, ap);
    printf("\n");
    va_end(ap);
    return true;
}

as_log_set_level(AS_LOG_LEVEL_DEBUG);
as_log_set_callback(example_log_callback);

If that doesn't help, try running the program in a debugger.

[onlynishant]

After enabling the client log, this is what I got. It is trying to connect to 172.17.0.2 (internal IP of aerospike docker container) which I have not mentioned anywhere in the code. Strange!

Add node BB9020011AC4202 172.17.0.2:3000
err(9)  at [(null):0]

when i enabled Debug log for go-client:, it's connecting to 192.168.99.100 as intended

logger2018/10/12 04:43:32 No connections available; seeding...
logger2018/10/12 04:43:32 Seeding the cluster. Seeds count: 1
logger2018/10/12 04:43:32 Node Validator has 1 nodes and they are: [192.168.99.100:3000]
logger2018/10/12 04:43:32 Adding node BB9020011AC4202 (192.168.99.100:3000) to the cluster.
logger2018/10/12 04:43:32 Node BB9020011AC4202 partition generation -2 changed to 0
logger2018/10/12 04:43:32 Tend finished. Live node count changes from 0 to 1
logger2018/10/12 04:43:32 New cluster initialized and ready to be used...

Then I modified aerospike.conf of Docker container:

        service {
                address any
                port 3000

                # Uncomment the following to set the `access-address` parameter to the
                # IP address of the Docker host. This will the allow the server to correctly
                # publish the address which applications and other nodes in the cluster to
                # use when addressing this node.
                 access-address  192.168.99.100
        }

I added access-address 192.168.99.100 line and then I am able to connect using c-client.

Any idea, why was it different on both client?

[BrianNichols]

The C client is attempting to detect a load balancer address (that is sometimes configured to avoid having to provide all seeds when connecting). The C client asks the node for it's real address. The server provides the "access_address". The problem is the server's "access-address" defaults to the local IP address.

It looks like the go client doesn't yet have this extra functionality. What versions of C client, Go client, and server are you using?

[onlynishant]

Aerospike server version: 3.14.1.1 (Docker on MAC)

C client:

commit a145ca8aa0b2ec6491ff728e4d7fce68575ee597
Author: Brian Nichols <bnichols@aerospike.com>
Date:   Thu Oct 11 10:58:23 2018 -0700

    Update version 4.3.19

Go-client was:

commit a769a21c4817057512f54a9718872175f3a48896
Author: Khosrow Afroozeh <parshua@gmail.com>
Date:   Thu Nov 30 21:02:24 2017 +0100

I updated Go-client now:

commit 69655035047d39963069ae4e6d046f0f03110792
Author: Khosrow Afroozeh <parshua@gmail.com>
Date:   Fri Oct 12 20:46:26 2018 +0200

and it it still able to connect to cluster (after commenting access-address from config) while C-client has same error.

2018/10/15 05:51:16 No connections available; seeding...
2018/10/15 05:51:16 Seeding the cluster. Seeds count: 1
2018/10/15 05:51:16 Node Validator has 1 nodes and they are: [192.168.99.100:3000]
2018/10/15 05:51:16 Host `192.168.99.100:3000` seems to be a load balancer. It is going to be replace by `172.17.0.2:3000`
2018/10/15 05:51:31 Connection to address `172.17.0.2:3000` failed to establish with error: dial tcp 172.17.0.2:3000: i/o timeout
2018/10/15 05:51:31 Inaccessible address `192.168.99.100:3000` as cluster seed. access-address is probably not configured on server.
2018/10/15 05:51:31 Adding node BB9020011AC4202 (192.168.99.100:3000) to the cluster.
2018/10/15 05:51:31 Node 192.168.99.100:3000 partition generation changed from -2 to 0
2018/10/15 05:51:31 Tend finished. Live node count changes from 0 to 1
2018/10/15 05:51:31 New cluster initialized and ready to be used...
2018/10/15 05:51:31 Starting the cluster tend goroutine...
test:set:key value goes here and can be any supported primitive:dc e5 f8 39 95 fd 2e 2f 5e 77 aa 45 a7 ee 84 fe 1b f4 fb 62  map[bin1:value1 bin2:value2] <nil>2018/10/15 05:51:31 Tend channel closed. Shutting down the cluster...

[khaf]

It looks like the go client doesn't yet have this extra functionality. What versions of C client, Go client, and server are you using?

The Go client does indeed support load balancers as seeds.

[BrianNichols]

I see the problem now. C client sockets are set to non-blocking mode even when using synchronous commands. This means socket connect() can initially return success on an non-existent address, but the first socket command will fail. The code was only calling connect() and assumed the success result code was enough to verify.

I have modified the code to also ask the node for its name and verify with original node's name. This extra socket call is sufficient to verify whether the IP address really does exist and is available.

This will appear in the next C client release.

[BrianNichols]

C client 4.3.20 has been released.

https://www.aerospike.com/download/client/c/4.3.20/