search

florinpatrascu / bolt_sips

Neo4j driver for Elixir
Apache License 2.0
256 stars 49 forks source link

No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER #86

Closed kalamarski-marcin closed 3 years ago

kalamarski-marcin commented 4 years ago

Environment:

Erlang/OTP 22 [erts-10.6.4] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1]

Elixir 1.9.4 (compiled with Erlang/OTP 22)

Kubernetes services

kubectl get svc -n neo4j
NAME                                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
sx-causal-cluster-neo4j                   ClusterIP   None         <none>        7474/TCP,7473/TCP,7687/TCP   19h
sx-causal-cluster-neo4j-readreplica-svc   ClusterIP   None         <none>        7474/TCP,7473/TCP,7687/TCP   19h

Service address: sx-causal-cluster-neo4j.neo4j.svc.cluster.local

The address is built according to the: https://github.com/neo-technology/neo4j-google-k8s-marketplace/blob/3.5/user-guide/USER-GUIDE.md#service-address

Kubernetes pods

kubectl get pods -n neo4j
NAME                               READY   STATUS      RESTARTS   AGE
sx-causal-cluster-deployer-dl8gz   0/1     Completed   0          19h
sx-causal-cluster-neo4j-core-0     1/1     Running     1          19h
sx-causal-cluster-neo4j-core-1     1/1     Running     0          19h
sx-causal-cluster-neo4j-core-2     1/1     Running     12         19h

Pods addresses:

sx-causal-cluster-neo4j-core-0.sx-causal-cluster-neo4j.neo4j.svc.cluster.local sx-causal-cluster-neo4j-core-1.sx-causal-cluster-neo4j.neo4j.svc.cluster.local sx-causal-cluster-neo4j-core-2.sx-causal-cluster-neo4j.neo4j.svc.cluster.local

Routing table Bolt.Sips.info()

%{
  default: %{
    connections: %{
      read: %{
        "sx-causal-cluster-neo4j-core-0.sx-causal-cluster-neo4j.neo4j.svc.cluster.local:7687" => 0,
        "sx-causal-cluster-neo4j-core-2.sx-causal-cluster-neo4j.neo4j.svc.cluster.local:7687" => 0
      },
      route: %{
        "sx-causal-cluster-neo4j-core-0.sx-causal-cluster-neo4j.neo4j.svc.cluster.local:7687" => 0,
        "sx-causal-cluster-neo4j-core-1.sx-causal-cluster-neo4j.neo4j.svc.cluster.local:7687" => 0,
        "sx-causal-cluster-neo4j-core-2.sx-causal-cluster-neo4j.neo4j.svc.cluster.local:7687" => 0
      },
      routing_query: %{
        params: %{context: %{}},
        query: "CALL dbms.cluster.routing.getRoutingTable({context})"
      },
      ttl: 300,
      updated_at: 1583695804,
      write: %{
        "sx-causal-cluster-neo4j-core-1.sx-causal-cluster-neo4j.neo4j.svc.cluster.local:7687" => 0
      }
    },
    user_options: [
      socket: Bolt.Sips.Socket,
      basic_auth: [username: "neo4j", password: "*******"],
      port: 7687,
      routing_context: %{},
      schema: "neo4j",
      hostname: "sx-causal-cluster-neo4j.neo4j.svc.cluster.local",
      max_overflow: 0,
      timeout: 15000,
      ssl: false,
      with_etls: false,
      prefix: :default,
      url: "neo4j://neo4j:pGeuKv647e@sx-causal-cluster-neo4j.neo4j.svc.cluster.local",
      pool_size: 1
    ]
  }
}

When I typed conn = Bolt.Sips.conn(:write) and tried to query the database Bolt.Sips.query!(conn, "CREATE (t:Test)") I got an error:

No write operations are allowed directly on this database. Writes must pass through the leader. The role of this server is: FOLLOWER

An error occurred couple of times, randomly. Somtimes I was able to query succesfully sometimes not. But definitely more often with a failure. Probably just beacuse Kubernetes service internally works as a load balancer for pods.

I was curious why.

To test what is going on I deployed the elixir docker image, entered to the pod, cloned Bolt.Sips repository and change some parts of the code.

After a while I discovered an interesting thing.

I inspected opts passed to the Bolt.Sips.Protocol.connect function:

[
  pool_index: 1,
  name: {:via, Registry,
   {:bolt_sips_registry,
    "default_write@sx-causal-cluster-neo4j-core-1.sx-causal-cluster-neo4j.neo4j.svc.cluster.local:7687"}},
  role: :write,
  hits: 0,
  port: 7687,
  host: 'sx-causal-cluster-neo4j-core-1.sx-causal-cluster-neo4j.neo4j.svc.cluster.local',
  socket: Bolt.Sips.Socket,
  basic_auth: [username: "neo4j", password: "*********"],
  routing_context: %{},
  schema: "neo4j",
  hostname: "sx-causal-cluster-neo4j.neo4j.svc.cluster.local",
  max_overflow: 0,
  timeout: 15000,
  ssl: false,
  with_etls: false,
  prefix: :default,
  pool_size: 1
]

Please have a look at the key :host and its value. It seems it looks OK. Actually, it is. But. The problem is, it is never used anywhere.

In Bolt.Sips.Protocol.connect function I've put IO.inspect(host) just before with {:ok, sock} <- socket.connect(host, port, socket_opts, timeout) and run:

{:ok, neo} = Sips.start_link(url: "neo4j://neo4j:*******@sx-causal-cluster-neo4j.neo4j.svc.cluster.local", pool_size: 1)
{:ok, #PID<0.281.0>}
'sx-causal-cluster-neo4j.neo4j.svc.cluster.local'
'sx-causal-cluster-neo4j.neo4j.svc.cluster.local'
'sx-causal-cluster-neo4j.neo4j.svc.cluster.local'
'sx-causal-cluster-neo4j.neo4j.svc.cluster.local'
'sx-causal-cluster-neo4j.neo4j.svc.cluster.local'
'sx-causal-cluster-neo4j.neo4j.svc.cluster.local'
'sx-causal-cluster-neo4j.neo4j.svc.cluster.local'
'sx-causal-cluster-neo4j.neo4j.svc.cluster.local'

All connections have been established directly to the Kubernetes service. None of them to the aforementioned host.

So I did a nasty hack in Bolt.Sips.Utils.default_config(opts) function:

hostname = if opts[:host], do: opts[:host], else: opts[:hostname]

    config
    |> Keyword.replace!(:hostname, hostname)
    |> Keyword.put_new(:port, System.get_env("NEO4J_PORT") || @default_bolt_port)
    |> Keyword.put_new(:pool_size, 5)
    |> Keyword.put_new(:max_overflow, 2)
    |> Keyword.put_new(:timeout, 15_000)
    |> Keyword.put_new(:ssl, false)
    |> Keyword.put_new(:socket, Bolt.Sips.Socket)
    |> Keyword.put_new(:with_etls, false)
    |> Keyword.put_new(:schema, "bolt")
    |> Keyword.put_new(:path, "")
    |> Keyword.put_new(:prefix, :default)
    |> or_use_url_if_present
    |> Enum.reject(fn {_k, v} -> is_nil(v) end)
    |> Keyword.put(:socket, ssl_or_sock)

And I run it again:

{:ok, neo} = Sips.start_link(url: "neo4j://neo4j:******@sx-causal-cluster-neo4j.neo4j.svc.cluster.local", pool_size: 1)
sx-causal-cluster-neo4j.neo4j.svc.cluster.local
sx-causal-cluster-neo4j.neo4j.svc.cluster.local
sx-causal-cluster-neo4j-core-0.sx-causal-cluster-neo4j.neo4j.svc.cluster.local
sx-causal-cluster-neo4j-core-2.sx-causal-cluster-neo4j.neo4j.svc.cluster.local
sx-causal-cluster-neo4j-core-0.sx-causal-cluster-neo4j.neo4j.svc.cluster.local
sx-causal-cluster-neo4j-core-1.sx-causal-cluster-neo4j.neo4j.svc.cluster.local
sx-causal-cluster-neo4j-core-2.sx-causal-cluster-neo4j.neo4j.svc.cluster.local
sx-causal-cluster-neo4j-core-1.sx-causal-cluster-neo4j.neo4j.svc.cluster.local

So, in the end, :get_tcp.connect received proper host each time invoked.

In that way I can query the database (write operation) without any problems. Always.

I've tested it locally. Neo4j launched via docker-compose. Here is docker-compose.yml

To make it work I had to find out IP address of the each running container and edit /etc/hosts file. In my case:

172.21.0.3 core1
172.21.0.2 core2
172.21.0.5 core3

Without it host couldn't be properly resolved.

Maybe I misunderstood the driver concept and Bolt.Sips is OK but my Neo4j configuration is bad (Deployed via Google Cloud Marketplace and I didn't change anything in the config).

@florinpatrascu What do you think?

Edited:

I've checked how it works under the hood in nodejs neo4j-driver. There is a function (src/internal/node/node-channel.js):

function connect (config, onSuccess, onFailure = () => null) {
  const trustStrategy = trustStrategyName(config)
  if (!isEncrypted(config)) {
    const socket = net.connect(
      config.address.port(),
      config.address.resolvedHost(),
      onSuccess
    )
    socket.on('error', onFailure)
    return configureSocket(socket)
  } else if (TrustStrategy[trustStrategy]) {
    return TrustStrategy[trustStrategy](config, onSuccess, onFailure)
  } else {
    onFailure(
      newError(
        'Unknown trust strategy: ' +
          config.trust +
          '. Please use either ' +
          "trust:'TRUST_CUSTOM_CA_SIGNED_CERTIFICATES' or trust:'TRUST_ALL_CERTIFICATES' in your driver " +
          'configuration. Alternatively, you can disable encryption by setting ' +
          '`encrypted:"' +
          ENCRYPTION_OFF +
          '"`. There is no mechanism to use encryption without trust verification, ' +
          'because this incurs the overhead of encryption without improving security. If ' +
          'the driver does not verify that the peer it is connected to is really Neo4j, it ' +
          'is very easy for an attacker to bypass the encryption by pretending to be Neo4j.'
      )
    )
  }
}

So I added console.log(config.address.resolvedHost()) just before net.connect. And here is the result:

/nodejs # node test.js
Resolved address 10.20.0.53
Resolved address sx-causal-cluster-neo4j-core-1.sx-causal-cluster-neo4j.neo4j.svc.cluster.local

The first is IP of the Kubernetes service. The second the leader address.

test.js file:

var neo4j = require('neo4j-driver')

var driver = neo4j.driver(
  'neo4j://sx-causal-cluster-neo4j.neo4j.svc.cluster.local',
  neo4j.auth.basic('neo4j', '********')
)

var session = driver.session({ defaultAccessMode: neo4j.session.WRITE })

session
  .run('create (b:B) return b', {
    nameParam: 'Alice'
  })
  .subscribe({
    onKeys: keys => {
      console.log(keys)
    },
    onNext: record => {
      console.log(record)
    },
    onCompleted: () => {
      session.close() // returns a Promise
    },
    onError: error => {
      console.log(error)
    }
  })
florinpatrascu commented 4 years ago

hi @kalamarski-marcin and thank you, for the very detailed report! We work a bit different than the js driver, that's for sure, but it's interesting to see how other fellows are coding ;) However, as far as identifying the server roles we should yield the correct behavior.

I'll look into the host param story, as it looks like you might've found a bug, sorry for that.

One aspect that worries me though is related to this:

An error occurred couple of times, randomly. Somtimes I was able to query succesfully sometimes not. But definitely more often with a failure. Probably just beacuse Kubernetes service internally works as a load balancer for pods.

I don't have a k8s cluster available to test with hence appreciating any feedback we get, but are you suspecting that k8s lb might load balance the traffic, say: between pods hosting neo4j servers of different roles?

From Bolt.Sips's perspective, the role of a connection i.e. :write, will instruct the driver what server addresses it should use, from the config, and only the servers having the :write roles will be used (in a round robin way..., if they are more than one) - this is not affected by the host param, unless I missed something. The address of the servers is refreshed after the route TTL, and they're configured dynamically upon the route refresh.

The server address you have in the driver config, will be that of the router, but you already know this.

kalamarski-marcin commented 4 years ago

Hi,

Thx for fast reply :)

I don't have a k8s cluster available to test with hence appreciating any feedback we get, but are you suspecting that k8s lb might load balance the traffic, say: between pods hosting neo4j servers of different roles?

In a short pod = core server. Each pod has more or less the same configuration. What makes the difference is advertised_address config option set differently for each pod. All of them have the same ports opened: 5000, 6000, 7000, 7473,7474, 7687.

According to the Kubernetes docs (service): Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them.

My service has opened ports 7474, 7473, 7687 and forwards the traffic to the ports 7474, 7473, 7687 respectively. So if all pods have opened i.a. 7867 port, theory that the service lb the traffic is the only one which is reasonable for me. That is why I got an error randomly.

I've made one more test. I did everything locally. I used your docker-compose.yml and started the Neo4j. Then I changed socket.ex file:

def connect(host, port, opts, timeout) do
    IO.puts "#{host}:#{port}"

    :gen_tcp.connect(host, port, opts, timeout)
 end

And run:

iex(1)> {:ok, neo} = Sips.start_link(url: "bolt+routing://neo4j:test@localhost:7687", pool_size: 1)
{:ok, #PID<0.280.0>}
localhost:7687
iex(2)> localhost:7687
localhost:7688
localhost:7689
localhost:7687
localhost:7688
localhost:7689
localhost:7687

As you can see it will work perfectly, I mean writing operations, because localhost in that case doesn't matter. What is important is used port in the particular connection. That is why :write, :read, :route connections are established to the right core server, always.

But I my case ports are the same and the most important thing is the host which, I think, must be the address of the pod not the service.

florinpatrascu commented 3 years ago

it seems that no one else can reproduce this, can this be closed? I'll close it and we can reopen it in case we get more reports or receive any PR to address it.