/_node/<nodename>/_system endpoint returns HTTP 500 when TLS is configured

[jamieluckett]

Description

1. Setup a clustered CouchDB environment with TLS configured in etc/vm.args

   # Snippet from our vm.args
   -pa /opt/opsview/erlang/lib/erlang/lib/ssl-9.6/ebin
   -proto_dist inet_tls
   -ssl_dist_optfile /opt/opsview/datastore/etc/ssl.args
   -ssl session_lifetime 300

2. Attempt to call the _node/<nodename>/_system endpoint.

curl user:password@localhost:5984/_node/<nodename>/_system

{
  "error": "invalid_ejson",
  "reason": "{badrpc,\n    {'EXIT',\n        {function_clause,\n            [{prim_inet,getstat,\n                 [<0.328.0>,\n                  [recv_oct,recv_cnt,recv_max,recv_avg,recv_dvi,send_oct,\n                   send_cnt,send_max,send_avg,send_pend]],\n                 []},\n             {chttpd_node,'-get_distribution_stats/0-fun-0-',1,\n                 [{file,\"src/chttpd_node.erl\"},{line,278}]},\n             {lists,map,2,[{file,\"lists.erl\"},{line,1239}]},\n             {chttpd_node,get_stats,0,\n                 [{file,\"src/chttpd_node.erl\"},{line,230}]},\n             {rpc,local_call,3,[{file,\"rpc.erl\"},{line,321}]},\n             {chttpd_node,call_node,4,\n                 [{file,\"src/chttpd_node.erl\"},{line,187}]},\n             {chttpd_node,handle_node_req,1,\n                 [{file,\"src/chttpd_node.erl\"},{line,119}]},\n             {chttpd,handle_req_after_auth,2,\n                 [{file,\"src/chttpd.erl\"},{line,323}]}]}}}",
  "ref": 1035438607
}

Expected Behaviour

Endpoint shouldn't return an HTTP 500 status and should contain the expected data as documented.

Your Environment

localhost:5984/

{
  "couchdb": "Welcome",
  "version": "3.1.1",
  "git_sha": "ce596c65d",
  "uuid": "41595977eddf264776561a5e3f4060b5",
  "features": [
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "vendor": {
    "name": "The Apache Software Foundation"
  }
}

localhost:5984/_membership

{
  "all_nodes": [
    "couchdb@jl-42569-solo-ds-1.os.opsview.com",
    "couchdb@jl-42569-solo-ds-2.os.opsview.com",
    "couchdb@jl-42569-solo-orch.os.opsview.com"
  ],
  "cluster_nodes": [
    "couchdb@jl-42569-solo-ds-1.os.opsview.com",
    "couchdb@jl-42569-solo-ds-2.os.opsview.com",
    "couchdb@jl-42569-solo-orch.os.opsview.com"
  ]
}

• CouchDB version used: 3.1.1
• Operating system and version: Ubuntu 18.04.5 LTS
• Erlang Version: 22 and 21

Additional Context

TLS is set up and enabled for communication between all nodes.

Spoke a fair bit with jan and rnewson on the CouchDB slack, it seems like Erlang 22.x has changed dist_ctrl a fair bit causing compatability issues with the current implementation of chttpd_node.

[weareu]

We are also getting this issue on 3.1.1, we monitor _system in various scenarios for restarts on the cluster and having issues lately on 3.1.1 that are hard to describe or pinpoint.

Is there any solution?

[Liquid-NeoN]

We seeing similar behaviour when calling the /_node/_local/_system end point, but we get a different error. Environment specs and /_node/_local/_system response below.

Operating system and version: Ubuntu 18.04.6 LTS

/

{
    "couchdb": "Welcome",
    "version": "3.2.1",
    "git_sha": "244d428af",
    "uuid": "2c788d3183f607ee7cb432038b940f78",
    "features": [
        "access-ready",
        "partitioned",
        "pluggable-storage-engines",
        "reshard",
        "scheduler"
    ],
    "vendor": {
        "name": "The Apache Software Foundation"
    }
}

/_membership

{
    "all_nodes": [
        "couchdb@db1.xxx.com",
        "couchdb@db2.xxx.com",
        "couchdb@db3.xxx.com",
        "couchdb@db4.xxx.com"
    ],
    "cluster_nodes": [
        "couchdb@db1.xxx.com",
        "couchdb@db2.xxx.com",
        "couchdb@db3.xxx.com",
        "couchdb@db4.xxx.com"
    ]
}

/_node/_local/_system

{
    "error": "invalid_ejson",
    "reason": "{badrpc,\n    {'EXIT',\n        {function_clause,\n            [{prim_inet,getstat,\n                 [<0.12402.0>,\n                  [recv_oct,recv_cnt,recv_max,recv_avg,recv_dvi,send_oct,\n                   send_cnt,send_max,send_avg,send_pend]],\n                 []},\n             {chttpd_node,'-get_distribution_stats/0-fun-0-',1,\n                 [{file,\"src/chttpd_node.erl\"},{line,312}]},\n             {lists,map,2,[{file,\"lists.erl\"},{line,1243}]},\n             {chttpd_node,get_stats,0,\n                 [{file,\"src/chttpd_node.erl\"},{line,264}]}]}}}",
    "ref": 878456848
}