docker: `start-single-node` doesn't support customized listen addr

[Leka4kin]

Describe the problem

According to docs, running in start-single-node mode I can set --listen-addr param and by default it sets to all ip addreses on interface( i.e. 0.0.0.0 ?) https://www.cockroachlabs.com/docs/v22.1/cockroach-start-single-node.html#networking

But in cockroach.sh, which is entrypoint for docker image, hardcoded localhost value for this var. => I cant run single-node that listens on anything but localhost

To Reproduce

Run cockroach in docker with args "start-single-node --listen-addr=192.168.0.2:26257"

Expected behavior

Not throwing error error: hostname of listen_addr must be \"$default_listen_addr_host\" or \"localhost\" Running on the addresses i set in listen-addr

Additional data / screenshots

Environment:

• cockroachdb/cockroach:v22.1.1
• Server OS: debian

Jira issue: CRDB-17503

[blathers-crl[bot]]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I was unable to automatically find someone to ping.

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[rafiss]

Thanks for the report!

cc @ZhouXing19 do you happen to remember why this restriction was needed?

[knz]

@ZhouXing19 and I just chatted about this. I will let Jane follow up with additional questions, but I'd like to answer this specific question from Rafi:

do you happen to remember why this restriction was needed?

The restriction is mainly because if we let customize --listen-addr, the other CLI client commands inside the docker init script will not know how to connect to the process (we'd need to teach the script to extract the value and pass it to the client commands).

[ZhouXing19]

Hi @Leka4kin, thanks for filing this issue. We'd like to know more about the specific use case of a customized --listen-addr, and also wonder if you can achieve the same goal by configuring the docker network setting.

[danielporto]

Hi there, any update on this? I'm using version 22.2.5 and it seem that the issue still remains. I have my dev env with traefik which redirects http calls to the respective container. However, since the cockroachdb only accepts localhost, traefik is unable to open the webcli page which is annoying. I dont want to expose the ports.

[serdarkalayci]

Hi @Leka4kin, thanks for filing this issue. We'd like to know more about the specific use case of a customized --listen-addr, and also wonder if you can achieve the same goal by configuring the docker network setting.

The restriction makes it impossible for other containers inside a docker-compose file reach the crdb via its service name, which resolves to IP address assigned by the docker network.

version: '3.7'
services:
  document:
    build: api/document/.
    container_name: document
    ports:
      - "5550:5550"
    environment: 
      BASE_URL : ":5550"
      ConnectionString: "document-db:26257"
      DatabaseName: docman
      DbUserName: docmanuser
      DbPassword: docmanpassword
    networks:
      - docman-network
    depends_on:
      - document-db
  document-db:
    image: cockroachdb/cockroach:latest
    container_name: document-db
    hostname: document-db
    ports:
      - 26257:26257
      - 8080:8080
    environment:
      COCKROACH_DATABASE: docman
      COCKROACH_USER: docmanuser
      COCKROACH_PASSWORD: docmanpassword
    command: start-single-node
    networks:
      - docman-network
    volumes:
      - ./api/database/document:/docker-entrypoint-initdb.d
      - documentdbdata:/cockroach/cockroach-data"
networks:
  docman-network:
    driver: bridge
volumes:
  documentdbdata:

I would assume a docker-compose file as this would work (which does for all other db engines including postgres) but I got this from the api 2023-05-06 18:42:49 2023/05/06 16:42:49 failed to connect tohost=document-db user=docmanuser database=docman: dial error (dial tcp 192.168.48.2:26257: connect: connection refused)

[Hennns]

Hi @Leka4kin, thanks for filing this issue. We'd like to know more about the specific use case of a customized --listen-addr, and also wonder if you can achieve the same goal by configuring the docker network setting.

Just throwing my 2 cents in with another use case - I was testing upgrading on a single node (before moving onto multiple nodes later) and reading trough the breaking changes listed here: https://www.cockroachlabs.com/docs/releases/v22.2#v22-2-0-backward-incompatible-changes

Specifically this part (from https://github.com/cockroachdb/cockroach/pull/85671):

Preferred: keep port 26257 for SQL, and allocate a new port, e.g., 26357, for node-node RPC connections. For example, you might configure a node with the flags --listen-addr=:26357 --sql-addr=:26257, where subsequent nodes seeking to join would then use the flag --join=othernode:26357,othernode:26257. This will become the default configuration in the next version of CockroachDB. When using this mode of operation, care should be taken to use a --join flag that includes both the previous and new port numbers for other nodes, so that no network partition occurs during the upgrade.

I was naturally surprised when this did not work, and the error message contradicts the preferred example. (In my case the error was on the port, but I assume it's the same underlaying issue)

error: port of listen_addr must be "26257"

[dwt]

@devs: Sorry is there any activity here? I don't get how I am supposed to run a single node cluster as part of a docker compose development environment with the --listen-address hardcoded to localhost?

That way I can never access the cluster from other hosts in the compose network?

Am I missing something?

[knz]

Have you considered docker run <image> /cockroach/cockroach <flags> instead of using the default RUN to cockroach.sh? i.e. run the cockroach binary directly?

When you do that you need to set up your TLS config yourself (cockroach.sh won't help you) but you can control the flags.

[bobvawter]

I stumbled into the overly-restrictive address check this morning. Consider the following docker-compose configuration for testing changefeeds, where we want multiple instances of cockroach that can also talk to the test rig running on the docker host.

services:
  source-cockroachdb-v23.1:
    image: cockroachdb/cockroach:latest-v23.1
    network_mode: host
    command: start-single-node --insecure --store type=mem,size=2G --listen-addr :5400 --http-addr :8081
  target-cockroachdb-v23.1:
    image: cockroachdb/cockroach:latest-v23.1
    network_mode: host
    command: start-single-node --insecure --store type=mem,size=2G --listen-addr :5401 --http-addr :8082

Adding entrypoint: /cockroach/cockroach worked around it, but it took a while to determine that my docker containers weren't starting with command-lines that I had verified locally.

[jkemp101]

My hack was to use this entrypoint:

    entrypoint:
      [
        "/bin/bash",
        "-c",
        "sed -i 's/default_listen_addr_host=\"127.0.0.1\"/default_listen_addr_host=\"0.0.0.0\"/' /cockroach/cockroach.sh && /cockroach/cockroach.sh start-single-node --certs-dir=/cockroach/cockroach-data/certs"
      ]

[kulogix]

This is more of an issue now that cockroach.sh has been enhanced to support COCKROACH_DATABASE, COCKROACH_USER, and COCKROACH_PASSWORD, as well as /docker-entrypoint-initdb.d.

Without setting listen-addr, from my understanding, the following error will also occur when trying to connect with SSL client cert verification: (tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match ...)

UPDATE: Reading through https://github.com/cockroachdb/cockroach/blob/master/build/deploy/cockroach.sh the quick fix is to just use --advertise-addr=_YOUR_CONTAINERNAME

UPDATE2: To get around "Running a server without --sql-addr, with a combined RPC/SQL listener, is deprecated.", per https://www.cockroachlabs.com/docs/releases/v22.2:

Preferred: keep port 26257 for SQL, and allocate a new port, e.g., 26357, for node-node RPC connections. For example, you might configure a node with the flags --listen-addr=:26357 --sql-addr=:26257, where subsequent nodes seeking to join would then use the flag --join=othernode:26357,othernode:26257. This will become the default configuration in the next version of CockroachDB. When using this mode of operation, care should be taken to use a --join flag that includes both the previous and new port numbers for other nodes, so that no network partition occurs during the upgrade.

However, cockroach.sh returns: error: port of listen_addr must be "26257"

Would suggest updating docs and/or cockroach.sh