docker run: Server selection timeout, current topology ReplicaSetNoPrimary

[justintilson]

I've mostly copied your docker-compose from here: https://github.com/rwynn/monstache/blob/rel6/docker/test/docker-compose.test.yml and scripts from https://github.com/rwynn/monstache/tree/rel6/docker/test/mongodb/scripts. I've included my docker-compose.yml at the end of this post.

docker run --rm --env-file .env --network ptrac --name monstache rwynn/monstache:6.7.1

I've tried the root user, admin user, and app user with the same results:

ERROR 2020/11/25 20:46:11 Unable to connect to MongoDB using URL mongodb://REDACTED@mongo:27017/?replicaSet=rs0: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: 127.0.0.1:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp 127.0.0.1:27017: connect: connection refused }, ] }

The docker run command appears to get reading the .env file correctly. The contents are below

MONSTACHE_MONGO_URL=mongodb://root:password@mongo:27017/?replicaSet=rs0

Mongo is up and running. When I login to the container and then mongo, it looks like the replication set is configured correctly:

$ docker exec -it mongo /bin/bash
root@mongo:/scripts# mongo
MongoDB shell version v4.4.1
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("2348a2c4-60d3-4d4c-bebc-bb46ac63ba4f") }
MongoDB server version: 4.4.1
rs0:PRIMARY> rs.status()
{
    "set" : "rs0",
    "date" : ISODate("2020-11-25T21:08:34.890Z"),
    "myState" : 1,

My docker-compose.yml

version: '3.7'
services:
  mongo:
    image: mongo
    hostname: mongo
    logging:
      driver: none
    container_name: mongo
    command: /scripts/mongo-run.sh
    working_dir: /scripts
    volumes:
      - ./scripts/mongo:/scripts
      - ./data/mongo:/data/db
    environment:
      MONGO_REPLICA_SET_NAME: rs0
      MONGO_REPLICAS: mongo:27017
      MONGO_REPLICA_SET_MEMBERS: "[{'_id':0,'host':'mongo:27017','priority':1}]"
      MONGO_USER_ROOT_NAME: root
      MONGO_USER_ROOT_PASSWORD: password
      MONGO_AUTH_SOURCE: admin
      MONGO_BIND_IP: "0.0.0.0"
      MONGO_DB_NAME: ptrac
    ports:
      - "27017:27017"
    networks:
      - ptrac
    healthcheck:
      test: "[ -f /data/health.check ] && exit 0 || exit 1"
      interval: 1s
      timeout: 30s
      retries: 300
    restart: unless-stopped
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.10.0
    container_name: elasticsearch
    hostname: elasticsearch
    environment:
      - xpack.security.enabled=false
      - discovery.type=single-node
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add:
      - IPC_LOCK
    volumes:
      - ./data/elasticsearch:/usr/share/elasticsearch/data
      - ./config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
    ports:
      - 9200:9200
      - 9300:9300
    networks:
      - ptrac
volumes:
  data:
  scripts:
networks:
  ptrac:
    external: true

[rwynn]

Hi @justintilson is it possible that monstache is not ending up on the same docker network as the elasticsearch and mongodb services for some reason? Can you verify with docker inspect <containername>?

I know that docker compose will prefix network names, but it shouldn't be doing that for external type networks like you have afaik.

[justintilson]

Thx for getting back to me Ryan. I'm blocked on this for a few days with hardware issues on my dev machine. I'll be working again this weekend and will follow up then.

[DSamuylov]

Hi @rwynn, I am having a similar issue. I basically start MongoDB, Elasticsearch and Monstache with separate commands, and I do not specify any network in docker-compose.yml. So the 3 services are running in different networks.

I can connect to MongoDB and Elasticsearch:

1) from Safari I can see messages on the default ports 2) from Python client using the same connection string that I provide to Monstache:

import pymongo

client = pymongo.MongoClient(
    "mongodb://admin:test@localhost:27017/?replicaSet=my-mongo-cluster"
)

However I get the error:

Attaching to monstache
monstache            | ERROR 2021/04/28 20:25:03 Unable to connect to MongoDB using URL mongodb://REDACTED@localhost:27017/?replicaSet=my-mongo-cluster: server selection error: server selection timeout, current topology: { Type: ReplicaSetNoPrimary, Servers: [{ Addr: localhost:27017, Type: Unknown, Average RTT: 0, Last error: connection() error occured during connection handshake: dial tcp 127.0.0.1:27017: connect: connection refused }, ] }

It is especially strange to see "Type: ReplicaSetNoPrimary", whereas I can see using rs.status() that one of the members is Primary:

...
  members: [
    {
      _id: 0,
      name: 'localhost:27017',
      health: 1,
      state: 1,
      stateStr: 'PRIMARY',
...

Is it a hard requirement that the containers have to be in the same network? What if I would like to deploy monstache on a separate server from Elasticsearch and Mongodb?

[DSamuylov]

Ok, I figured it out. Just for other people for the future, this is the answer that helped me to understand the issue: https://stackoverflow.com/questions/53887738/server-selection-timeout-error-mongodb-go-driver-with-docker

Basically, you cannot use localhost in this case. Instead you create an external network and attach all containers to that network.

[luharprashant]

Can you pls share the solution, below is my docker file and I'm trying to connect like clientOptions := options.Client().ApplyURI("mongodb://datastore:27017")

version: '3'
services:
  datastore:
    image: mongo
    restart: always
    ports:
      - 27017:27017
    volumes:
      - ./mongodata/db:/data/db
      - ./mongodata/shared:/usr/games
  web:
    restart: always
    build: .
    ports: ["50051:50051"]
    depends_on:
      - datastore
#  mongo-express:
#    image: mongo-express
#    restart: always
#    ports:
#      - 8081:8081

[povesteam]

You are using a Go driver, so you can try forcing a direct connection using connect=direct option, so that mongo driver ignores the replica set hosts and uses only the provided seed.

Your connection URL could look like this:

mongodb://user:password@mongo:27017/?connect=direct

You can also try with the directConnection=true option, for if your go mongo driver is newer than 1.4

Relevant links:

• go documentation
• mongo documentation
• mongo go driver source code

[okikechinonso]

Can you pls share the solution, below is my docker file and I'm trying to connect like clientOptions := options.Client().ApplyURI("mongodb://datastore:27017")

version: '3'
services:
  datastore:
    image: mongo
    restart: always
    ports:
      - 27017:27017
    volumes:
      - ./mongodata/db:/data/db
      - ./mongodata/shared:/usr/games
  web:
    restart: always
    build: .
    ports: ["50051:50051"]
    depends_on:
      - datastore
#  mongo-express:
#    image: mongo-express
#    restart: always
#    ports:
#      - 8081:8081

version: "3.9"  # optional since v1.27.0

services:
  countries-api:
    build:
      context: .
      dockerfile: dockerfile
    ports:
      - "8081:8081"
    networks:
      - api
    volumes:
      - .:/app
      - $GOPATH/pkg/mod:/go/pkg/mod
    depends_on:
      - db
  db:
    image: "mongo:latest"
    ports:
        - "44004:44004"
    volumes:
        - database-data:/data/db
    networks:
      - api

volumes:
  database-data:
    driver: local
networks:
  api:
    driver: bridge