search

bitnami / containers

Bitnami container images
https://bitnami.com
Other
3.37k stars 4.83k forks source link

[bitnami/mongodb-sharded] Shard Fails to Join the Cluster #39475

Closed sbaxter2908 closed 1 year ago

sbaxter2908 commented 1 year ago

Name and Version

bitnami/mongodb-sharded:6.0.6

What architecture are you using?

amd64

What steps will reproduce the bug?

In AWS ECS for EC2...

Make sure to have 3 ECS Instances t3.large. Each task will run on a different server. Mongos task is behind a network load balancer. Shard is attempting to connect through network load balancer.

Create a task definition for configsvr:

{
    "containerDefinitions": [
        {
            "name": "mongodb-configsvr-primary",
            "image": "bitnami/mongodb-sharded:6.0.6",
            "cpu": 1536,
            "memory": 6144,
            "portMappings": [
                {
                    "containerPort": 27017,
                    "hostPort": 27017,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "MONGODB_REPLICA_SET_MODE",
                    "value": "primary"
                },
                {
                    "name": "MONGODB_REPLICA_SET_NAME",
                    "value": "config-replicaset"
                },
                {
                    "name": "MONGODB_SHARDING_MODE",
                    "value": "configsvr"
                },
                {
                    "name": "MONGODB_ROOT_PASSWORD",
                    "value": "password123"
                },
                {
                    "name": "MONGODB_REPLICA_SET_KEY",
                    "value": "replicasetkey123"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "mongodb",
                    "containerPath": "/bitnami/mongodb"
                }
            ],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/mongodb",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "configsvr"
                }
            }
        }
    ],
    "family": "configsvr_primary",
    "taskRoleArn": "arn:aws:iam::156888724897:role/service-la-ecs",
    "networkMode": "host",
    "revision": 10,
    "volumes": [
        {
            "name": "certs",
            "host": {
                "sourcePath": "/data/pki"
            }
        },
        {
            "name": "mongodb",
            "host": {
                "sourcePath": "/mongodb"
            }
        }
    ],
    "status": "ACTIVE",
    "placementConstraints": [
        {
            "type": "memberOf",
            "expression": "attribute:configsvr == primary"
        }
    ],
    "compatibilities": [
        "EXTERNAL",
        "EC2"
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "tags": []
}

Create a task definition for mongos:

{
    "containerDefinitions": [
        {
            "name": "mongos_1",
            "image": "bitnami/mongodb-sharded:6.0.6",
            "cpu": 1536,
            "memory": 6144,
            "portMappings": [
                {
                    "containerPort": 27017,
                    "hostPort": 27017,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "MONGODB_SHARDING_MODE",
                    "value": "mongos"
                },
                {
                    "name": "MONGODB_CFG_REPLICA_SET_NAME",
                    "value": "config-replicaset"
                },
                {
                    "name": "MONGODB_ROOT_PASSWORD",
                    "value": "password123"
                },
                {
                    "name": "MONGODB_REPLICA_SET_KEY",
                    "value": "replicasetkey123"
                },
                {
                    "name": "MONGODB_CFG_PRIMARY_HOST",
                    "value": "10.113.133.4"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "mongodb",
                    "containerPath": "/bitnami/mongodb"
                }
            ],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/mongodb",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "mongos"
                }
            }
        }
    ],
    "family": "mongos_1",
    "taskRoleArn": "arn:aws:iam::156888724897:role/service-la-ecs",
    "networkMode": "host",
    "revision": 9,
    "volumes": [
        {
            "name": "certs",
            "host": {
                "sourcePath": "/data/pki"
            }
        },
        {
            "name": "mongodb",
            "host": {
                "sourcePath": "/mongodb"
            }
        }
    ],
    "status": "ACTIVE",
    "placementConstraints": [
        {
            "type": "memberOf",
            "expression": "attribute:mongos == 1"
        }
    ],
    "compatibilities": [
        "EXTERNAL",
        "EC2"
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "tags": []
}

Create a task definition for shard:

{
    "containerDefinitions": [
        {
            "name": "mongodb-shard0-primary",
            "image": "bitnami/mongodb-sharded:6.0.6",
            "cpu": 1536,
            "memory": 6144,
            "portMappings": [
                {
                    "containerPort": 27017,
                    "hostPort": 27017,
                    "protocol": "tcp"
                }
            ],
            "essential": true,
            "environment": [
                {
                    "name": "MONGODB_MONGOS_HOST",
                    "value": "mongodb-cluster-780b535f69350a6e.elb.us-east-1.amazonaws.com"
                },
                {
                    "name": "MONGODB_REPLICA_SET_NAME",
                    "value": "shard0"
                },
                {
                    "name": "MONGODB_SHARDING_MODE",
                    "value": "shardsvr"
                },
                {
                    "name": "MONGODB_REPLICA_SET_KEY",
                    "value": "replicasetkey123"
                },
                {
                    "name": "BITNAMI_DEBUG",
                    "value": "true"
                },
                {
                    "name": "MONGODB_REPLICA_SET_MODE",
                    "value": "primary"
                },
                {
                    "name": "MONGODB_ROOT_PASSWORD",
                    "value": "password123"
                }
            ],
            "mountPoints": [
                {
                    "sourceVolume": "mongodb",
                    "containerPath": "/bitnami/mongodb"
                }
            ],
            "volumesFrom": [],
            "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-create-group": "true",
                    "awslogs-group": "/ecs/mongodb",
                    "awslogs-region": "us-east-1",
                    "awslogs-stream-prefix": "shard0"
                }
            }
        }
    ],
    "family": "shard0_primary",
    "taskRoleArn": "arn:aws:iam::156888724897:role/service-la-ecs",
    "networkMode": "host",
    "volumes": [
        {
            "name": "certs",
            "host": {
                "sourcePath": "/data/pki"
            }
        },
        {
            "name": "mongodb",
            "host": {
                "sourcePath": "/mongodb"
            }
        }
    ],
    "status": "ACTIVE",   
    "placementConstraints": [
        {
            "type": "memberOf",
            "expression": "attribute:shard0 == primary"
        }
    ],
    "compatibilities": [
        "EXTERNAL",
        "EC2"
    ],
    "requiresCompatibilities": [
        "EC2"
    ],
    "tags": []
}

What is the expected behavior?

Shard successfully connects to the cluster

What do you see instead?

Shard Log:
mongodb 16:40:18.13 Welcome to the Bitnami mongodb-sharded container
mongodb 16:40:18.14 Subscribe to project updates by watching https://github.com/bitnami/containers
mongodb 16:40:18.14 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mongodb 16:40:18.14
mongodb 16:40:18.14 INFO  ==> ** Starting MongoDB Sharded setup **
mongodb 16:40:18.19 INFO  ==> Validating settings in MONGODB_* env vars...
mongodb 16:40:18.24 INFO  ==> Initializing MongoDB Sharded...
mongodb 16:40:18.27 INFO  ==> Writing keyfile for replica set authentication...
mongodb 16:40:18.29 INFO  ==> Enabling authentication...
mongodb 16:40:18.29 INFO  ==> Deploying MongoDB Sharded with persisted data...
mongodb 16:40:18.32 DEBUG ==> Waiting for primary node...
mongodb 16:40:18.32 INFO  ==> Trying to connect to MongoDB server mongodb-cluster-780b535f69350a6e.elb.us-east-1.amazonaws.com...
mongodb 16:40:18.34 INFO  ==> Found MongoDB server listening at mongodb-cluster-780b535f69350a6e.elb.us-east-1.amazonaws.com:27017 !
mongodb 16:40:22.06 INFO  ==> MongoDB server listening and working at mongodb-cluster-780b535f69350a6e.elb.us-east-1.amazonaws.com:27017 !
mongodb 16:40:23.17 DEBUG ==> Starting MongoDB in background...
{"t":{"$date":"2023-06-30T16:40:23.205Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"-","msg":"Applied --setParameter options","attr":{"serverParameters":{"enableLocalhostAuthBypass":{"default":true,"value":false}}}}
about to fork child process, waiting until server is ready for connections.
forked process: 105
child process started successfully, parent exiting
mongodb 16:40:25.28 INFO  ==> Joining the shard cluster
mongodb 16:44:24.79 ERROR ==> Unable to join the sharded cluster
mongodb 16:44:24.80 INFO  ==> Stopping MongoDB...

Additional information

Please let me know what else you need from me. Googling has failed me. Everybody seems to be trying to run this on Kubernetes and not ECS on EC2 instances

sbaxter2908 commented 1 year ago

Ok, I figured out what the problem was.

First, you can't use the load balancer FQDN for the MONGODB_MONGOS_HOST environment variable.

Second, set the MONGODB_ADVERTISED_HOSTNAME on each node to the hostname of the server.

Doing both of those fixed the issue