[Mongo plugin] vault read database/creds/<role> throws "not master" error after primary step down

[zhecksum]

Describe the bug

Mongo version 4.0+ returns a "not master" error instead of an EOF error when Vault tries to connect after a step down of the replica set primary.

/plugins/database/mongodb/mongodb.go:

case err == io.EOF, strings.Contains(err.Error(), "EOF"):

needs to be changed to

case err == io.EOF, strings.Contains(err.Error(), "EOF"), strings.Contains(err.Error(), "not master"):

to catch the "not master" error message thrown by Mongo version 4.0+.

To Reproduce Steps to reproduce the behavior:

vault server -dev -dev-root-token-id='abc' -log-level=trace

export VAULT_ADDR=http://127.0.0.1:8200

vault secrets enable database

# create network
docker network create my-mongo-cluster

# create mongos
docker run -d --net my-mongo-cluster -p 27017:27017 --name mongo1 mongo:4.2 mongod --replSet my-mongo-set --port 27017
docker run -d --net my-mongo-cluster -p 27018:27017 --name mongo2 mongo:4.2 mongod --replSet my-mongo-set --port 27017
docker run -d --net my-mongo-cluster -p 27019:27017 --name mongo3 mongo:4.2 mongod --replSet my-mongo-set --port 27017

CMD="mongo <<EOF
db = (new Mongo('localhost:27017')).getDB('test')
config={'_id':'my-mongo-set','members':[{'_id':0,'host':'mongo1:27017'},{'_id':1,'host':'mongo2:27017'},{'_id':2,'host':'mongo3:27017'}]}
rs.initiate(config)
EOF"

sleep 2

# setup replica set
docker exec mongo1 sh -c "$CMD"

sleep 20

vault write database/config/mongodb \
    plugin_name=mongodb-database-plugin \
    connection_url="mongodb://127.0.0.1:27017,127.0.0.1:27018,127.0.0.1:27019/admin?replicaSet=my-mongo-set" \
    allowed_roles="*"

vault write database/roles/adm \
    db_name=mongodb \
    creation_statements='{ "db": "admin", "roles": [{ "role": "readWrite" }, {"role": "read", "db": "foo"}] }' \
    default_ttl="1h" \
    max_ttl="24h"

vault read database/creds/adm

# exec into master
docker exec -it mongo1 sh 

# enter mongo shell
mongo

#step down master
rs.stepDown()

vault read database/creds/adm

Error reading database/creds/adm: Error making API request.

URL: GET http://127.0.0.1:8200/v1/database/creds/adm
Code: 500. Errors:

* 1 error occurred:
    * not master

Expected behavior

Should return credentials.

Environment:

• Vault Server Version 1.2.3.
• Vault CLI Version 1.2.3.
• MongoDB Version 4.2
• Server Operating System/Architecture: Ubuntu 16.04

Vault server configuration:

vault server -dev -dev-root-token-id='abc' -log-level=trace

Additional context

Great job @jefferai and @calvn on resolving previous issue, #2973. Just needs to be updated for Mongo 4+!

[jefferai]

Given that you know the exact code change necessary, any chance you would PR it? :-)

[zhecksum]

@jefferai sure thing! I'll submit a request right away!

[zhecksum]

Digged a little bit deeper into this, and it turns out that I was wrong.

MongoDB version 4.0 works as expected.

It is only version 4.2 that does not work.

The reason for this is in the MongoDB docs:

Starting in MongoDB 4.2, rs.stepDown() method no longer closes all client connections.

To support version 4.2, we will need to close the connection manually and reconnect to the new master. I am working on getting that done now.

[michelvocks]

Hi @zhecksum!

This issue has been resolved with https://github.com/hashicorp/vault/pull/8140. Closing this for now.

Cheers, Michel