User/password not valid right after creation.

[heprotecbuthealsoattac]

When trying to use credentials against the cluster right after they are returned by vault I'm getting a following failure:

err  - "Database connection failed", {"name":"MongoError","message":"authentication fail","errors":[{"name":"my-shard-00-01-xxx.mongodb.net:12345","err":{"name":"MongoError","message":"Authentication failed.","ok":0,"errmsg":"Authentication failed.","code":18,"codeName":"AuthenticationFailed"}},{"name":"my-shard-00-00-xxx.mongodb.net:12345","err":{"name":"MongoError","message":"Authentication failed.","ok":0,"errmsg":"Authentication failed.","code":18,"codeName":"AuthenticationFailed"}},{"name":"my-shard-00-02-xxx.mongodb.net:12345","err":{"name":"MongoError","message":"Authentication failed.","ok":0,"errmsg":"Authentication failed.","code":18,"codeName":"AuthenticationFailed"}}],"stack":"MongoError: authentication fail\n    at Function [...]

However the same credentials work just right after waiting for a few seconds before connecting. So to sum up (not an actual code):

BROKEN:

const creds = await queryVaultAndGetUserAndAPass();
return await MongoClient.connect(creds);

WORKS:

const creds = await queryVaultAndGetUserAndAPass();
await sleep(15000);
return await MongoClient.connect(creds);

[themantissa]

This should be expected behavior. The database user is created at the project level and depending on the number of clusters in a project may take time to apply across them.

[heprotecbuthealsoattac]

It never happens with mLab, also it should be possible to set write concern so that the write returns only when the write was propagated to the majority of the nodes.

[themantissa]

Hopefully I can help shed some additional light here. This is different in that we aren’t creating the user directly on a cluster (by cluster I mean replica set or sharded cluster) but via the MongoDB Atlas API. Once requested via the API (in this case via Terraform Provider) the Database User is stored at the Project Level and provisioned into all the clusters currently in the Project and as new clusters are added. With mLab one is creating the user on the shell for a cluster - so it’s direct and one cluster at a time. In fact that’s why we are creating this new Vault plugin, there’s already a Vault plugin (aka Secrets Engine) when creating a MongoDB Database user on the shell but it does not work for the Atlas API.

[heprotecbuthealsoattac]

I understand the technical difficulties however from a user perspective this is problematic as well because the user is invalid when vault returns the credentials. How do we deal with that on the client side? Poll for the user existence? This complicates client code significantly.

[JnMik]

Same issue here. The containers boot, get credentials from vault, tries to connect and fails.

I think the current state of things is it makes the plugin unusable for production. Nobody wants applications that goes crazy and restart because of failing connections.

I believe someone could workaround this using a sidecar container to init the credentials and put some kind of delay before the actual app boots. But the issue would still exist for the lease renewal right ?

@themantissa Are you aware of any discussion in your team that would make this behavior go away and enforce the plugin usability ?

Or maybe for the moment we should stick to store the username / password directly in vault as Key: value ?

[themantissa]

@JnMik A few comments:

This Secrets Engine (plugin) is not verified yet by HashiCorp, that review is happening so this is considered in development as per the README.

We have an internal request to the Atlas team to get an endpoint that gives us the status of applying the new user to all clusters in the Atlas project, however that will take some time before we have it available. Then we'll need to work that into this plugin.

Finally, in the mean time if an ideal solution to the delay between user creation and success of that command and the full apply across all the clusters in the project can't be found outside of this plugin then I would suggest waiting till we have a solution to accommodate that.

[JnMik]

Cool, at least it's in the pipeline ! We'll wait for now until it progresses and we'll check back later :)

thanks for the quick reply