MongoDB spontaneously drops the `nmdcedge` database (Jetstream2)

[mflynn-lanl]

Data in the Mongodb disappears after a day or so. Mongodb is running in its own container. It stores its data to a directory on a mounted volume. After some undetermined amount of time, but at least several hours, the data in the database is lost. This doesn't happen after destroying the container and re-starting it.

[eecavanna]

Thanks, @mflynn-lanl.

This is in reference to the instance running on the Jetstream2 platform.

[mflynn-lanl]

Added a mongod.conf file to the mongo container. The mongod.conf file specifies the mongo log file and data directory

[mflynn-lanl]

{"t":{"$date":"2024-07-03T02:37:47.981+00:00"},"s":"D2", "c":"QUERY",    "id":20926,   "ctx":"conn51","msg":"Only one plan is available","attr":{"query":"ns: nmdcedge.users query: {} sort: {} projection: {}","planSummary":"COLLSCAN"}}
{"t":{"$date":"2024-07-03T02:37:49.953+00:00"},"s":"I",  "c":"COMMAND",  "id":20338,   "ctx":"conn51","msg":"dropDatabase - dropping collection","attr":{"db":"nmdcedge","namespace":"nmdcedge.users"}}
{"t":{"$date":"2024-07-03T02:37:49.953+00:00"},"s":"I",  "c":"STORAGE",  "id":22206,   "ctx":"conn51","msg":"Deferring table drop for index","attr":{"index":"_id_","namespace":"nmdcedge.users","uuid":{"uuid":{"$uuid":"48d2f668-3c3a-4970-a847-8b8beb002972"}},"ident":"index-15--6314124107840558945","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-07-03T02:37:49.953+00:00"},"s":"I",  "c":"STORAGE",  "id":22206,   "ctx":"conn51","msg":"Deferring table drop for index","attr":{"index":"email_1","namespace":"nmdcedge.users","uuid":{"uuid":{"$uuid":"48d2f668-3c3a-4970-a847-8b8beb002972"}},"ident":"index-6--8112619477707477091","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-07-03T02:37:49.953+00:00"},"s":"I",  "c":"STORAGE",  "id":22206,   "ctx":"conn51","msg":"Deferring table drop for index","attr":{"index":"code_1","namespace":"nmdcedge.users","uuid":{"uuid":{"$uuid":"48d2f668-3c3a-4970-a847-8b8beb002972"}},"ident":"index-13--8112619477707477091","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-07-03T02:37:49.953+00:00"},"s":"I",  "c":"STORAGE",  "id":22214,   "ctx":"conn51","msg":"Deferring table drop for collection","attr":{"namespace":"nmdcedge.users","ident":"collection-14--6314124107840558945","commitTimestamp":{"$timestamp":{"t":0,"i":0}}}}
{"t":{"$date":"2024-07-03T03:06:06.133+00:00"},"s":"D2", "c":"QUERY",    "id":20921,   "ctx":"conn17","msg":"Collection does not exist. Using EOF plan","attr":{"namespace":"nmdcedge.users","canonicalQuery":"ns: nmdcedge.users query: { email: \"admin@nmdc-edge.org\" } sort: {} projection: {} batchSize: 1 limit: 1"}}
{"t":{"$date":"2024-07-03T03:12:37.120+00:00"},"s":"D2", "c":"QUERY",    "id":20921,   "ctx":"conn65","msg":"Collection does not exist. Using EOF plan","attr":{"namespace":"nmdcedge.users","canonicalQuery":"ns: nmdcedge.users query: {} sort: {} projection: {}"}}
{"t":{"$date":"2024-07-03T03:13:09.862+00:00"},"s":"I",  "c":"STORAGE",  "id":20320,   "ctx":"conn69","msg":"createCollection","attr":{"namespace":"nmdcedge.users","uuidDisposition":"generated","uuid":{"uuid":{"$uuid":"2efd19c4-42a6-4448-96ae-24fc5f13593c"}},"options":{}}}
{"t":{"$date":"2024-07-03T03:13:09.886+00:00"},"s":"I",  "c":"INDEX",    "id":20345,   "ctx":"conn69","msg":"Index build: done building","attr":{"buildUUID":null,"collectionUUID":{"uuid":{"$uuid":"2efd19c4-42a6-4448-96ae-24fc5f13593c"}},"namespace":"nmdcedge.users","index":"_id_","ident":"index-3--5931235663522279702","collectionIdent":"collection-2--5931235663522279702","commitTimestamp":null}}

[eecavanna]

Wow! Thanks for posting that (and keeping an eye on the logs).

The users collection was dropped (I copy/pasted this line from the previous comment—and then reformatted it):

{
  "t": {
    "$date": "2024-07-03T02:37:49.953+00:00"
  },
  "s": "I",
  "c": "COMMAND",
  "id": 20338,
  "ctx": "conn51",
  "msg": "dropDatabase - dropping collection",
  "attr": {
    "db": "nmdcedge",
    "namespace": "nmdcedge.users"
  }
}

Then, approximately 36 minutes later, the users collection was created (re-created). I assume this is when someone happened to create a new user account.

[mflynn-lanl]

Enabled the firewall to prevent remote connections that could be used to delete Mongodb data:

sudo ufw allow 22
sudo ufw allow 443
sudo ufw enable

[eecavanna]

I want to add that that firewall was enabled on the VM/host (not within a Docker container/guest).

[ssarrafan]

@eecavanna @mflynn-lanl is this waiting for review? Who needs to review it?

[eecavanna]

This is still in progress. Even after enabling the firewall, @mflynn-lanl has seen the database get dropped again. We are still looking into it.

[ssarrafan]

Moving to next sprint for further troubleshooting per Eric's last comment.

[eecavanna]

Moving to next sprint because I will be out Friday and won't get to this on Thursday.

[eecavanna]

Update: We have enabled authentication on the Mongo server. We updated the app accordingly in https://github.com/microbiomedata/nmdc-edge/releases/tag/v0.1.0-alpha.16. We will keep an eye out for future occurrences of this symptom.

[eecavanna]

@mflynn-lanl confirmed that the database has persisted for a couple weeks so far, with no signs of problems. Based on that, we agreed to close this issue.

As a reminder, GitHub's Dependabot is reporting multiple security issues in the code base: https://github.com/microbiomedata/nmdc-edge/issues/230