Server stops working if connection to MongoDB is lost

[invakid404]

Describe the bug

If the your_spotify server loses connection to MongoDB for whatever reason, it doesn't crash, but it also never attempts to reconnect and just stops working until I restart it manually. The relevant error:

[info]  Starting loop for 1 users
[info]  Refreshing songs for inva the kid
[info]  No missing tracks, passing...
[error]  MongoServerSelectionError: connect ECONNREFUSED 10.43.252.21:27017
    at Timeout._onTimeout (/app/node_modules/mongodb/lib/sdam/topology.js:278:38)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7) {
  reason: TopologyDescription {
    type: 'Single',
    servers: Map(1) {
      'mongodb.core.svc.cluster.local:27017' => [ServerDescription]
    },
    stale: false,
    compatible: true,
    heartbeatFrequencyMS: 10000,
    localThresholdMS: 15,
    setName: null,
    maxElectionId: null,
    maxSetVersion: null,
    commonWireVersion: 0,
    logicalSessionTimeoutMinutes: null
  },
  code: undefined,
  [Symbol(errorLabels)]: Set(0) {}
}

Expected behavior

The your_spotify server should either handle the connection error and attempt to reconnect the next time, or even just crash and let whatever orchestrator is running it restart it.

Additional context

I'm running your_spotify and mongodb in Kubernetes. I don't think my exact setup is too interesting, but it can be found here.

Screenshots

N/A

Shared link

N/A

[Yooooomi]

Hello! It looks like you're getting a connection refused error. It does seem like it's trying to connect once the server has been started for a while but it can't. I feel like restarting the server would not change anything here. The current behavior seems to be that it tries to reconnect id say. You should maybe check that your mongo does not go down or change ip.

[invakid404]

It definitely doesn't try to reconnect, as there have been many occasions where I updated my Mongo (which has downtime) and then checked your_spotify a few hours later only to see that it just stopped tracking my listening history when Mongo went down. This has happened to me every single time I've had to bring my Mongo offline for any reason, I've only reported this as an issue now that I've determined it happens consistently.

Here's a video of me reproducing it by manually bringing Mongo down (waiting obviously not included for brevity):

https://github.com/Yooooomi/your_spotify/assets/38114607/d5fb62b4-e7c1-4572-a06d-8ba8bd40c9fa

When Mongo initially goes down, the server doesn't error out immediately, but it dies on the next iteration, after which it does absolutely nothing, even if Mongo goes back up.

I clicked around the codebase to see if there was any reconnection logic, and I couldn't find any. I'm not too sure if Mongoose tries to do anything about the connection error, but by the looks of it, nothing happens after the error.

[invakid404]

Now that I took a closer look at the code, it makes a lot of sense why the behavior is what it is:

The error gets thrown from within dbLoop, and since there's no try-catch in the loop itself, the connection error makes its way to the closest catch, which is here, after which the loop stops and there's nothing to restart it.

Mongoose itself does try to reconnect on each operation, but since the loop itself is dead, the server doesn't do anything from then on. The application doesn't close since the error is caught and logged, after which it's just the HTTP server keeping it alive.

[invakid404]

@Yooooomi with the information above in consideration, I think the .catch() for dbLoop() should either rethrow the error to crash the application or call dbLoop again after some time. Which one makes more sense to you? I can open a PR with the preferred solution once we settle on one.

[fr3fou]

Any updates on this? My your_spotify also crashes when updating my MongoDB :D

[Yooooomi]

I feel like this issue is more of a devops one, since I feel like the backend should not try to investigate when mongo is down. It would add a bunch of logic that can just be fixed by restarting services when one is down.

[invakid404]

@Yooooomi I'm sorry, but did you even read what I wrote in the issue? The fix is a literal one line change, it doesn't involve "a bunch of logic", Mongoose already does everything for you. And even if you refuse for whatever reason to let Mongoose handle auto reconnects for you, please just let the server crash if Mongo goes down. The current behavior is not adequate at all. The server just stops working, no error, no log, nothing. If you don't want to bother implementing this, at least let me know which of the two solutions I proposed you prefer, so that I can do it for you.

[Yooooomi]

I expect a bit of respect. Come back when you are just a little calmer