Loopback app won’t connect to MongoDB after first database connection failure – No retry

[rranjann]

The Loopback based Microservices won't connect to MongoDB after first database connection failures and it always fail with Internal Server Error. In kubernetes HA environment when all the Mater nodes are rebooted and if MongoDB is not fully functional the loopback microservices will fail to connect to MongoDB and as expected.

Later even if the MongoDB has recovered the database connection will fail with same status code 500. There is no retry logic for the above issues even following attribute has used in data source file.

  autoReconnect: true,
  reconnectTries: 100,
  reconnectInterval: 10000,
  connectionTimeout: 30000,
  keepAlive: true 

Unhandled error for request GET /identity/api/v1/account: Error: Timeout in connecting after 5000 ms
   at Timeout._onTimeout (/Users/workspace/<application-name>/node_modules/loopback/node_modules/loopback-datasource-juggler/lib/datasource.js:2461:10)

Description/Steps to reproduce

- Do NOT start the Mongo DB
- Start the Strongloop based Application
- Send first request and it fails to connnect to the DB
- Start the Mongo DB
- Requests will be always fail with status code `500` interval server error. The database connection is not working at all and it always fail with same status code.
- Workaround : Restart Application and then the connection served correctly. 

Expected result

- Do NOT start the Mongo DB
- Start the Strongloop based Application
- Send first request and it fails to connnect to the DB
- Start the Mongo DB
- The database connection requests should be served correctly

Additional information

- node -e 'console.log(process.platform, process.arch, process.versions.node)'
  # darwin x64 6.11.1
- npm ls --prod --depth 0 | grep loopback :

├── loopback@3.0.0
├── loopback-boot@2.26.1
├── loopback-component-explorer@4.2.0
├── loopback-connector-cloudant@1.2.5
├── loopback-connector-mongodb@1.17.0
├── loopback-datasource-juggler@2.39.2  

[raymondfeng]

The issue should have been fixed by https://github.com/strongloop/loopback-datasource-juggler/pull/1585.

[dhmlau]

@rranjann , could you please verify?

[dhmlau]

Closing this as resolved.
@rranjann , feel free to let us know if you still encounter this problem. Thanks!

[sk185050]

The issue should have been fixed by #1585.

Hello Ray, I am observe a case, where I assume would be the case. The DB we are using is oracle, I hope same is applied with all the connector ?

[sk185050]

If you allow me if you can help in with your input on a case, There are two process/pod are running on and each has its own pool used by oracle connector. What happened was in one instance the oracle is able to connect and process the requests but one the second pod/process I have noticed the NJS-040 error connection times out. As both the running on the same environment, so connectivity will not be an issue. I am seeking your help or input to know the reason could be connection pool exhausted? As I seen the number of connection/session on DB was less in number that was configured. When I re-start the pod/process every thing come back to normal. Some Qs that come in to my though process, for which I am seeking your help on, if you can guide and give your valuable input on.

If the connection pool is used (oracle connector )it should release the objects as well and exhaustion will have the less chances on?
After the process/pod is started and get the connection object in data source via connector ( oracle), and network issue happened and connectivity is lost between the process and db, does the connector recover that? As my observation, it seems it should what I have noticed in other process that was recovered, What could cause it in one process when it is running with the scope?

depencecies: "loopback": "3.23.0", "loopback-connector-oracle": "4.1.1", "oracledb": "2.3.0"