loopbackio / loopback-connector-mysql

Loopback Connector for MySQL
Other
125 stars 183 forks source link

Connection fails: Error: connect ETIMEDOUT #210

Closed s26 closed 7 years ago

s26 commented 8 years ago

Hello,

I am getting following error randomly when I run try to restart my app. Not able to figure out how can I fix it. Can someone please assist me to resolve this error?

Web server listening at: http://0.0.0.0:3000
Browse your REST API at http://0.0.0.0:3000/explorer
Connection fails: Error: connect ETIMEDOUT
It will be retried for the next request.
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: connect ETIMEDOUT
    at PoolConnection.Connection._handleConnectTimeout (/Users/xxx/api/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Connection.js:425:13)
    at Socket.g (events.js:291:16)
    at emitNone (events.js:86:13)
    at Socket.emit (events.js:185:7)
    at Socket._onTimeout (net.js:334:8)
    at tryOnTimeout (timers.js:232:11)
    at Timer.listOnTimeout (timers.js:202:5)
    --------------------
    at Protocol._enqueue (/Users/xxx/api/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:141:48)
    at Protocol.handshake (/Users/xxx/api/node_modules/loopback-connector-mysql/node_modules/mysql/lib/protocol/Protocol.js:52:41)
    at PoolConnection.connect (/Users/xxx/api/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Connection.js:136:18)
    at Pool.getConnection (/Users/xxx/api/node_modules/loopback-connector-mysql/node_modules/mysql/lib/Pool.js:48:16)
    at MySQL.connect (/Users/xxx/api/node_modules/loopback-connector-mysql/lib/mysql.js:86:17)
    at Object.initializeDataSource [as initialize] (/Users/xxx/api/node_modules/loopback-connector-mysql/lib/mysql.js:40:28)
    at DataSource.setup (/Users/xxx/api/node_modules/loopback-datasource-juggler/lib/datasource.js:339:19)
    at new DataSource (/Users/xxx/api/node_modules/loopback-datasource-juggler/lib/datasource.js:117:8)
    at Registry.createDataSource (/Users/xxx/api/node_modules/loopback/lib/registry.js:355:12)
    at dataSourcesFromConfig (/Users/xxx/api/node_modules/loopback/lib/application.js:440:19)
superkhau commented 7 years ago

Is your MySQL server running?

s26 commented 7 years ago

Yes. My MySql server is running. I am able to start the app successfully, but not all the time. I get this error randomly. 8 out of 10 times I get above error while restarting the app. It is annoying, Sometimes I have to try for 10-15 times to start the app. Everytime I am getting above error.

superkhau commented 7 years ago

@s26 That is the worse case because it should be 100% of the time. It'll make it much harder to reproduce if it's intermittent like that. Can you try scaffolding a new basic LoopBack application to see if it occurs there? I'm thinking it's something else.

ernie58 commented 7 years ago

I have this to lately.

Mostly I have it when I'm starting my application in the morning the first time (when a lot of other processes are starting too), or when I try to start it with the debugger (which is kind a heavy) .

I think it occurs when my vagrant is under heavy load. Is there any way to set a timeout option in my loopback datasource config to see if increasing the timeout helps?

bbito commented 7 years ago

@ernie58 I'm not sure that it solves your problem, but a connectTimeout connection option value can be passed to the underlying mysqljs/mysql module through your Loopback \server\datasources.json configuration file.

connectTimeout: The milliseconds before a timeout occurs during the initial connection to the MySQL server. (Default: 10000)

https://github.com/mysqljs/mysql#connection-options

You would add the option to datasources.json something like this for a 30 second timeout:

{
  "my_mysql_db": {
    "port": 3306,
    "url": "localhost",
    "database": "my_mysql_db",
    "password": "myPassword",
    "name": "my_mysql_db",
    "user": "my_node_mysql_user",
    "connector": "mysql",
    "connectTimeout": 30000
  }
}
superkhau commented 7 years ago

Would you like to submit a PR? You can see the options we pass in at https://github.com/strongloop/loopback-workspace/blob/master/available-connectors.json#L280

bbito commented 7 years ago

@superkhau If the above comment is addressed to me, I'm confused...

  1. I wasn't suggesting a change to the Loopback connector
  2. The datasources.json modification I suggested works as is - that connectTimeout value will pass through to mysqljs/mysql

Connection options beyond those listed in the JSON you linked to _will_ pass through to mysql. I have been using dateStrings for some time and just tested connectTimeout.

superkhau commented 7 years ago

@bbito I thought we wanted to set a default timeout permanently to make it more explicit (ie. the underlying driver's 10000 you mentioned). Anyways, it's all good and thanks for helping out. You are right though, any options you set in datasources.json will be passed down to the driver -- that should solve @s26's issue.

@s26 Can you confirm whether @bbito's solution works for you? Can I close this?

probil commented 7 years ago

I have the same issue. It happens much often if I try to debug app with WebStorm. Without debugging it happens in ~5%, with debugging it happends in ~70% for me.

bbito commented 7 years ago

@probil does extending the timeout reduce the appearance of the errors?

probil commented 7 years ago

@bbito Yeah, it seems like this works. But sometimes I got another error:

Connection fails: Error: Handshake inactivity timeout
It will be retried for the next request.
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: Handshake inactivity timeout
    at Handshake.<anonymous> (/media/multimedia/Code/node-js/test/node_modules/mysql/lib/protocol/Protocol.js:160:17)
    at emitNone (events.js:86:13)
    at Handshake.emit (events.js:185:7)
    at Handshake._onTimeout (/media/multimedia/Code/node-js/test/node_modules/mysql/lib/protocol/sequences/Sequence.js:126:8)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)
    --------------------
    at Protocol._enqueue (/media/multimedia/Code/node-js/test/node_modules/mysql/lib/protocol/Protocol.js:141:48)
    at Protocol.handshake (/media/multimedia/Code/node-js/test/node_modules/mysql/lib/protocol/Protocol.js:52:41)
    at PoolConnection.connect (/media/multimedia/Code/node-js/test/node_modules/mysql/lib/Connection.js:136:18)
    at Pool.getConnection (/media/multimedia/Code/node-js/test/node_modules/mysql/lib/Pool.js:48:16)
    at MySQL.connect (/media/multimedia/Code/node-js/test/node_modules/loopback-connector-mysql/lib/mysql.js:89:17)
    at Object.initializeDataSource [as initialize] (/media/multimedia/Code/node-js/test/node_modules/loopback-connector-mysql/lib/mysql.js:43:28)
    at DataSource.setup (/media/multimedia/Code/node-js/test/node_modules/loopback-datasource-juggler/lib/datasource.js:339:19)
    at new DataSource (/media/multimedia/Code/node-js/test/node_modules/loopback-datasource-juggler/lib/datasource.js:117:8)
    at Registry.createDataSource (/media/multimedia/Code/node-js/test/node_modules/loopback/lib/registry.js:355:12)
    at dataSourcesFromConfig (/media/multimedia/Code/node-js/test/node_modules/loopback/lib/application.js:440:19)
    at EventEmitter.app.dataSource (/media/multimedia/Code/node-js/test/node_modules/loopback/lib/application.js:235:14)
    at /media/multimedia/Code/node-js/test/node_modules/loopback-boot/lib/executor.js:190:9
    at /media/multimedia/Code/node-js/test/node_modules/loopback-boot/lib/executor.js:281:5
    at Array.forEach (native)
    at forEachKeyedObject (/media/multimedia/Code/node-js/testt/node_modules/loopback-boot/lib/executor.js:280:20)
    at setupDataSources (/media/multimedia/Code/node-js/test/node_modules/loopback-boot/lib/executor.js:180:3)
bbito commented 7 years ago

Hi @probil - thanks, that is useful information. I assume that your CPU is stressed since enabling debugging is having a significant effect. Is this a dev or production environment? Perhaps @superkhau is familiar with the Handshake inactivity error you reach with the increased connectTimeout.

probil commented 7 years ago

It's dev environment. Sometimes I got the error even without debugging enabled

probil commented 7 years ago

I thinks it should be possible to pass acquireTimeout param

probil commented 7 years ago

@bbito It seems like no issue anymore with:

{
 "mySQL":{
    //...
    "connectTimeout": 20000,
    "acquireTimeout": 20000
}

in datasources.json

s26 commented 7 years ago

@superkhau I have uninstasll strongloop and reinstall it. And cloned the project again and it is working fine. Thanks

ernie58 commented 7 years ago

FWIW I work with webstorm debugger to. I Did have the connect timeout and handshake activity timeout. Setting the 2 variables in the config helped me out

PaddyMann commented 7 years ago

Had the same issue running Loopback + PM2 + MariaDB on Docker. Setting larger timeouts has fixed - thanks :)

jcolemorrison commented 7 years ago

Just to note if anyone else has problems - I had to put this timeout directly on datasources.json. Putting it on one of the env namespaced ones doesn't seem to work i.e datasources.production.js

CoderRathore commented 7 years ago

Hello,

I am getting following error randomly when I run try to restart my web. Not able to figure out how can I fix it. Can someone please assist me to resolve this error?

C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\app.js:140 if (err) throw err; ^

Error: connect ETIMEDOUT at Connection._handleConnectTimeout (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\Connection.js:425:13) at Socket.g (events.js:291:16) at emitNone (events.js:86:13) at Socket.emit (events.js:185:7) at Socket._onTimeout (net.js:339:8) at ontimeout (timers.js:365:14) at tryOnTimeout (timers.js:237:5) at Timer.listOnTimeout (timers.js:207:5)

at Protocol._enqueue (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\protocol\Protocol.js:141:48)
at Protocol.handshake (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\protocol\Protocol.js:52:41)
at Connection.connect (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\Connection.js:136:18)
at getOpenTarget (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\app.js:112:5)
at Query._callback (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\app.js:89:13)
at Query.Sequence.end (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\protocol\sequences\Sequence.js:85:24)
at Query._handleFinalResultPacket (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\protocol\sequences\Query.js:144:8)
at Query.EofPacket (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\protocol\sequences\Query.js:128:8)
at Protocol._parsePacket (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\protocol\Protocol.js:280:23)
at Parser.write (C:\Users\User\Desktop\20-7-2017\webservice_cron\webservice_cron\webservice\node_modules\mysql\lib\protocol\Parser.js:74:12)
ivanvaccari commented 4 years ago

Still have this issue after YEARS.

I have literally tried everything. It just won't connect randomly where all the other app have no problems.

damianmr commented 4 years ago

Thank you so much @probil, I got this problem today as well.

Loopback just started failing when trying to connect to the database. It was completely random. Increasing the timeouts, as suggested, fixed the problem.

prateek-raj commented 4 years ago

I am getting the same issue. The solution doesn't work for me.

strideynet commented 3 years ago

Our solution was to remove the initial database connection from the loopback boot by enabling lazyConnect in the datasource config, and then to separately call the datasource connection after the boot is finished. We came to a similar conclusion that CPU stress during the loopback boot causes the db connection to fail.

bmaupin commented 3 years ago

We used the connectTimeout and acquireTimeout workaround for a while but started getting this error again. Increasing them further didn't work.

Since we're using Kubernetes, we were able to increase the CPU limit (spec.containers[].resources.limits.cpu) and that fixed the issue.

But I'm very interested in the solution proposed by @strideynet . Do you have any code snippets you can share?

Thanks!

bmaupin commented 3 years ago

Interesting, it looks like lazyConnect is an undocumented feature of the MySQL connector: https://github.com/strongloop/loopback-connector-mysql/blob/7ce66b5b819669114fc828c2dbef5b16c4ff8fd8/lib/mysql.js#L39

I imagine it works similarly as described in the MongoDB documentation:

When set to true, the database instance will not be attached to the datasource and the connection is deferred. It will try to establish the connection automatically once users hit the endpoint. If the MongoDB server is offline, the app will start, however, the endpoints will not work.

(https://loopback.io/doc/en/lb3/MongoDB-connector.html#additional-properties)

This issue seems to have some steps on how to use it:

  1. Specify "lazyConnect": true in datasources.json for the given datasource.
  2. Programmatically connect to this datasource after boot using:
    app.datasources.MyDatasource.connect(...);

(https://github.com/strongloop/loopback-datasource-juggler/issues/1592)

sivaiagpolagani commented 3 years ago

I am getting this error while trying to connect with the MySQL database, It was already hosted on the server. image