search

rethinkdb / rethinkdb-rs

A native RethinkDB driver written in Rust
Apache License 2.0
210 stars 27 forks source link

Timeout error when performing queries in a timed loop #15

Closed opensourcegeek closed 3 years ago

opensourcegeek commented 7 years ago

I'm working with rethinkdb 2.3.6 both in prod and dev, I've got a loop which looks for data every minute. I can see that code work fine on dev but on prod it seems to end up with Driver(GetTimeout(GetTimeout(Some("driver")))) error.

I reduced the time to sleep in that loop to 20 seconds and it seems to work fine. Any ideas what parameter I should pass to connect method so that it does not timeout?

rushmorem commented 7 years ago

This error is coming from the r2d2 connection pool. It means your connection pool is running out of connections. Normally, you would configure r2d2 separately and pass in your pool configuration but we don't expose it publicly since I want to explore the possibility of using tk-pool first to integrate more tightly with Tokio.

I've got a loop which looks for data every minute.

It sounds to me like you are polling your RethinkDB servers. If so, did you consider using changefeeds instead?

EDIT: Also make sure you set canonical-address for each server and that it is set to an IP address that can be reached by your Rust app. This is because we pull this address from your server in realtime using changefeeds and then reconfigure your connection pool when any server comes up or goes down.

opensourcegeek commented 7 years ago

@rushmorem Thanks for the suggestion. I haven't considered changefeeds yet as I'm doing a running log of what's there in the database every minute and I'm not sure how it would work out when I need a timed log of all the data. I'll look around to see if I can use it for my case.

One thing I've noted is, if I change max_lifetime to say 86400 seconds then the connection is fine.

I'm not sure how the pooling of connections work especially why is there a max_lifetime for pool? Does it create a new pool each time I call connect. I'd like one pool with X number of connections live always, anytime the connection goes away then I'd expect the pool to be reinitialised. Is that how this pool works?

opensourcegeek commented 7 years ago

@rushmorem I've just created some extra mechanism to re-initialise the pool if I hit the error and it seems to be working fine for now.

rushmorem commented 7 years ago

One thing I've noted is, if I change max_lifetime to say 86400 seconds then the connection is fine.

I think the real problem here is that your app is failing to create new connections. Keeping connections around for that long may mask it but it has its own problems. This is why I actually reduced it from the default of 30 minutes.

I'm not sure how the pooling of connections work especially why is there a max_lifetime for pool?

See max_lifetime documentation for an explanation of how it works.

Does it create a new pool each time I call connect.

Yes. You shouldn't call connect multiple times in your application. You should call it only once from your main function and pass the connection object around. It's Copy so you don't even need to borrow it, which saves you from lifetime issues.

Are you calling connect multiple times? This can lead to the problem you are facing too. Your app may be spawning too many threads.

I'd like one pool with X number of connections live always, anytime the connection goes away then I'd expect the pool to be reinitialised. Is that how this pool works?

Yes. This is pretty much how it works.

Did you also see my previous comment about canonical-address? That's very important and pretty much a requirement for using this driver correctly.

opensourcegeek commented 7 years ago

@rushmorem thanks - I don't think canonical-address is set on both dev/prod. Do I have to set it even if there is only one single instance of rethinkdb running?

rushmorem commented 7 years ago

The thing is that if you do not set it, RethinkDB will set some defaults for you eg 127.0.0.1 and ::1 when you run rethinkdb without any arguments. You can see the ones set by your server by running:-

r.db("rethinkdb").table("server_status")

If your app can access these IP addresses, eg you are running it from the same machine as your RethinkDB server then that's fine. If not, that's a problem.

opensourcegeek commented 7 years ago

I've checked, the addresses are in canonical-addresses list. I'll play with the pool initialisation and see how it works - I've not checked out what r2d2 does yet.

rushmorem commented 7 years ago

Do you have the logger enabled? See https://github.com/rust-rethinkdb/reql/blob/1f502f79514f5aaab73489cc58dfde28df5d8c87/examples/map.rs#L26 for an example...

opensourcegeek commented 7 years ago

I haven't got slog setup, but I use log4rs at the moment. I see the following in the log messages which I believe is coming from here.

2017-08-21T17:02:58.237253543+01:00 ERROR r2d2 - driver 2017-08-21T17:02:59.714655640+01:00 ERROR r2d2 - driver 2017-08-21T17:02:59.714981716+01:00 ERROR r2d2 - driver 2017-08-21T17:02:59.736931669+01:00 ERROR r2d2 - driver 2017-08-21T17:02:59.737289472+01:00 ERROR r2d2 - driver 2017-08-21T17:02:59.737848165+01:00 ERROR r2d2 - driver 2017-08-21T17:03:01.215194532+01:00 ERROR r2d2 - driver 2017-08-21T17:03:01.215620389+01:00 ERROR r2d2 - driver 2017-08-21T17:03:01.237664926+01:00 ERROR r2d2 - driver 2017-08-21T17:03:01.237961755+01:00 ERROR r2d2 - driver 2017-08-21T17:03:01.238307710+01:00 ERROR r2d2 - driver 2017-08-21T17:03:02.715912986+01:00 ERROR r2d2 - driver 2017-08-21T17:03:02.716181132+01:00 ERROR r2d2 - driver 2017-08-21T17:03:02.738181798+01:00 ERROR r2d2 - driver 2017-08-21T17:03:02.738659475+01:00 ERROR r2d2 - driver 2017-08-21T17:03:02.738806473+01:00 ERROR r2d2 - driver 2017-08-21T17:03:04.216687655+01:00 ERROR r2d2 - driver 2017-08-21T17:03:04.216932838+01:00 ERROR r2d2 - driver 2017-08-21T17:03:04.238728066+01:00 ERROR r2d2 - driver 2017-08-21T17:03:04.239082425+01:00 ERROR r2d2 - driver 2017-08-21T17:03:04.239626897+01:00 ERROR r2d2 - driver 2017-08-21T17:03:05.717218839+01:00 ERROR r2d2 - driver 2017-08-21T17:03:05.717534337+01:00 ERROR r2d2 - driver

I'm not sure if this indicates any issues, I'll probably have to setup slog to find it I suppose.

rushmorem commented 6 years ago

I'll probably have to setup slog to find it I suppose

Yes. slog will probably help here since it's the one this library uses for logging. Alternatively, If you can come up with minimal code we can use to reproduce this, that will be great.

rushmorem commented 3 years ago

Closing as the latest versions no longer use r2d2 for connection pooling.