search

contribsys / faktory

Language-agnostic persistent background job server
https://contribsys.com/faktory/
Other
5.76k stars 229 forks source link

Run faktory with an external Redis deployment #337

Closed dustMason closed 3 years ago

dustMason commented 3 years ago

I know I'm far from the first to express this (this issue is a duplicate of https://github.com/contribsys/faktory/issues/200)

We would love to be able to run faktory against a separate redis deployment. We have been running sidekiq enterprise in production for years and have a great deal of trust in it. There is a project on our team to create a "paved road" replacement for sidekiq to support our golang services, and faktory is an excellent fit. It has been rejected for consideration because we are unable to operate our own backing redis cluster at this time. The reasoning that customers would blame faktory for redis-related performance / scaling issues would not apply to our case.

Please consider allowing users of Faktory Enterprise to run Faktory against an external Redis cluster 🙏

mperham commented 3 years ago

I'm working on this right now. I hope to have something people can test this month.

mperham commented 3 years ago

Is it sufficient to merely detect the presence of REDIS_URL in the ENV and use that? How does Elasticache make itself known to a given process?

dustMason commented 3 years ago

Elasticache just provides an endpoint URL. An env var would work fine for us 👍

mperham commented 3 years ago

If anyone wants to test Faktory Enterprise with remote Redis, email mike@contribsys.com. I will send you a raw Faktory Enterprise binary that you can test. You will need an Enterprise license to use it in production mode.

All you need to do is provide REDIS_URL:

REDIS_URL=redis://some.host:6379/0 faktory ...

mperham commented 3 years ago
Screen Shot 2021-02-22 at 11 13 16 AM
mperham commented 3 years ago

Low Redis round-trip latency is important for Faktory performance. My local Redis shows 100-200µs latency during testing. If your latency is above 5-10ms (i.e. 5000-10000µs), you will start to see performance issues.

mperham commented 3 years ago

This will be in 1.5.0, coming in a week or two.

mperham commented 3 years ago

https://github.com/contribsys/faktory/wiki/Ent-Remote-Redis