adjust / rmq

Message queue system written in Go and backed by Redis
MIT License
1.57k stars 206 forks source link

panic on redis timeouts #38

Closed MathieuNls closed 6 years ago

MathieuNls commented 6 years ago

Hi,

We get panics when redis timeouts. The redis is alive but under pressure. How can we raise the limit or gracefully retry ?

Attached stack

2018/07/13 15:06:06 rmq redis error is not nil write tcp 172.17.0.4:51830->someip:6379: i/o timeout
panic: rmq redis error is not nil write tcp 172.17.0.4:51830->someip:6379: i/o timeout

goroutine 40 [running]:
log.Panicf(0xa2fd38, 0x1d, 0xc4861f5e00, 0x1, 0x1)
  /usr/local/go/src/log/log.go:333 +0xda
/my/go/path/vendor/github.com/adjust/rmq.checkErr(0xa8c540, 0xc479161f40, 0x5a)
  /my/go/path/vendor/github.com/adjust/rmq/redis_wrapper.go:96 +0xcc
/my/go/path/vendor/github.com/adjust/rmq.RedisWrapper.LRem(0xc4200b27d0, 0xc4202a81e0, 0x5a, 0x1, 0xc4f95f0000, 0xafa3827, 0x28, 0x79bb15)
  /my/go/path/vendor/github.com/adjust/rmq/redis_wrapper.go:51 +0xc4
/my/go/path/vendor/github.com/adjust/rmq.(*wrapDelivery).Ack(0xc47b168e10, 0xa46ae8)
  /my/go/path/vendor/github.com/adjust/rmq/delivery.go:43 +0x60
/my/go/path/queue.(*dequeuer).Consume(0xc420190030, 0xa92600, 0xc47b168e10)
  /my/go/pathqueue/consumer.go:103 +0xf9
/my/go/path/vendor/github.com/adjust/rmq.(*redisQueue).consumerConsume(0xc4202e2000, 0xa8c220, 0xc420190030)
  /my/go/path/vendor/github.com/adjust/rmq/queue.go:326 +0x6c
created by /my/go/path/vendor/github.com/adjust/rmq.(*redisQueue).AddConsumer
  /my/go/path/vendor/github.com/adjust/rmq/queue.go:229 +0x8d
MathieuNls commented 6 years ago

Hi,

I've found the following while pocking around. It answers my question:

    return rmq.OpenConnectionWithRedisClient(
        name,
        redis.NewClient(&redis.Options{
            Network:      network,
            Addr:         ip,
            DB:           bdd,
            MaxRetries:   5,
            DialTimeout:  time.Second * 15,
            ReadTimeout:  time.Second * 10,
            WriteTimeout: time.Second * 10,
        }),
    )

It uses the Options struct of redis : https://godoc.org/github.com/go-redis/redis#Options