Worker and Bot should tolerate Redis connection issues

dave-tucker commented 4 months ago

If Redis isn't ready when the bot is started, it currently hangs. If Redis isn't ready when the worker is started it retry loops forever but there is never a log once connection is established so it's hard to say for sure that it's working.

In both cases the code needs to be more tolerant of redis not being ready (yet) or a connection hiccup.

russellb commented 4 months ago

related to #44

dave-tucker commented 4 months ago

omg I had the same good idea twice 😆

nerdalert commented 4 months ago

Here is an example to reproduce some of this behavior.

Run without a redis instance running:

<runtime> run  -it --rm ghcr.io/instructlab/instructlab-bot/instructlab-gobot:main

and you will get a blast of messages with no backoff:

2024-04-30T03:32:31.369Z    ERROR   cmd/root.go:245 Redis Client Error: dial tcp: lookup redis on 172.31.0.2:53: no such host
github.com/instructlab/instructlab-bot/gobot/cmd.receiveResults
    /src/cmd/root.go:245
github.com/instructlab/instructlab-bot/gobot/cmd.run.func4
    /src/cmd/root.go:170
2024-04-30T03:32:31.369Z    ERROR   cmd/root.go:245 Redis Client Error: dial tcp: lookup redis on 172.31.0.2:53: no such host
github.com/instructlab/instructlab-bot/gobot/cmd.receiveResults
    /src/cmd/root.go:245
github.com/instructlab/instructlab-bot/gobot/cmd.run.func4
    /src/cmd/root.go:170
2024-04-30T03:32:31.369Z    ERROR   cmd/root.go:245 Redis Client Error: dial tcp: lookup redis on 172.31.0.2:53: no such host
github.com/instructlab/instructlab-bot/gobot/cmd.receiveResults

instructlab / instructlab-bot

Worker and Bot should tolerate Redis connection issues #244