self host: Following the Self Host guide for Heroku errors

[judahrand]

I follow the guide for creating a self hosted Kodiak exactly afaik and am receiving the following errors:

https://gist.github.com/judahrand/85b26f57a8a93b71845d3f6a95b20fc6

It seems like Kodiak is unable to connect to Redis add-on?

[chdsbd]

Do you have the version SHA of Kodiak you were using?

It looks like Kodiak isn't able to connect to the Redis instance. Can you verify that the Redis instance is accessible?

[judahrand]

I can confirm that I am able to connect to the Redis instance from my dev machine using the credentials given to me by Heroku.

This is the return on CLIENT LIST on the Redis instance:

id=64 addr=18.203.137.81:39056 laddr=10.0.147.72:17009 fd=12 name= age=153 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=65 addr=18.203.137.81:39058 laddr=10.0.147.72:17009 fd=13 name= age=153 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=bzpopmin user=default redir=-1
id=66 addr=18.203.137.81:39060 laddr=10.0.147.72:17009 fd=14 name= age=153 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=67 addr=18.203.137.81:39062 laddr=10.0.147.72:17009 fd=15 name= age=153 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=68 addr=18.203.137.81:39064 laddr=10.0.147.72:17009 fd=16 name= age=153 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=69 addr=18.203.137.81:39066 laddr=10.0.147.72:17009 fd=17 name= age=153 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=76 addr=18.203.137.81:58376 laddr=10.0.147.72:17009 fd=23 name= age=91 idle=89 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=77 addr=18.203.137.81:58378 laddr=10.0.147.72:17009 fd=24 name= age=91 idle=91 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1
id=78 addr=18.203.137.81:58380 laddr=10.0.147.72:17009 fd=25 name= age=91 idle=86 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=79 addr=18.203.137.81:58382 laddr=10.0.147.72:17009 fd=26 name= age=91 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=80 addr=18.203.137.81:58384 laddr=10.0.147.72:17009 fd=27 name= age=91 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=81 addr=18.203.137.81:58386 laddr=10.0.147.72:17009 fd=28 name= age=91 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=hgetall user=default redir=-1
id=82 addr=18.203.137.81:58390 laddr=10.0.147.72:17009 fd=29 name= age=91 idle=91 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1
id=83 addr=18.203.137.81:58388 laddr=10.0.147.72:17009 fd=30 name= age=91 idle=91 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1
id=84 addr=18.203.137.81:58392 laddr=10.0.147.72:17009 fd=31 name= age=91 idle=91 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=auth user=default redir=-1
id=70 addr=18.203.137.81:39068 laddr=10.0.147.72:17009 fd=18 name= age=153 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
id=71 addr=18.203.137.81:39070 laddr=10.0.147.72:17009 fd=19 name= age=153 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
id=73 addr=18.203.137.81:39642 laddr=10.0.147.72:17009 fd=21 name= age=151 idle=24 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=25 obl=0 oll=0 omem=0 tot-mem=20537 events=r cmd=bzpopmin user=default redir=-1
id=74 addr=82.32.201.156:64261 laddr=10.0.147.72:17009 fd=22 name= age=134 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=26 qbuf-free=40928 argv-mem=10 obl=0 oll=0 omem=0 tot-mem=61466 events=r cmd=client user=default redir=-1
id=72 addr=18.203.137.81:39072 laddr=10.0.147.72:17009 fd=20 name= age=153 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=bzpopmin user=default redir=-1

What's strange is that I've set REDIS_POOL_SIZE to 10. So there should only be 11 connections (one from me). But it looks like the pool is abandoning connections when they fail but not closing them.

Any suggestions?

[judahrand]

Oh and I forgot the commit SHA. I've tried 77d0a2a0fb5ff70d2778406661f03f29e4279359 and 5ae8ec7a32f22a99c1c547a57b2f87d536a419f1

[judahrand]

I think I might have solved this. I think it was down to there actually being 2 Redis connection pools defined in the code. One of which default to poolsize=50 and one poolsize=20. This obviously results in 70 connections which is too many for any free tier Redis in Heroku.

Weirdly, these two pools are defined with two different env vars: REDIS_POOL_SIZE and USAGE_REPORTING_POOL_SIZE. If I set these two env vars to sum to less than my connection limit then the errors seem to go away. Though I'll need a bit more testing to confirm.

https://github.com/chdsbd/kodiak/blob/77d0a2a0fb5ff70d2778406661f03f29e4279359/bot/kodiak/queue.py#L237 https://github.com/chdsbd/kodiak/blob/b85994e812997b7ea59d74e231b6516b946c3139/bot/kodiak/queue.py#L476

What's the reason for having 2 different connection pools @chdsbd? Is this something that should be consolidated? Or just documented better? At the moment the Self-Hosting instructions will always fail afaik.

[judahrand]

I think the issue here is the use of the get_redis function in get_subscriptions. Though I could be wrong. This means that both connection pools are needed even if USAGE_REPORTING is False. Which is incredibly confusing.

https://github.com/chdsbd/kodiak/blob/77d0a2a0fb5ff70d2778406661f03f29e4279359/bot/kodiak/queries/__init__.py#L1204

[judahrand]

And looking into the commit history a bit more it looks like this subscription checking into Redis should only be happening if SUBSCRIPTIONS_ENABLED is True. I'll open a PR for that change.

[judahrand]

718

[judahrand]

Should have labelled with bug 😛

[chdsbd]

Thanks for looking into this. I'm don't think there's a good reason for using two separate connection pools