Running OpenResty on Fly

[karllhughes]

Initially, we talked about running OpenResty to serve and cache static files, but it seems like you can do that with vanilla NGinx, so unless I misunderstood, that doesn't seem like that exciting of an example.

What about running OpenResty to serve a Redis backend on Fly? Seems like a unique and interesting case and it uses the lua-resty-redis driver, so it's more unique to OpenResty.

I'm familiar with NGinx, but not as experienced with OpenResty and Lua, so I welcome any direction/feedback on this one. It seems like there are a lot of directions you could go with this based on the OpenResty docs.

[mrkurt]

I'm a huge fan of OpenResty examples, I tend to conflate openresty + nginx (just because openresty is the best way to do lua in nginx). I think it makes total sense to hook up openresty, even if we just call it an nginx example.

One interesting use case for this would be per region rate limiting with Redis: https://github.com/TiVo/lua-resty-rate-limit

[mrkurt]

Oh, auth is another good thing to show with OpenResty: https://github.com/zmartzone/lua-resty-openidc

[karllhughes]

Here's the outline for what I'm thinking as a good first article:

• Introduction
  • What is OpenResty? - An overview of the tool, and its relationship to NGINX/Lua.
  • Why use OpenResty? - Some common reasons people might use OpenResty (microservices, caching, scripting on top of Nginx, etc.)
  • Why OpenResty on Fly? - Specific reasons that deploying OpenResty to edge hosting like Fly might be advantageous: speed, failover
• How to Run OpenResty on Fly
  • Setting up a new Fly app
  • Configuring the Dockerfile
  • Connecting to a backing service
  • Adding Per-Region Rate Limiting with OpenResty: https://github.com/TiVo/lua-resty-rate-limit
  • Adding caching with Redis: https://github.com/openresty/lua-resty-redis
• Conclusion

---

I think the auth example is good, but worth a separate article because I'd need to explain some of the other prerequisites in more detail. If that sounds good, I could create a new issue to write a follow-up article on using OpenResty to do auth?

[mrkurt]

Yeah I like it. Auth makes total sense standalone.

[karllhughes]

I've got the basic app going, but can't get the Nginx logs to show up in Fly.

OpenResty is supposed to be symlinking Nginx access and error logs to stdout and stderr (and this works locally with Docker), but I can't get it working on Fly.

Is there an example of this somewhere? Any ideas?

[karllhughes]

Nevermind, figured this out.

[mrkurt]

Oh I'm interested in what you found, we've had a few people with issues in different frameworks writing to /dev/stdout so I think this is harder than it should be.

[karllhughes]

Well, in my case this time, I needed to add this to my server block in my Nginx conf:

    access_log  /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

It's weird that I didn't need that for my local installation - it somehow knew to log correctly anyway.

Nope, that wasn't the issue. I think my problem was that my updates to the Nginx Conf file weren't taking place because when I did a flyctl deploy nothing had changed in the Dockerfile.

[karllhughes]

One more weird one:

Redis on Fly isn't working for me.

Every time I authenticate in Lua, it returns the error, closed.

When I connect from my local terminal using redis-client, I can get in, but can't set or retrieve any keys. Every time I try it says, Error: Server closed the connection.

Any ideas about that?

[mrkurt]

Can you hit https://debug.fly.dev and paste the output here? I want to see which region you're connecting to.

[karllhughes]

=== Headers ===
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Sec-Fetch-Site: cross-site
X-Forwarded-Ssl: on
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36
Sec-Fetch-Mode: navigate
Accept-Encoding: gzip, deflate, br
X-Forwarded-Port: 443
Fly-Region: ord
Sec-Fetch-Dest: document
Referer: https://github.com/superfly/fly/issues/278
Fly-Client-Ip: 2601:241:8580:33a0:c16e:ba37:2133:945f
Fly-Forwarded-Proto: https
Fly-Forwarded-Ssl: on
Fly-Forwarded-Port: 443
Accept-Language: en-US,en;q=0.9,pl;q=0.8
Sec-Fetch-User: ?1
X-Forwarded-For: 2601:241:8580:33a0:c16e:ba37:2133:945f, 2a09:8280:1:763f:8bdd:34d1:c624:78cd
X-Forwarded-Proto: https
Via: 2 fly.io

=== ENV ===
FLY_ALLOC_ID=2d17c02b-5e09-826f-6207-f361009e0ae0
FLY_PUBLIC_IP=2604:1380:4060:4fe:0:2d17:c02b:1
FLY_REGION=yyz
HOME=/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
TERM=linux
WS=this
is
a
test
cgroup_enable=memory
pci=off

2020-07-22 21:54:17.180140755 +0000 UTC m=+69290.759627029

[karllhughes]

I played around with this more this morning - tried destroying my app and redeploying - and my Redis is still not working. Just immediately closes the connection whether I use redis-client locally or try to connect in my Lua app.

Is there anything else I can do to debug this or try to narrow down the issue? Can I force my app to use a different region or something? I could also send you my connection string so you could try it @mrkurt. Let me know what you think.

[michaeldwan]

@karllhughes could you share the connection string you're using with support@fly.io? When you say redis-client isn't working locally do you mean redis-cli -u URL is failing to connect?

[karllhughes]

I'm starting to understand my issue a little more clearly, and it appears to actually be that Lua can't resolve the DNS record. Probably this: https://github.com/openresty/lua-resty-redis/issues/159

I'm going to set a resolver, which should fix it. I'll let you know if I continue to have issues though, thanks!

[karllhughes]

@mrkurt @KittyBot I just finished editing this article and transferred it to the Fly-Examples org: https://github.com/fly-examples/fly-openresty

I started down the path of caching the API's response with Redis, but that ended up making the example very complicated because I couldn't use Nginx's default reverse proxy. So, I opted to use a simpler example of connecting to Redis to check some cached API keys for simple authentication.

I think there are a ton of good deep-dives we could do with OpenResty, but I tried to keep this tutorial approachable while still showcasing some of the power of OpenResty on Fly.

Let me know if you have any feedback!

[wwi-marvinmatos]

fyi for those who want to set up an example at a k8s provider https://github.com/openresty/lua-resty-redis/issues/159#issuecomment-776054684