search

superfly / fly

Deploy app servers close to your users. Package your app as a Docker image, and launch it in 17 cities with one simple CLI.
https://fly.io
985 stars 47 forks source link

Kong API Gateway on Fly #277

Open karllhughes opened 4 years ago

karllhughes commented 4 years ago

In this post, I'll show users how they can deploy a Kong API gateway to Fly.io. The API Gateway will sit in front of a public API (probably something from this list unless you have another suggestion). I'll explain what Kong Gateway is and why you might want to run it on the edge to improve performance as well as some of the limitations you might encounter.

Outline

Open questions

I've used Kong locally, but haven't deployed it to an edge platform like Fly, so I'm guessing I might run into other snags, but I only have one question initially:

mrkurt commented 4 years ago

I think using Heroku Postgres is best for now. I think Kong is pretty good about not querying the database on every request, and if you use Redis for caching it should be able to avoid it entirely. Worth a test though, once you get the app running you can try it in different regions to see what the effect is.

karllhughes commented 4 years ago

Yeah, I'll definitely have to test this out.

As for caching, it looks like version 0.33 of the proxy caching plugin included Redis configuration options, but there's no mention of Redis in the most recent version.

I'll keep googling around and see what I can figure out, but let me know if you have any insight into that.

mrkurt commented 4 years ago

Oh that's curious.

I think caching is an easy example but maybe not the most appropriate for an API gateway. Auth / rate limiting / maybe logging, or even geo load balancing/failover would all be reasonable with Kong if the caching is a non-starter.

karllhughes commented 4 years ago

Sounds good.

I'll start digging in this week. Thanks!

karllhughes commented 4 years ago

I've got the Kong database set up and am trying to run an instance of the gateway, but I've got two questions:

1. Where should I write to fly.io logs?

When you run the Kong image, it needs to be able to write to the stdout/stderr logs. Normally you set these as environment variables, but I realized that I didn't know the location of the fly.io logs. I tried /dev/stdout and got an error: could not open error log file: open() "/usr/local/kong/logs/error.log" failed (13: Permission denied) 2020-07-12T16:17:12.161Z d18f397a ewr [info] 2020/07/12 16:17:12 [emerg] 553#0: open() "/dev/stderr" failed (13: Permission denied)

My workaround to get things running was to send the logs to /dev/null for now.

Is there a place on the filesystem I can write logs so they'll be picked up by Fly's logs? I couldn't find this in the logging docs.

2. Can fly run multiple ports? Yes

I figured this one out today. I ended up creating a fly.toml file where Fly terminates the SSL and passes HTTP requests back to Kong:

# Kong Gateway
[[services]]
  internal_port = 8000
  protocol = "tcp"

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

# Kong Admin
[[services]]
  internal_port = 8001
  protocol = "tcp"

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "10001"

So I've got a working MVP going. Let me know if you have any input on the logs. Thanks!

karllhughes commented 4 years ago

I have a first draft ready for this, but still not sure how I can pipe logs to Fly. @mrkurt @KittyBot is this possible?

I tried piping to /dev/stdout and /dev/stderr, but I'm getting a permission error. Right now I'm working around the issue to get Kong working by piping to /dev/null.

Once I get this, I'll update and submit the first draft. Thanks!

mrkurt commented 4 years ago

Ahhh missed your question.

Logs need to go to stdout and stderr, there aren't any file logs. Based on this question you might have to launch it as an nginx process to get stdout to work: https://discuss.konghq.com/t/log-to-stdout-stderr/429

karllhughes commented 4 years ago

No worries, that makes sense!

I assumed it was a Fly-specific issue, but it sounds like it's more of a Kong+Docker issue. I'm not sure why they don't mention that in their Docker setup docs.

Thanks, this should just be a minor update then.

karllhughes commented 4 years ago

Ok, I've narrowed this down a little better. Unfortunately, the link above is no longer correct.

Here's what I learned:

I'm going to try to use chmod to set the permissions so I don't have to use root within the Dockerfile, but I'm not 100% sure why this file permissions issue would happen on Fly, but not in my local setup. I'll make note of it in the article though.

karllhughes commented 4 years ago

I wonder if this might be related: https://github.com/moby/moby/issues/31243

mrkurt commented 4 years ago

That is probably related. We aren't actually running Docker, we just expand the root fs into a volume and run a VM with it. But it's almost certainly a permissions issue on /dev/stdout. This is pretty useful info: https://unix.stackexchange.com/questions/38538/bash-dev-stderr-permission-denied

You don't need to fix that, necessarily, but if you want to you could try writing a wrapper script that changes permissions on /dev/stdout then runs nginx as a non-root user.

karllhughes commented 4 years ago

Ok, I spent a couple hours trying to figure out the log issue and had no luck, so I just solved it by changing the container's user to root with a link to the Github issue from Docker.

Here's my first draft: https://github.com/karllhughes/fly-kong

I can either transfer the repo to Fly, or delete it and make a PR on this repo with the readme? I wasn't sure what the next part of the process is.

Thanks!

mrkurt commented 4 years ago

@karllhughes yeah go ahead and transfer it, @codepope or I will review and do notes by tomorrow.

karllhughes commented 4 years ago

@mrkurt Github says I don't have permission to transfer repos to the superfly organization.

Let me know once you've added the permission and I'll transfer it. Thanks!

mrkurt commented 4 years ago

Ah these go in the fly-examples org, can you try that one?

mrkurt commented 4 years ago

@karllhughes have a look at the published version and see if you want to make any changes before we push it out: https://fly.io/docs/app-guides/kong-api-gateway/

karllhughes commented 4 years ago

Looks great! Let me know when I've got the okay and I'll share it as well.

mrkurt commented 4 years ago

Yeah go for it! We're going to pace when we promote these ourselves, but feel free to share them whenever you'd like. You don't have to wait on us.

daemswibowo commented 2 years ago

Hi, I want to ask. How can I connect kong to postgres database on fly?

The result always say this when I do the migration stuff:

Error: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused

even I do proxying the database to my local computer on port 5432. Thank you

megafinz commented 2 years ago

@daemswibowo I was trying to setup Kong on Fly today and it turns out that Kong isn't compatible with Postgres 14 having default password encryption set to scram-sha-256 right now: https://github.com/Kong/kong/issues/8259

megafinz commented 2 years ago

By the way if you're having troubles with Kong not finding your .internal addresses due to DNS lookup failures (you're seeing errors in logs that say DNS resolution failed: dns client error: 101 empty record received), you need to make sure that Kong searches for AAAA records either by adjusting your kong.conf file and adding AAAA to dns_order (e.g. dns_order=LAST,SRV,A,AAAA,CNAME) or using KONG_DNS_ORDER environment variable to do the same. This happens because by default Kong doesn't include AAAA records when it queries DNS for records.

Took me whole day to troubleshoot that, hope this will save you some time.