Caddy v2?

[polarathene]

Is this project going to remain targeting v1 alone, or is there plans to support v2 (currently in Beta, aimed to release early 2020) at a later date?

[lucaslorentz]

To be honest, I don't know, and I didn't spend any time to play with caddy v2 yet.

I wonder if there is still value in improving this project, given that:

• Docker Swarm is losing more and more space to Kubernetes.
• Caddy team is already implementing an ingress controller for Kubernetes.
• Docker Swarm now supports using Kubernetes as the orchestrator.
• Traefik is a very good alternative to this project, it can be used with Docker Swarm orchestrator as well.

Could you please describe your use-case? I presume that you use this project already.

• Do you use it on production?
• Do you use it with Docker Swarm or just raw docker containers?
• Do you plan to migrate to Kubernetes?
• Did you consider using Traefik? If yes, why do you prefer Caddy for load balancing your containers?

[unixfox]

I do agree with you that Docker Swarm is loosing some popularity and lot of people are migrating to Kubernetes. Including me, I plan to switch to Kubernetes when Caddy will have a proper ingress controller for Kubernetes.

I think it would be kind a waste of time to support the second version of Caddy because it would serve a niche of people due to the fact that the second version of Caddy still haven't any (good) plugins and the first version of Caddy is still very usable.

[polarathene]

I presume that you use this project already.

I do not, I'm evaluating Caddy.

Do you use it on production?

It would be for production on a single server(quadcore with 16GB RAM), only around 1k active users during peak traffic.

Do you use it with Docker Swarm or just raw docker containers?

I do not use swarm but have kept an eye on it since release. Just using containers with a few compose configs.

Do you plan to migrate to Kubernetes?

Eventually yes. This is volunteer work in my spare time and the amount of terminology/knowledge required to migrate to k8s has offset that for now. At least it seems like it has a higher learning curve than getting other volunteer devs accustomed to k8s vs compose.

Did you consider using Traefik? If yes, why do you prefer Caddy for load balancing your containers?

I have tried traefik and I like it,. but v2 does make dealing with redirect to HTTPS a tad annoying(compared to v1). Brotli support has been requested for some time, but no response from developers, OCSP stapling support is another one among a few other TLS support features, again lacking much activity from the developers currently while they bring v2 to release.

Caddy afaik addresses all of that, and this project helps get a more traefik like experience, but I'd like to know if the functionality will be available with v2 release next year. I could alternatively look at just using Caddy for TLS termination and forwarding to Traefik for routing, which would also alleviate my concerns with Traefik.

[pwFoo]

I like that swarm is that simple and just works. compose syntax is easy and I haven't found a as simple and extensive documentation for kubernetes... So I would still prefer swarm with caddy reverse proxy.

[cmizzi]

Traefik doesn't allow (or badly) HA, while Caddy supports it very well thanks to its plugins (see my blog post about that). The use of Kubernetes belongs to everyone, but I agree with @pwFoo about the simplicity of writing under Swarm. I didn't really read up on the new features of Caddy version 2 but I think it would be interesting, anyway, to port this plugin to version 2.

Currently, I work with about 150 clients that I host on the same server. On the other side, I have several servers containing the main services offered (database, etc.). These were using Traefik because it initially offered a wide variety of options for easy configuration. However, nowadays and taking into account my needs, Caddy corresponds better to my expectations. I don't use K8S and my containers need to be accessible from the outside. If one day I have to switch to version 2 of Caddy, I will still have the same need, namely to be able to listen to the events of the Swarm in order to generate the associated routings.

[polarathene]

Traefik doesn't allow (or badly) HA

Was that referring to v1.x? Your issue appeared to be with Consul for K/V use(which I've also seen others have bad experiences with), and I think that only recently became available in v2(2.1)? I haven't used consul personally or HA myself, so I might not be understanding the issue right.

Your blog mentioned the size limit with consul, and how Traefik was managing/storing certs in one large JSON file?(which doesn't sound like it'd be suitable with Consul, especially with the possibility of growing beyond that 512KB limit) While your solution with Caddy + Consul was better because certs were separate entries for consul to manage.

I haven't looked into the topic too much, but regarding HA, on v1.x consul or not, there was issues with LetsEncrypt, that was removed from v2 and made available again, but only via EE version. HA in general though is meant to be fine for Traefik:

Because Traefik is stateless. So you can deploy as many instances as you like, and each instance will do its job (just like Envoy / Nginx / other solutions).

So your HA issue with Traefik was specifically about keeping certs in sync across Traefik instances? Or the 2nd issue raised in the blog post regarding the error, which based on the github issue linked, indicates it was more of a problem with LetsEncrypt interacting with the same instance for renewing a cert(stateful). A DNS challenge probably would have worked better than HTTP in that case?

It's not clear what Caddy is doing differently here, other than the Consul plugin having might smaller entries to sync/update, which if Caddy isn't doing anything differently, may just happen to sync the change across all instances in a timely manner for the LetsEncrypt HTTP challenge to work if it hits a different instance?

With v2 of Traefik, afaik it focuses on handling the data plane of a Service Mesh, while Maesh acts as a control plane. So in this case, isn't Docker Swarm providing your HA with multiple Traefik/Caddy instances?

---

EDIT: It's now also possible to avoid Traefik EE and deal with LetsEncrypt externally, currently there are docs for setting it up with k8s. Granted, not as nice as it's one more thing to setup, however I think your blogpost misunderstood why Traefik ran into the issues when Caddy didn't?

[piaste]

Just adding a response to your questionnaire to provide an extra data point.

• Do you use it on production?

Yes.

• Do you use it with Docker Swarm or just raw docker containers?

With Docker Swarm.

• Do you plan to migrate to Kubernetes?

No such plans right now.

It's definitely an option and I'm well aware that it won the orchestration war, but Swarm so far is more than sufficient for our limited needs and is much simpler to manage. I would want to have at least two fully-trained K8S admins on staff before moving over.

• Did you consider using Traefik? If yes, why do you prefer Caddy for load balancing your containers?

I was aware of Traefik, but we'd already had a stellar experience with Caddy in non-Docker environments and we were familiar with Caddyfiles configuration, so it was our first choice, and since it handled all our requirements fine on the first try we didn't spend time on alternatives.

Having said that, in the last few months I've had two show-stopping failures - one was #104, and another was a mysterious case of the Caddy service somehow filling up the entire disk space that I can't pin down any more specifically than an unhelpful "it goes away if i restart the service by scaling it down to 0 and back up again".

This is perfectly understandable for an open-source project of this scope, of course, but it does mean that I now intend to find the time to give Traefik a try.

[codeagencybe]

@lucaslorentz Well i'm still interested in having this solution for Caddy2.

Docker Swarm may have less interest but it still is a huge user group. Don't forget many people are not yet into K8s as it's way more complex than Swarm. We gave it a try also with Rancher and while some parts are easy, some others like specific routing/ingress and persistant storage are a hell for starters. I have never spent so much time on troubleshooting to get an application reachable. At the end I just gave up and went back to my good old Swarm.

And believe me, there are tons of exact use cases like this. Swarm is way easier and simpler to get running in production than K8s, not to forgot also a hell lot cheaper (in terms of burning hours due to steep learning curve).

Secondly, your plugin still has a very interesting feature and value that is unique in the market, with the exeption of Traefik. The power of auto discovery thanks to labels! It's either Caddy1 or Traefik, there is nothing else. So it would make many Caddy1 users very convenient if they can move over from v1 to v2 without losing this feature. Makes sense?

Anyway, I'm in the difficult scenario at the moment of also trying Traefik and bagging my head against the wall for several weeks and now decided to give Caddy2 a try. And having your plugin working for v2 would absolutely help me out.

Could you give an estimation on how much time you need to make this possible? Maybe there are other options possible like community donating some money to help you with the resources?

It would be a real shame if something like this gets abandoned as long as Docker swarm is around, which I believe will still be for at least several years.

Thanks for your great value and creating this plugin! I really hope you can find the goodwill to bring a v2 soon. If I can help you out in any way, please let me know.

[pwFoo]

I love the power of that proxy plugin because of the configuration flexibility! It's possible to generate each configuration with some (more or less complex) labels.

It would be awesome to configure caddy webserver and caddy proxy just with docker labels! Maybe it not need to be swarm (or reverse proxy) only?

[gamalan]

Agreed. I use Caddy because Traefik removing the on-demand tls feature. There's alternative OpenResty+Nginx, but I don't think it's good enough. Moving to kubernetes also might not be feasible for our team right now.

Aside that, i got few understanding about this plugin, so I will probably try to update it for v2.

[rgdev]

+1 We use Traefik but sadly 1.7 suffers from issues with distributed stores and 2.X removed the ability to store certificates in KV Stores in favor of their Enterprise Edition so we'd very much like to move to Caddy2 and this addon would be perfect for our needs.

[mholt]

Subscribing to this thread :eyes:

I do think this plugin is super cool, and I want to see some functionality like it for v2. Caddy 2 is extremely flexible and extensible, and although we're wrapping up the 2.0 release in the next few days or maybe 2 weeks at the most, there's still a lot we can add to it after that.

(Btw, we're looking for help to finish building the ingress controller; it's not my cup of tea! Let me know if you're interested.)

I might as well mention a few things while I'm here:

Caddy 2 has a really flexible and intuitive API which can be used for any sort of dynamic, on-line config changes like adding upstreams and stuff: https://caddyserver.com/docs/api -- so you can implement any logic around that API without even needing a Caddy plugin.

@unixfox

due to the fact that the second version of Caddy still haven't any (good) plugins and the first version of Caddy is still very usable.

That's quickly changing! I know of about 12-14 plugins in development currently (aside from config adapters): WebDAV handler, DynamoDB storage, Redis storage, Consul storage, several HTTP authentication/authorization plugins, git client, a distributed cache handler, secure HTTP/2 forward proxy, several DNS providers, dynamic DNS app, and several others. (That's also not counting the fact that many plugins that were necessarily plugins in v1 aren't needed in v2 at all anymore; for example, the realip, ipfilter, and cors plugins can mostly be done natively.)

Give the plugin ecosystem a few months to mature -- but mature it will, as long as we keep reminding ourselves that v1 will eventually/soon be discontinued. 🙂 We're trying to encourage people to upgrade here.

@polarathene

Caddy afaik addresses all of that, and this project helps get a more traefik like experience, but I'd like to know if the functionality will be available with v2 release next year. I could alternatively look at just using Caddy for TLS termination and forwarding to Traefik for routing, which would also alleviate my concerns with Traefik.

I will say that is not a bad idea, to at least put Caddy in front; Caddy has objectively better TLS handling (see: recent PKI events such as extended OCSP responder outages and mass revocation events and you'll notice that only Caddy was unaffected -- among many other good reasons) and is more flexible in the long run.

We've also significantly improved the on-demand TLS feature in Caddy 2.

It's not clear what Caddy is doing differently here, other than the Consul plugin having might smaller entries to sync/update, which if Caddy isn't doing anything differently, may just happen to sync the change across all instances in a timely manner for the LetsEncrypt HTTP challenge to work if it hits a different instance?

Caddy coordinates certificate management across a cluster for all instances that are configured with the same storage backend. Deploy 2 or 20 or 2000 Caddy instances sharing the storage, and they'll share certificates, OCSP staples, and even TLS session ticket keys if you want. Take a few instances down, no problem, the site will go on. It's automatic, and not an enterprise-only feature like with Traefik. That's the primary difference. Also, as I said above, Caddy's cert logic is superior in many ways and will keep your site online when other servers... well... don't.

---

Proposal

Our goal is fewer moving parts, so less to maintain, less sunk costs, and less that can go wrong. So although Caddy 2's API can already do these kinds of dynamic changes, it would require another moving part. (The API is not bad -- it is actually very good and you can totally use it! But when possible, we can bake it in.)

So, here is my proposal for how this might work in v2 without needing its API:

If all you need is to dynamically update the list of proxy upstreams/backends (see the "upstreams" array), I could imagine a config where instead of specifying a static list of upstreams, you specify an upstream source. An upstream source could be a module that simply keeps the list of upstreams updated. (Under the hood, we'd refactor things so that the static list of upstreams actually is an upstream source, so the singular source of truth is the upstream source module.)

One such upstream source module could use Docker labels.

In other words, instead of:

...
"upstreams": [
    {"dial": "some-app:80"}
]

you'd do something like:

...
"upstream_source": {
    "module": "docker",
    ...
}

This config is more static, and simply keeps the list of upstreams current according to Docker labels.

I don't know how all the Docker stuff works though. So I'd need some help from any/all of you. I could help with the changes needed in Caddy's reverse proxy, you'd just have to implement the code that gets the labels from Docker, which this plugin already does I suppose.

---

Please let me know your thoughts.

[lucaslorentz]

@mholt About kubernetes ingress controller, it is something I like to work on, but I will not have time to work on it anytime soon. I will let you know if this changes.

Here are my 2 cents for the ingress controller: Most ingress controllers are ditching the native ingress resource concept, and going towards CRD. That affects their integration with things like external-dns, and generates lot of complains. I believe that's why Traefik 2.2 focused so much in ingress integration again. So, if you focus only on CRD, like a lot of other projects are doing, make sure existing projects like external-dns would be able to integrate with it.

About the list of upstream, I don't think that would support additional configuration, like declaring a new website to be hosted and configuring http filters unrelated to proxy. Being able to configure the full caddyfile in a distributed way (in different docker services) is key for this project.

Traefik have a good architecture for this, you can plug in a single or multiple source of configuration. Consul, Kubernetes, Docker and so on. And if I configure the same website on Consul and Docker, their config should be merged and it should proxy to both back-ends.

I think the concept of multiple source of config should be baked in caddy, including merging config from those multiple sources. Then this plugin would only convert docker labels to caddy config. If not baked in caddy, there should be a caddy configurer project, that is able to read configs from multiple sources and call caddy API to setup the final merged configuration from all sources.

If we go down the Caddy V2 API path, and we have a cluster of caddy instances. Should we call the API in every instance to keep them in sync? Or are they able to sync configuration automatically?

[mholt]

Thanks for the explanation @lucaslorentz .

I think the concept of multiple source of config should be baked in caddy, including merging config from those multiple sources.

I think it can be done. The admin endpoint is extensible. We just need a piece of code that lives in a Caddy module that can keep track of all the different config sources, assemble them into one structure, and call Load().

I don't know how all those tools work, but I'm willing to help people integrate the code that interacts with them into Caddy.

If we go down the Caddy V2 API path, and we have a cluster of caddy instances. Should we call the API in every instance to keep them in sync? Or are they able to sync configuration automatically?

Caddy 2 is capable of this, but it's not yet implemented. We'll see if/when it's needed. But I think we won't need this to support auto-updating configs; as I said above, I think we can make a Caddy module that takes care of it within Caddy.

[andrewdhastings]

I'm no expert in all of this, but I wanted to jump in and add my vote for these features.

To be able to configure labels/env on a docker-compose file and have it automatically detect and add them to Caddy2 would be very helpful.

I have been using a version of this functionality using Nginx (https://github.com/Verisage/docker-nginx-letsencrypt-odoo) But to move this to a clean simple Caddy2 setup with one container would be awesome.

[mholt]

@andrewdhastings Awesome, I think we can make it happen -- we got your email and will be in touch!

[lucaslorentz]

I started working on caddy v2 integration for this plugin.

My plan so far is to create a caddy v2 app called "docker".

That app would convert caddy labels to docker v2 json and apply them using caddy API.

API urls would be configurable and would default to local API.

User can chose to run docker app and http server in separate containers or together.

Will push a branch when I have something more concrete.

@mholt what do you think?

[mholt]

@lucaslorentz I think that will work pretty well -- but as long as the docker app is generating the entire Caddy config to be used, then you can easily pass it into caddy.Load() (or its sister function, caddy.Run() which takes a *Config, but you'd still have to populate the 'Raw' fields of json.RawMessage where modules are used either way). You don't necessarily need to use the API endpoint.

[mholt]

FYI, this plugin is getting some very positive attention in a reddit thread: https://www.reddit.com/r/selfhosted/comments/gj550w/been_looking_for_a_new_resvere_proxy_to_use_with/fqinyrt/

I think there'll be a lot of interest in a v2 equivalent of this; so, definitely ping me if I can help, where I am able (despite my lack of Docker knowledge).

@lucaslorentz If you want an invite to our dev slack, if having that kind of feedback cycle would be helpful to you, let me know which email to invite.

[lucaslorentz]

Thanks @mholt Please, invite lucaslorentzlara[at]hotmail.com So far I found everything I needed in caddy docs, issues, and/or source code. But slack might be useful for future questions/discussions.

I Implemented what I described above, generating JSON and as caddy app, and learned with it. Now I don't think that's the right path for this project, because caddy JSON config is quite complex to be expressed in caddy labels.

Now I'm implementing it focused on Caddyfile, like the current version. A migration path for users from caddy v1 to v2 with docker should be quite straight forward, as all existing features and basic labels will stay the same.

And instead of an app, I'm doing it as a caddy command. It will be executed with caddy docker-proxy, and current CLI options will be available as well.

I made good progress so far, I will probably create a PR tomorrow, then feedback will be very welcome.

[mholt]

@lucaslorentz Thanks for the update -- invite sent!

(Again, I don't know much about Docker, but a command sounds good to me too, since plugins can also add commands to the CLI. :+1: )

[andrewdhastings]

I'm happy to do some testing and feedback when you have something ready.

On Sat, May 16, 2020 at 11:27 AM Matt Holt notifications@github.com wrote:

@lucaslorentz https://github.com/lucaslorentz Thanks for the update -- invite sent!

(Again, I don't know much about Docker, but a command sounds good to me too, since plugins can also add commands to the CLI. 👍 )

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lucaslorentz/caddy-docker-proxy/issues/130#issuecomment-629680095, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHYGNQ2K2OVDOCDB2YJPRDTRR3EIPANCNFSM4J2HDDZQ .

--

Andrew Hastings The Catalyst Dream Mountain Services P: M: (801) 770-4869 (435) 890-0526 W: E: DreamMtn.Services http://dreammtn.services/ Andrew@DreamMtn.Services andrew@dreammtn.services https://www.facebook.com/DreamMtnServices/ https://twitter.com/DreamMtnService https://www.linkedin.com/in/andrewdhastings https://www.instagram.com/dreammtnservices/

Request a Free Odoo/ERP Consultation http://dreammtn.services/contact-us

[lucaslorentz]

PR is there. https://github.com/lucaslorentz/caddy-docker-proxy/pull/137

PR is big and has a lot of refactorings.

For now, I don't have docker images for testing, once I merge the PR the images tagged with CI will have the latest changes.

[lucaslorentz]

Sorry, this is still work in progress Another PR will come soon wrapping up things

[lucaslorentz]

Now we have the first docker images for testing: lucaslorentz/caddy-docker-proxy:ci lucaslorentz/caddy-docker-proxy:ci-alpine

Readme is also updated with new caddy v2 instructions: https://github.com/lucaslorentz/caddy-docker-proxy

Keep in mind that special labels (address, targetport...) were removed. I understand this is a breaking change, but this is the best moment to do it, while people are transitioning to V2 and already expect some mismatches.

Please, play with it a bit and provide feedback.

PS: we might still do major changes before releasing a final V2 version

[andrewdhastings]

I was able to run some tests successfully. Super excited for this! Will be doing some more extensive testing in the coming weeks.

I did have a question, is there a way to set global variables with environment or labels under the caddy service? https://caddyserver.com/docs/caddyfile/options Mainly, it would be nice to set an email for ACME account.

Also, not a big deal, but I think you have your volumes and networks names swapped in this file. https://github.com/lucaslorentz/caddy-docker-proxy/blob/master/examples/efs-volume.yaml

[francislavoie]

As noted in the PR, I think you set global options by having the labels be first, and setting them like this: caddy.email = you@example.com. As soon as you define a snippet or site block then you can't set those anymore, so they need to be in order. Correct me if I'm wrong @lucaslorentz, I've yet to try it all out myself.

Actually, looking at the test here https://github.com/lucaslorentz/caddy-docker-proxy/blob/master/plugin/caddyfile/fromlabels_test.go#L130 this looks incorrect. Global options must be within braces, like this:

{
    email you@example.com
}

So I think that might be incorrectly implemented... but I'd need to dig deeper.

[lucaslorentz]

To be honest, I just saw the global block documentation for the first time: https://caddyserver.com/docs/caddyfile/options

It was a coincidence I used the term global as well. And my reply on that PR is wrong. I was using the term global to refer to directives outside any server block, like:

localhost

reverse_proxy /api/* localhost:9001
file_server

I will adjust implementation to do:

caddy.directive=arg1 arg2
↓
{
  directive arg1 arg2
}

As that's probabbly just code removal, because that would be the natural behaviour because you didn't set keys for caddy caddy=something.

[rgdev]

Where should the global parameters be defined ? Since they don't belong in the scope of a proxied container I assume you would read labels on the caddy container/service itself ? If we do that it means Caddy starts with a blank conf and serves the default caddy welcome page until the docker proxy addon kicks in and replace its caddyfile conf

What we did with Traefik is pass all the global static config as parameters on Traefik itself and let it merge the dynamic config (labels) on top of that :

  traefik:
    image: traefik:1.7.24-alpine
    command:
      - "--api"
      - "--entrypoints=Name:http Address::80 Redirect.EntryPoint:https"
      - "--entrypoints=Name:https Address::443 TLS"
      - "--defaultentrypoints=http,https"

AFAIK Caddy don't support that kind of mechanism

[lucaslorentz]

@rgdev This should get it ready for global options: https://github.com/lucaslorentz/caddy-docker-proxy/pull/143 Once merged, you will be able to define them as labels in any docker service. Including caddy itself. Example: https://github.com/lucaslorentz/caddy-docker-proxy/blob/63a38c9c29e8746862bc0c6ee3ae71f8f649a50b/examples/example.yaml#L21

[francislavoie]

I think we can probably close this issue now @lucaslorentz, what do you think? Anything remaining can be made new issues and tracked separately.

Of course, we'd appreciate any testing/feedback from anyone so we can continue to improve this! :grin:

[andrewdhastings]

This is perfect.

Also, sorry if this is a noob question, but is there a difference between:

labels:
        caddy: whoami0.example.com
        caddy.reverse_proxy: "{{upstreams 8000}}"
        caddy.tls: "internal"

and

labels:
        - caddy= whoami0.example.com
        - caddy.reverse_proxy= {{upstreams 8000}}
        - caddy.tls= internal

[ptman]

@andrewdhastings are you sure that's how you wanted to write? one single line?

[andrewdhastings]

No, sorry, it was multiple lines but when I posted the comment it put them on one line...

On Tue, May 19, 2020 at 9:13 AM Paul Tötterman notifications@github.com wrote:

@andrewdhastings https://github.com/andrewdhastings are you sure that's how you wanted to write? one single line?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lucaslorentz/caddy-docker-proxy/issues/130#issuecomment-630886612, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHYGNQ47A5XEXTEXXEFYPGTRSKOYNANCNFSM4J2HDDZQ .

--

Andrew Hastings The Catalyst Dream Mountain Services P: M: (801) 770-4869 (435) 890-0526 W: E: DreamMtn.Services http://dreammtn.services/ Andrew@DreamMtn.Services andrew@dreammtn.services https://www.facebook.com/DreamMtnServices/ https://twitter.com/DreamMtnService https://www.linkedin.com/in/andrewdhastings https://www.instagram.com/dreammtnservices/

Request a Free Odoo/ERP Consultation http://dreammtn.services/contact-us

[francislavoie]

Use `` instead of just a single backtick, on their own lines before and after the code to preserve newlines. Single backticks are for inline codelike this`.

I updated the comment for you (click edit to see what I did :sweat_smile:)

Anyways, yes, I think those should be the same. Personally I'd put a space before the = for readability but that shouldn't affect functionality.

[andrewdhastings]

Thanks for both tips :)