NATS ?

[gedw99]

I wonder if NATS is useful ?

https://github.com/nats-io/nats-server

It can be embedded also, so nothing else to run

I change to a KV or blob is then distributed to all other NATS Servers.

can also be used for syncing any config too, so your Caddy Server all look the same etc.

[darkweak]

It could be implemented, I plan to add a FS only storage too so we could add NATS at the same time.
Do you want to open a PR for NATS? I'm quite busy ATM.

[darkweak]

Hey @gedw99 to keep you informed about that, NATS storage will be in the next release.

[gedw99]

Hey @darkweak super.

I was thinking... Yes it's dangerous.

NATS Leaf and NATS Server does caching in a way. When a KV value changes in NATS Server, NATS Leaf nodes get it. It has TTL properties you can control too.

Its kind of like its own CDN, so I use wanted to put this out there, because its relevant to how the integration is done in a smart way...

Here is a pretty decent refence Project that works with NATS KV in a way thats kind of similar too. Might help I think. CLI is pretty nice and it has most of the Auth techniques too.

https://github.com/hooksie1/piggybank

Many users will be happy for Caddy to embed NATS JetStream, so then there is nothing to run. Can be run as NATSLeaf Node or NATS Origin Node.

Also worth setting up a Super cluster locally for testing with proper Config. https://natsbyexample.com/examples/topologies/supercluster-jetstream/cli

I wish I had time to contribute but life is short and so is money :)

---

[darkweak]

@gedw99 don't worry, I have some time to spend on that. I will just need some beta-testers for this storage when implemented.

[gedw99]

Glad that your experienced with nats.

yes I will be happy to beta test .

[darkweak]

Hey @gedw99 can you try with the following command to build your caddy instance with Souin and the nats storages please?

xcaddy build \
    --with github.com/darkweak/souin/plugins/caddy@b0a36db1b550b66f768a3793701b69358200fed3 \
    --with github.com/darkweak/souin@b0a36db1b550b66f768a3793701b69358200fed3 \
    --with github.com/darkweak/storages/nats/caddy

[gedw99]

hey @darkweak

It compiles and runs on my Mac m2:

If you have a demo folder with a Caddyfile to drive it, I can dog food it some more.

 go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest

xcaddy build \
    --with github.com/darkweak/souin/plugins/caddy@b0a36db1b550b66f768a3793701b69358200fed3 \
    --with github.com/darkweak/souin@b0a36db1b550b66f768a3793701b69358200fed3 \
    --with github.com/darkweak/storages/nats/caddy
2024/07/17 21:52:56 [INFO] absolute output file path: /Users/apple/workspace/go/src/junk/darkweak/caddy
2024/07/17 21:52:56 [INFO] Temporary folder: /Users/apple/workspace/go/src/junk/darkweak/buildenv_2024-07-17-2152.1108159417
2024/07/17 21:52:56 [INFO] Writing main module: /Users/apple/workspace/go/src/junk/darkweak/buildenv_2024-07-17-2152.1108159417/main.go
package main

import (
    caddycmd "github.com/caddyserver/caddy/v2/cmd"

    // plug in Caddy modules here
    _ "github.com/caddyserver/caddy/v2/modules/standard"
    _ "github.com/darkweak/souin/plugins/caddy"
    _ "github.com/darkweak/souin"
    _ "github.com/darkweak/storages/nats/caddy"
)

func main() {
    caddycmd.Main()
}
2024/07/17 21:52:56 [INFO] Initializing Go module
2024/07/17 21:52:56 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod init caddy 
go: creating new go.mod: module caddy
go: to add module requirements and sums:
    go mod tidy
2024/07/17 21:52:56 [INFO] Pinning versions
2024/07/17 21:52:56 [INFO] exec (timeout=0s): /opt/homebrew/bin/go get -d -v github.com/caddyserver/caddy/v2 
go: added github.com/beorn7/perks v1.0.1
go: added github.com/caddyserver/caddy/v2 v2.8.4
go: added github.com/caddyserver/certmagic v0.21.3
go: added github.com/caddyserver/zerossl v0.1.3
go: added github.com/cespare/xxhash/v2 v2.2.0
go: added github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572
go: added github.com/google/pprof v0.0.0-20231212022811-ec68065c825e
go: added github.com/google/uuid v1.6.0
go: added github.com/klauspost/cpuid/v2 v2.2.7
go: added github.com/libdns/libdns v0.2.2
go: added github.com/mholt/acmez/v2 v2.0.1
go: added github.com/miekg/dns v1.1.59
go: added github.com/onsi/ginkgo/v2 v2.13.2
go: added github.com/prometheus/client_golang v1.19.1
go: added github.com/prometheus/client_model v0.5.0
go: added github.com/prometheus/common v0.48.0
go: added github.com/prometheus/procfs v0.12.0
go: added github.com/quic-go/qpack v0.4.0
go: added github.com/quic-go/quic-go v0.44.0
go: added github.com/zeebo/blake3 v0.2.3
go: added go.uber.org/mock v0.4.0
go: added go.uber.org/multierr v1.11.0
go: added go.uber.org/zap v1.27.0
go: added go.uber.org/zap/exp v0.2.0
go: added golang.org/x/crypto v0.23.0
go: added golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842
go: added golang.org/x/mod v0.17.0
go: added golang.org/x/net v0.25.0
go: added golang.org/x/sync v0.7.0
go: added golang.org/x/sys v0.20.0
go: added golang.org/x/term v0.20.0
go: added golang.org/x/text v0.15.0
go: added golang.org/x/time v0.5.0
go: added golang.org/x/tools v0.21.0
go: added google.golang.org/protobuf v1.34.1
2024/07/17 21:52:56 [INFO] exec (timeout=0s): /opt/homebrew/bin/go get -d -v github.com/darkweak/souin/plugins/caddy@b0a36db1b550b66f768a3793701b69358200fed3 github.com/caddyserver/caddy/v2 
go: accepting indirect upgrade from github.com/onsi/ginkgo/v2@v2.13.2 to v2.15.0
go: added github.com/RoaringBitmap/roaring v1.2.3
go: added github.com/antlabs/stl v0.0.1
go: added github.com/antlabs/timer v0.0.11
go: added github.com/armon/go-metrics v0.4.1
go: added github.com/bits-and-blooms/bitset v1.5.0
go: added github.com/buraksezer/consistent v0.10.0
go: added github.com/buraksezer/olric v0.5.4
go: added github.com/bwmarrin/snowflake v0.3.0
go: added github.com/coreos/go-semver v0.3.1
go: added github.com/coreos/go-systemd/v22 v22.5.0
go: added github.com/darkweak/go-esi v0.0.5
go: added github.com/darkweak/souin v1.6.49
go: added github.com/darkweak/souin/plugins/caddy v0.0.0-20240717102502-b0a36db1b550
go: added github.com/darkweak/storages/core v0.0.4
go: added github.com/dgraph-io/badger/v3 v3.2103.5
go: upgraded github.com/dgraph-io/ristretto v0.1.0 => v0.1.1
go: added github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f
go: added github.com/dolthub/maphash v0.1.0
go: added github.com/dolthub/swiss v0.2.1
go: added github.com/gammazero/deque v0.2.1
go: added github.com/go-redis/redis/v8 v8.11.5
go: added github.com/gofrs/flock v0.8.1
go: added github.com/gogo/protobuf v1.3.2
go: added github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: added github.com/google/btree v1.1.2
go: added github.com/google/flatbuffers v23.1.21+incompatible
go: added github.com/hashicorp/errwrap v1.1.0
go: added github.com/hashicorp/go-immutable-radix v1.3.1
go: added github.com/hashicorp/go-msgpack v0.5.5
go: added github.com/hashicorp/go-multierror v1.1.1
go: added github.com/hashicorp/go-sockaddr v1.0.2
go: added github.com/hashicorp/golang-lru v0.5.5-0.20200511160909-eb529947af53
go: added github.com/hashicorp/logutils v1.0.0
go: added github.com/hashicorp/memberlist v0.5.0
go: upgraded github.com/imdario/mergo v0.3.12 => v0.3.13
go: added github.com/maypok86/otter v1.1.1
go: added github.com/mschoch/smat v0.2.0
go: added github.com/nutsdb/nutsdb v0.14.3
go: upgraded github.com/onsi/ginkgo/v2 v2.13.2 => v2.15.0
go: added github.com/pierrec/lz4/v4 v4.1.21
go: added github.com/pquerna/cachecontrol v0.2.0
go: added github.com/redis/rueidis v1.0.31
go: added github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529
go: added github.com/tidwall/btree v1.6.0
go: added github.com/tidwall/match v1.1.1
go: added github.com/tidwall/redcon v1.6.2
go: added github.com/vmihailenco/msgpack/v5 v5.3.5
go: added github.com/vmihailenco/tagparser/v2 v2.0.0
go: added github.com/xujiajun/mmap-go v1.0.1
go: added github.com/xujiajun/utils v0.0.0-20220904132955-5f7c5b914235
go: added go.etcd.io/etcd/api/v3 v3.5.12
go: added go.etcd.io/etcd/client/pkg/v3 v3.5.12
go: added go.etcd.io/etcd/client/v3 v3.5.12
go: added go.opencensus.io v0.24.0
2024/07/17 21:52:57 [INFO] exec (timeout=0s): /opt/homebrew/bin/go get -d -v github.com/darkweak/souin@b0a36db1b550b66f768a3793701b69358200fed3 github.com/caddyserver/caddy/v2 
go: upgraded github.com/darkweak/souin v1.6.49 => v1.6.50-0.20240717102502-b0a36db1b550
2024/07/17 21:52:58 [INFO] exec (timeout=0s): /opt/homebrew/bin/go get -d -v github.com/darkweak/storages/nats/caddy github.com/caddyserver/caddy/v2 
go: accepting indirect upgrade from github.com/darkweak/storages/core@v0.0.4 to v0.0.5
go: added dario.cat/mergo v1.0.0
go: upgraded github.com/darkweak/storages/core v0.0.4 => v0.0.5
go: added github.com/darkweak/storages/nats v0.0.5
go: added github.com/darkweak/storages/nats/caddy v0.0.5
go: upgraded github.com/nats-io/nats.go v1.15.0 => v1.36.0
go: upgraded github.com/nats-io/nkeys v0.3.0 => v0.4.7
2024/07/17 21:52:59 [INFO] exec (timeout=0s): /opt/homebrew/bin/go get -d -v  
2024/07/17 21:53:00 [INFO] Build environment ready
2024/07/17 21:53:00 [INFO] Building Caddy
2024/07/17 21:53:00 [INFO] exec (timeout=0s): /opt/homebrew/bin/go mod tidy -e 
2024/07/17 21:53:00 [INFO] exec (timeout=0s): /opt/homebrew/bin/go build -o /Users/apple/workspace/go/src/junk/darkweak/caddy -ldflags -w -s -trimpath -tags nobadger 
2024/07/17 21:53:02 [INFO] Build complete: ./caddy
2024/07/17 21:53:02 [INFO] Cleaning up temporary folder: /Users/apple/workspace/go/src/junk/darkweak/buildenv_2024-07-17-2152.1108159417

././caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

-/caddy run
2024/07/17 11:53:18.699 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/07/17 11:53:18.699 INFO    serving initial configuration

[darkweak]

@gedw99 you can find the documentation website here https://docs.souin.io/docs/storages/nats/
Also the Caddyfile example that you can check https://github.com/darkweak/souin/blob/master/plugins/caddy/Caddyfile#L106-L126
You can check the Caddyfile in the nats folder here https://github.com/darkweak/storages/blob/main/nats/caddy/Caddyfile

Let me know if you need more resources about that, maybe the documentation is not clear enough and any advices to improve it are welcome. 🙂

[gedw99]

thanks @darkweak !!

thanks you so much for those links. I would have missed that fact that you had setup a really nice test setup. I will pull this tomorrow and see how it goes. Amazing work btw - wish I could do this level of coding.

[gedw99]

Will def feedback on Docs etc too.

[gedw99]

saw a spelling mistake at https://docs.souin.io/docs/configuration/

"yuo" is wrong.. you can search for this 👍

"The cdn prefix configure the upfront CDN yuo have placed before Souin"