Athens leaks etcd leases

[uhthomas]

Describe the bug

Athens creates a new session (lease) for each stash request, and then never closes the session. The leases are then held indefinitely, or until the process is restarted, which causes excessive load on the etcd cluster and eventually overloads it.

The spikes are from restarting Athens.

Error Message

N/A

To Reproduce Steps to reproduce the behavior:

1. Run Athens with etcd configured for singleflight.
2. Make a few thousands requests for modules which aren't already cached.
3. Observe the number of leases increases indefinitely.

Expected behavior

Athens should either create a single session for all locks, or close the session. I am not sure which is correct as I could not find great documentation on what is recommended by the etcd authors. I imagine it's probably best just to make sure the session is closed.

Environment (please complete the following information):

• OS: Linux
• Go version: N/A
• Proxy version: https://github.com/gomods/athens/commit/e248d228921243998aca56dab01786e3cb938919
• Storage (fs/mongodb/s3 etc.): etcd and s3

Additional context

[santiagoPinzonD]

Hello, @uhthomas wanted to know how did you solve this problem ? I know that the fix is already in the code but not yet released. this is my singleflight config, I have a lot of traffic and I have to restart constantly because it consumes the machine's memory.

[uhthomas]

@santiagoPinzonD You will have to build Athens from source.

FROM golang:1.21.3-alpine AS build

RUN apk update && apk add git

RUN go install github.com/gomods/athens/cmd/proxy@main

FROM gomods/athens:v0.12.1

COPY --from=build /go/bin/proxy /bin/athens-proxy

[manugupt1]

Please use canary tag which has more fixes and built from master's latest. I will see if I can do a release in coming weeks

[uhthomas]

Thanks @manugupt1, wasn't aware of this image - though it is a shame that there aren't tags for each commit.

Current latest digest for canary: gomods/athens:canary@sha256:372c9cb22c767f86e8c15a4534b3dcdc837f6bc7a667647a0458814cc0cdc59e

[r-ashish]

We do have tags for every commit but under a separate registry: https://hub.docker.com/r/gomods/athens-dev/tags. Feel free to use it if you wanna pin to a specific commit until the next stable version is released.

[santiagoPinzonD]

Done, thanks guys yesterday I put in production the canary, let's see if it manages to solve the memory problem, I take this opportunity to ask you, we currently have two organizations in github, so we are running two athens container on the same server, this apparently has not been a problem, until we doubled the traffic an average of 20 million request, I opened another issue regarding this issue and commented that you can outsource the database. is there any guide about this? and how can I improve the performance?

[uhthomas]

What specifically are you asking, sorry? There is no database required for Athens. Just use an S3-compatible storage system and the existing etcd cluster. You shouldn't need a separate instance of Athens per GitHub organisation either.