search

commanded / eventstore

Event store using PostgreSQL for persistence
MIT License
1.05k stars 146 forks source link

Remove subscription lock at boot #287

Open ericlyoung opened 3 months ago

ericlyoung commented 3 months ago

How would I disable the subscription advisory lock at startup of my app. It is preventing me from being able to smoothly release a new version of my backend. The lock prevents the new app from being able to subscribe until the lock times out, 60 sec. The old lock hangs around, while that container is already shut down.

I would like to remove the lock at boot or disable the lock in the first place.

There is no way someone is using this in production and has not run into this issue.

ericlyoung commented 3 months ago

What is the easiest way to disable the advisory lock for now?

ericlyoung commented 3 months ago

been on this for many hours. I've discovered that docker swarm is too aggressively killing the container with SIGKILL, when running docker stack deploy onto an already running stack (for auto-update), which prevents the lock release I've discovered that docker service rm ... does correctly send SIGTERM, which allows the lock release to run

So, currently as a work-around I'm doing docker service rm MYAPPSERVICE && docker stack deploy --with-registry-auth --compose-file stack.yml MYAPPSTACK which gives me the lock cleanup and app update

I did notice you have a concurrency_limit option for the subscription, which gave me hope that I wouldn't have to mess with any of this but I set that to 99 and it didn't work.

more updates to come as I seek the best solution to this

ericlyoung commented 3 months ago

I'll be honest, the code is so all-over-the-place that I can't figure out how to disable the subscription advisory locks. Please help me disable it.

jdewar commented 3 months ago

This is something I will likely see for myself in the next few weeks. If I have any insight, I will post here.

As an aside, if the issue is the KILL stop signal, and a softer signal would suffice, it looks like there are a few options: https://github.com/moby/moby/issues/25696

ericlyoung commented 3 months ago

This is something I will likely see for myself in the next few weeks. If I have any insight, I will post here.

As an aside, if the issue is the KILL stop signal, and a softer signal would suffice, it looks like there are a few options: moby/moby#25696

After a huge amount of research I discovered that swarm is somewhat abandoned, and one of the things that's broken is the fact that you have no control how containers are stopped. SIGKILL is just plain sent. I tried every single mechanism that is supposed to work, 3-4 different ones. Maybe they worked 4 years ago, but all are ignored now.

Since my needs are very simple, I ended up switching to docker compose on the server, which does send the correct signal to stop containers, and so I do not hit this wall anymore.

All that being said, this project should have an option to force remove subscription lock at boot OR to not use it at all.