Deploy with kamal

[gedw99]

https://github.com/MightyMoud/sidekick?tab=readme-ov-file#inspiration

Would be awesome if we used this . Being golang it’s easier to manage. It’s still a little new. It works on DO and Hetzner because they both provide nested virtualisation . Just like Kamal I believe .

curious what you think .

I think we could also use gossip to control sidekick for managing fleets . Just an idea ..

[leomorpho]

Hey thanks for this idea! I must say I really like it, first time I hear of this project.

I am not quite clear on whether Sidekick can support deploying more than a single binary in a docker file? For the current state of Goship, it still depends on Redis. Now, it's not that hard to remove it, and asynq (which requires redis) can be switched with backlite like in this PR from Pagoda. Personally, I'm just not 100% convinced by backlite yet...and I've been looking at other task runners but haven't found one I'm in love with.

It seems reasonable to add different ways to deploy, though we'd need to keep that clean. Currently, kamal files live in the root, and adding another deployment tool might add more stuff? If that's the case, we'd need to revisit the file organization just to keep it sane. Long-term, if Sidekick turns out to be better and easier to manage, it might make sense to drop Kamal for a simpler tool, and one that is in Go too.

Would you be interested to help integrating Sidekick? Let me know :)

[gedw99]

Yes task runnners are hard to pick .

backlight looks pretty good. SQLite , so no HA, and replication if the SQLite will not I suspect solve that .

The nats task runner is also cool if you need HA , retry and global redundancy.

Yes I would be happy to help with getting sidekick working for goship.

[gedw99]

To avoid the docker compose problem I use runner . Works very well for my projects .

https://github.com/cirello-io/runner

So docker is still used but runner is injected and runs all your stuff for you .

Runner can even run multiples to simulate clusters . They call it formations .

I hook it up with nats so I can control how many of what I want across globally distributed servers.

you can also she it further and do partial updates of binaries running inside a docker using nats .

[leomorpho]

Sidekick support sounds good to me. For the docker-compose problem, I think we could ignore it to start and simply deploy for a single binary? Redis is the last standalone service dependency, and apart from the task runner, it's only used for SSE events, a feature that can be reimplemented to remove that need for Redis. Thoughts?

I've read up a bit about nats, it's new to me. One concern there for me is that it increases the complexity. I also would want to eventually keep this as a single binary, and only have the option for Postgres/Redis if one requires it. I guess I don't see the need to introduce nats? Perhaps I'm lacking context...definitely possible...

[gedw99]

If Redis is only used for SSE then it could be replaced .

Nats can run in process . So no extra process . And it can do the sse push .

https://m.youtube.com/watch?v=cdTrl8UfcBo

so 1 process is the result .

—-

the multi process way is to run side kick with runner , nats, golang main binary .

https://github.com/cirello-io/runner Kicks off and ensures the running of all processes.

So I can run without docker or with docker .

[gedw99]

What currently runs your Procfile ?

[gedw99]

Hey thanks for this idea! I must say I really like it, first time I hear of this project.

I am not quite clear on whether Sidekick can support deploying more than a single binary in a docker file? For the current state of Goship, it still depends on Redis. Now, it's not that hard to remove it, and asynq (which requires redis) can be switched with backlite like in this PR from Pagoda. Personally, I'm just not 100% convinced by backlite yet...and I've been looking at other task runners but haven't found one I'm in love with.

It seems reasonable to add different ways to deploy, though we'd need to keep that clean. Currently, kamal files live in the root, and adding another deployment tool might add more stuff? If that's the case, we'd need to revisit the file organization just to keep it sane. Long-term, if Sidekick turns out to be better and easier to manage, it might make sense to drop Kamal for a simpler tool, and one that is in Go too.

Would you be interested to help integrating Sidekick? Let me know :)

https://github.com/choria-io/asyncjobs Is based on asyncq

I use it .

because it uses nats you can run it in process , single server of cluster or multi region cluster or st edge ( with air gapping ) using nats leaf

[leomorpho]

Currently to run my Procfile I use overmind, but that's used only for local dev. I don't think there is any need for docker-compose as the goal is to remove Redis as a dependency, so the final project is a simple binary that can be deployed with whatever tool suits one's need.

I am totally in to add more deployment ways, but it needs to remain highly organized, i.e. all deploy-specific configs/code should live within for example a deploy directory, with one child directory per deployment method. I think the project can already feel overwhelming when first picked up by someone, and I want to go towards a direction of simplicity and functional organization. As long as whatever is added fits that paradigm, I am all for it.

NATs seems pretty cool to me, but I am not convinced increasing the system complexity with it is warranted. I took a look at asyncjobs and it seems interesting. I couldn't find any easy to use UI for the task monitoring. That is actually something I'd want. It would either need to be a standalone tool (like asynqmon) or built into goship, which frankly seems out of scope. The reason I'd want a UI is it makes one's life simpler for monitoring in prod.

I think a good first step, if you plan to use sidekick with goship is to add support for it? Let me know what you think. That would also keep the scope limited for now. What do you think?

[gedw99]

Yep Side Kick for deployments in good first step. And yes limited scope.

pagoda is 100% SQLITE these days, and does not need Redis or PostreSQL: https://github.com/mikestefanello/pagoda?tab=readme-ov-file#storage

So once you flip over to that, its possible to use a single docker and hence use sidekick.

I prefer SQLITE because its easy to scale out globally, and so use the bring data to the compute.

marmot ( which uses nats) scales out SQLITE easily in a master / master fashion.

Runner can run Pagoda in the same docker and scale out with 3 per DC, and say 3 DC = 9 which gives local and global redundancy.

Runner can also run anything you need in the same docker. With its formations, you can turn on different parts globally also across a global cluster, since it has a Discovery system, and so allows you to real time modify it.