Open Acconut opened 1 year ago
After a small attempt myself, I don't think vice is perfect for this use case. Redis can be used as a message broken but it will only be in-memory and no guaranteed arrival confirmation. Meaning, if uploader A sends a lock message over Redis and uploader B isn't listening at the time, it's stuck forever. The other message queue technologies supported by vice likely don't have this problem
In the case of Redis, they designed an algorithm specifically for this called RedLock. Which I'm going to try soon.
Thanks for the feedback! RedLock does look interesting, so let me know if you have some more insights.
https://github.com/matryer/vice Never happened in terms of nats integration for global concurrency use cases .
Instead Nats has this ability and it can be embedded. It’s just golang
the lock is just a kv in nats .
kine which is an etcd cluster also uses nats to a hieve global state.
Thanks, NATS does also look interesting on its own.
@Acconut
https://github.com/maxpert/marmot is a good starter.
It's designed to do data replication between SQLITE databases, but the concepts apply for tracking locks etc. It also embedded the NATS Server if you want.
The other way is that NATS has an Object Store, that can hold the chunks per user upload. When the upload is done, then you can extract it all out. https://github.com/a-h/natsjson uses the KV, which is a lot like using the Object Store.
Interesting links, thank you. Let us know if you are interested in trying out NATS for tusd's locks and we can assist you :)
I am interested because I want to use it with Pocketbase, so that uploads can be resilient
https://github.com/maxpert/marmot/issues/101 Is a brainstorm on doing it with NATS. It already does SQLite replication and file replication is easy to add.
The upload aspect is also handled there but I don’t know if the tusd protocol is conflating things tbh.
I am interested because I want to use it with Pocketbase, so that uploads can be resilient
Great idea!
The upload aspect is also handled there but I don’t know if the tusd protocol is conflating things tbh.
Not sure what you mean by this. Do you want to handle file uploads directly via NATS? If so, the tusd might not be necessary for you.
I am interested because I want to use it with Pocketbase, so that uploads can be resilient
Great idea!
The upload aspect is also handled there but I don’t know if the tusd protocol is conflating things tbh.
Not sure what you mean by this. Do you want to handle file uploads directly via NATS? If so, the tusd might not be necessary for you.
Yes you got it. NATS on the client side will chunk the file ( or any data) up to the NATS server. If the client goes down , when it comes back up it will ensure the file or data eventually gets to the NATS server. Once it gets to the Server it can then push it to Tusd or anything else. NATS in this case is acting as a Plane on top of Tusd.
NATS can currently used with Pocketbase to allow all data transacted to the Pocketbase SQLite DB to be scaled out to all others Pocketbase Sqlite servers. But its file uploads are scaled out.
Interesting approach, but this issue talks about a different use case for distributed systems. We want to look into how a distributed system can be used to lock upload resources and ensure exclusive access in the presence of multiple requests (see https://github.com/tus/tusd/blob/main/docs/locks.md). What you are proposing is something different AFAIU
thanks for the link @Acconut . I read it pretty quickly and it makes sense.
"the locks do not extend to every server.". You can use NATS to do this so all servers have full knowledge of any LOCK.
See k3 and Kine which uses NATS jet stream to do this also.
NATS will decouple things.
You know the problem domain much better than me however.....
Currently we only support memory and file locks, whose reach is limited to a single machine. If you want to properly scale horizontally, a distributed lock is needed, such as Redis, Consul, etcd etc.
@Murderlon proposed the use of https://github.com/matryer/vice, which wraps multiple messaging services into a single interface. Maybe that is helpful. More infos at https://medium.com/@matryer/introducing-vice-go-channels-across-many-machines-bcac1147d7e2.