[Feature/Question] Client (not cli) and Rate Limiting

[bearrito]

Two questions,

1. Has anyone in the community create clients for the CLI. The CLI is generally useful but a few things make it difficult to use for our use case.

2. Would anyone else find it a useful feature to implement rate limiting in the client? Would the maintainers want a feature like this? I think something like this would need to go in Assemble but haven't look much at it. For our use case, this is because our clients need to guarantee command and control traffic is prioritized, we need to trickle in chunks slowly over time. It does look like it's possible to throttle in some ways, with some notion of workers (not documented elsewhere) as far as I could see.

   // Start the workers, each having its own filehandle to write concurrently
   for i := 0; i < options.N; i++ {

[folbricht]

1. Is the question if there exists another cli implementation for desync? I'm not aware of any. It's written in a way that it could be though. What are the things that make the cli difficult to use?

2. Implementing rate-limiting would be quite straightforward, however, adding it the command-line in a simple way is not. To implement such a feature, there's no need to go into the low-level functions that assemble the output files. This could be done much more cleanly by introducing something like a RateLimitedStore object which implements the Store and possibly the WriteStore interface (see https://github.com/folbricht/desync/blob/master/store.go) and wraps another store. This way the reading (and writing) of chunks from/to the store could be slowed down. And it would work with any kind of store really. But again, how to express which store should be limited and by how much in the command-line would require some careful thought.

[bearrito]

@folbricht

Thanks for the feedback.

First off there is nothing wrong with CLI, our use case is just particular.

1. Using the CLI requires shelling out to a subprocess and parsing results/codes etc. Totally doable but not as clean as we might want. It's always possible to create a library on top of subprocess calls that hides that, that's what I do today.
2. Getting feedback is important to our command and control system. Using the subprocess method, I'm not aware of how to parse the visual/graphic progress bar. Maybe there is a way to do this...

Rate limiting

Thanks, I probably won't look at that much at the moment, because we are happy with the current approach for testing purposes, but will likely be something I could contribute too in a few weeks.

[folbricht]

Implementing your own client should be quite easy actually since it's designed for that. And if you implement a RatelimitStore wrapper that could then be used directly, without figuring out how to use it from the existing cli.

As for parsable progress, have you looked if setting DESYNC_ENABLE_PARSABLE_PROGRESS=1 in the environment helps you at all?

[bearrito]

I haven't looked at that but I'll give it a shot.