[Meta] Distributed encoding

[master-of-zen]

Issue for tracking the progress and milestones than need to achieved for distributed encoding with av1an.

As i understand server-client model is best way forward.

Here are the points/steps so far:

• [x] Make Chunk the basic unit distributed data. It should contain all information (aside from the video) that is required to start encode on client side.
• [ ] Decouple progress bar from broker, so broker have channel that returns progress made on encoding chunks, which later could be used for multiple methods of tracking progress.
• [ ] Make broker have dynamic resizable worker queue. Be able to add/suspend/delete chunks from the queue.

[lastrosade]

The second part of this list makes me think this would solve #220

[zerefel]

Hello, I'm interested in getting the encoding progress and other metadata from the process programatically. This is the only issue I could find mentioning something like that.

You may hate me for asking this but is there an ETA when we'll be able to tap into this data? I can't figure it out in the current release.

[gianni-rosato]

@zerefel As in printing the number of frames encoded over the total number of frames to compute progress for yourself, or something similar?

[zerefel]

@gianni-rosato What I mean is the ability to get the raw progress data and be able to manipulate it more easily. For example -- a log file which includes frames encoded/total frames and/or other measures which can then be parsed by a script. Imagine the av1an docker container running alongside another process which is responsible for communicating the progress to other services.

[gianni-rosato]

https://github.com/gianni-rosato/av1an-progbar

My fork outputs frames encoded/total frames. I'm sure you could pipe to a log file if you wanted. This might be a good place to start versus vanilla Av1an if you want to dive in and make some changes