search

YourFin / transcodebot

Distributed multiplatform video transcoding for consumer hardware 🎞🤖
MIT License
1 stars 0 forks source link

Interesting Project to Follow #1

Open tony199555 opened 6 years ago

tony199555 commented 6 years ago

First star ever on this project. Hoping you could keep the vibe high.

Few questions though:

  1. Are files get pushed around in different machines?
  2. Are they able to segmented into chunks?
  3. How efficient is the system?
  4. Is it on Linux system?

Few suggestions/non-sense:

  1. The first sentence on the readme is confusing. What does it mean "...have other jobs, like being workstation..."? It would be better to get the readme revised.
  2. TBA

Wish you the best of luck!

YourFin commented 6 years ago

Wow I'm kinda surprised you found this; out of curiosity, what led you here? Most of this is an attempt to replace this janky ruby script and the GNU parallel solution to distribution to something that can be put on random computers I have lying around (read: windows) with no additional dependencies.

As for the questions:

  1. Not totally sure what you mean, but yeah the plan is to transfer files back and forth over the network
  2. Totally possible, but not a major goal right now though. I have about 14tb of video to run through so it'll be a negligible difference for my use case. Help would be welcome though ;) Go is a really easy language to learn if you don't know it already.
  3. No clue; although the main bottlenecks are ffmpeg/network speed, so I'm not too worried about the speed of the wrapper so much as just trying to get at the most hardware
  4. Server has to be linux (windows and OSx will probably work too, but I'm not testing it). The client/transcoder boxes can be any OS that ffmpeg is complied for though.

The readme: I'm somewhat more interested in having something to have a readme about. The program does nothing useful at the moment. Pull requests on the matter are welcome though

tony199555 commented 6 years ago

Oh, about how to found this... most recent updated sorting XD, and you are the first one on the list. Click, saw some interesting progress, and just want to show some support to a fellow who embraced the OSS world.

I was looking for a solution that could segment a video and get it transcoded on different computers (FFmpeg, handbrake, avconv, etc) and combined them back once done. I saw a few other projects but not really what I expected (only 1 codec or 1 container type or just simply too old for this world).

The main reason I am looking for such solution is that I am experimenting video encoded to the new open codec "AV1", but it is extremely, extremely slow (~10fps encoding @ 480p, link https://github.com/xiph/rav1e). However the bitmovin is using some solutions to get it encoding @ sub-realtime, so... thousands of times faster.

Regarding the help I could provide... I know a bit about Go, but don't think it is good enough to participate in a project yet.

I think it might be creepy to just get someone randomly posting things on your project, so it is totally fine to ignore me if you think that is for the best.

Good luck.

YourFin commented 6 years ago

Oi I had thought that rav1e was a bit faster than that. I've been wanting to try messing with av1 but was mildly discouraged after ffox claimed to have support in the newer stable releases but couldn't play that bitmovin sample video. Let me know if you figure out how people do video partitioning for these distributed things, I haven't seen anything other than just using ffmpeg to make mini videos, transcoding those, and then stitching them back together.

Don't worry about being a creep, the project wouldn't be online if I didn't want people to drop by

tony199555 commented 6 years ago

So after 2 hours of scratching the web, I think what bitmovin did is already in the streamline. They use the MPEG-DASH/HLS method plus some proprietary tech (as of lately... AI Deep Learning Video Analysis to improve speed and quality). Based on their webpage (1), the video is still split into chunks like FFMpeg except with DASH, you get a unique file called MPD which keeps track of all of the files that got generated throughout the process, even it is multi-codec (2). On a coop Github, they showcase the what they have for DASH encoding with "...the variation of the segment size, bitrate, resolution, encoding settings, URL, etc..."(3) The good thing about DASH is that it keeps the GOP and other important stuff intact (or at least that is what I would hope/think). Lastly, I have found out FFMpeg actually natively support DASH MPD generation. You could try out with the accepted answer on this page(4).

Now the questions remain: how to get FFmpeg to transcode/segment fast and steady? (Take less time than just jam the whole file) How to assign segments to different machines and still keep track of it? How to stitch the transcoded segments back to a full file, given the metadata has lost in the process?

Reference:

  1. https://goo.gl/Er8E5P 2.https://arxiv.org/pdf/1803.06874.pdf 3.https://github.com/slederer/DASHEncoder 4.https://stackoverflow.com/questions/48256686/how-to-create-multi-bit-rate-dash-content-using-ffmpeg-dash-muxer

P.S. Firefox playback works for me. Also, I have tried Chrome Canary v69 which also works. What system /hardware have you tried?

P.P.S. While I was writing, I was wondering if all of those are still within the intended scope of your project. But I mean, it is not bad to just talk about it, right?

P.P.P.S After I finished writing, I have noticed on the (1), they have the AV1 as "ABR encoded content", which means they don't stitch it back, but just deliver them as DASH via CDN.