lisamelton / video_transcoding

Tools to transcode, inspect and convert videos.
MIT License
2.39k stars 160 forks source link

Request for info: Encoding samples. #256

Open khaosx opened 5 years ago

khaosx commented 5 years ago

All-

If you wouldn't mind, please reply with your top two or three go-to videos / chapters that you use to test encoding strategies. I'd like to have a good sample set established when and if the wiki happens. If we build a community set then we can all speak a common language when testing new features. Let's see what we have in common, encoding geeks :)

My personal favorites:

  1. Chapter 1 - Guardians of the Galaxy vol 1 - SIGNIFICANT potential for color and grayscale banding when the ship illuminates the field.

  2. Any chapter from Clerks. It's film, it's black and white, it's super grainy by design.

  3. Chapter 1 - Star Trek III - It's my goto for testing forced subs

lisamelton commented 5 years ago

@khaosx What a great idea! I'll get you a whole list later after dinner. (It could take awhile, though. :) )

I didn't know about "Clerks (1994)" and "Star Trek III: The Search for Spock (1984)" because I stupidly don't have either.

This would also make great content for our future Wiki! (Once I get off my ass and start it.)

lisamelton commented 5 years ago

@khaosx OK, this is a partial list of the many Blu-ray movies and TV episodes that I use to test transcode-video, particularly my four ratecontrol systems and the --quick option.

Some of these titles I've transcoded not just dozens, but hundreds of times.

lisamelton commented 5 years ago

@khaosx Hopefully that was enough to start you off with. :) Let me know if you have questions or need clarifications on the descriptions.

I might be able to add more later if I can find the time.

samhutchins commented 5 years ago

@donmelton I don't think it was me who said anything about the great gatsby. I'm not sure I've even seen it to be honest...

@khaosx Planet Earth, S01E01. The opening sequence has a shot with a lot of birds. Like A lot of the example's Don gave, it's a challenging scene for compression, and it's interesting to see the effects of it in the scenes following

damorrison commented 5 years ago

Deepwater Horizon - first scene which is deep underwater watching the drill cap shift and bubble. Horrible banding and colour matching on almost all rips (even the Netflix version isn't great).

khaosx commented 5 years ago

This is a really good start! I'm taking all of them as they arrive and putting them in order and category. Like I said, I'm hoping to have them in a wiki-ready format later this week. Keep them coming folks!

lisamelton commented 5 years ago

@samhutchins D'oh! I thought that was you. Maybe it was @JMoVS in that long email thread I tortured you and him with last year? And, dang, now I'm going to have to invest in nature documentaries. :)

@damorrison Yikes! Does that color banding happen with the default ratecontrol system or one of my other ones? Looks like I'll have to buy that movie as well. :)

JMoVS commented 5 years ago

@donmelton yes, it was me with the gatsby confetti scene ;-)

lisamelton commented 5 years ago

@JMoVS Aha! The culprit identifies himself! :)

Seriously, that is one gnarly damn sequence to transcode. Great find, sir. Great find.

lisamelton commented 5 years ago

@JMoVS @khaosx I updated the credit in my previous comment, too!

khaosx commented 5 years ago

@donmelton Noted. I forked the repo today and started seperating the readme into component sections for the wiki. Once I get it done, I'll do a pull request so that it can be reviewed. (I think that's how it works, let me know if I'm wrong.)

lisamelton commented 5 years ago

@khaosx Thanks! And that's, more or less, how it works. :)

Keep in mind that we won't be moving the whole README into the Wiki. The "About," "Installation," "Rationale," "Usage," "Feedback," "Acknowledgements" and "License" sections will likely remain. Although "Rationale" could go either way.

But the "Guide," "Explanation" and "FAQ" sections are prime freakin' candidates for the Wiki.

I am concerned what happens to our SEO rankings when we move those. And you would think with my experience in Web browsers that I would know. But that wasn't my area of expertise. :/

Maybe somebody on the team will have some suggestions. We can hope.

Right now we're in the first page of results when you use Google to search for "video transcoding" and I don't want to lose that ranking.

Also, we'll probably move the "History" section into a separate file in the main project at the top level. Probably naming it something clever and exotic like "History.md" because, dammit, that's just how I roll. :)

I'll open an issue this week with some kind of rambling, coffee-addled plan for how we do this. Especially the organization of the Wiki. I've been looking at few on GitHub and to describe some of them as "organized" is being kind. Ouch.

We need to think ahead on where we put our support files like scripts, images, logs, media, etc. And we'll need at least some file/title nomenclature and style rules for the actual Wiki pages. Mostly because I'm an anal-retentive control freak. :)

Stay tuned!

vr8hub commented 5 years ago

I don't think you want to move that much information to the Wiki. The Wiki is for information not in the Readme.

My guess, if precedent holds, is that many more people are going to read the Readme than are going to read the Wiki. Given that, you don't want to move information you want most people to see. For example, the FAQ about VBR underflows should definitely be in the Readme, IMO. As should the note about not doing a GUI. (If you moved that, you would probably see a sudden increase in requests for one.) And so forth.

The Explanation section is about the only thing I would consider moving. I would keep most if not all of Guide, and the same of FAQ.

Things I'd like to see in the Wiki:

lisamelton commented 5 years ago

@vr8hub OK, I can see your point about the FAQ. Let's keep that in the README then.

But I would still like to move the Guide because it really needs to be expanded and I don't want to put all of that in the README. That seems to be perfect for a Wiki. And I can take that content and break it into separate pages easily. In a way, the Wiki should be a guide.

And your list of topics for the Wiki? Yep! I want to see all of those as well. Great ideas, sir.

I can probably create stubs for most of those topics when I do the initial setup.

Right now I'm working on the structure, style guide, contributors checklist, code of conduct, license and a README. Yes, it will have its own README. :) Not a long one, mind you. Mostly a statement of purpose and a pointer to the front page of the Wiki itself.

I know everyone is raring to go, but I've been involved in Open Source projects for over 25 years and you have to get this shit right the first time. Otherwise wars can start. :)

Anyway I'm typing as fast as I can but life, even retired life, keeps getting in the way.

vr8hub commented 5 years ago

But I would still like to move the Guide because it really needs to be expanded and I don't want to put all of that in the README.

Right. You don't put all of that in the README. You put the expanded parts in the Wiki. But (much of) the Guide should stay in the README, IMHBAO. They are things everyone should see, not just the over-committed nerds enthusiasts who will read the Wiki. Notice I said "a much fuller discussion of your four steps". The four steps should still be in the README; that's good information for everyone. Want to know more? Read the Wiki.

Get this stuff right, or a war might start.

lisamelton commented 5 years ago

@vr8hub OK, I see what you mean now. Good point!

And, BTW, there's no air between nerd and enthusiast anymore. :)

vr8hub commented 5 years ago

I was being kind to the enthusiasts who haven't figured that out yet. (I said something in passing in Sunday School one time about Galaxy Quest being a takeoff of Star Trek. "It was not!" said the guy with a three-foot diameter Enterprise in his front yard.)

lisamelton commented 5 years ago

@vr8hub Apparently too kind to them! :)

khaosx commented 5 years ago

I'm standing down on working with the wiki until we come up with guidelines / mission, but I agree with @vr8hub. The readme should be "this is how things are", and the Wiki should be "Here's why things are the way they are". Far too many projects (especially ones as feature-rich as this one) are fantastic at the former and neglect the latter. @donmelton has done a great job balancing both to this point, but splitting them into two distinct entities (where Don maintains complete and absolute autocratic control over the readme, and where a trusted group of collaborators can assist with the wiki) will serve the community better going forward and take some of the pressure off of Don. The folks who help maintain the wiki will be in a better position to jump in and help with questions and issues by virtue of their involvement with their sections of the wiki.

Win / Win!

JMoVS commented 5 years ago

@donmelton Yeah that confetti scene was me like "OMG, wow, did they record this digitially on source? This must be horror for any encoding algorithm" :D

about history: As you tag the releases, you could make it the following way:

compare that to what you see for transcoding

This way we could retain history, directly associate with the relevant tags and remove it from the readme and maybe only leave a hint in the readme "go to releases to see the history of new features for each release"

lisamelton commented 5 years ago

@khaosx @vr8hub ...

The readme should be "this is how things are", and the Wiki should be "Here's why things are the way they are".

That's great! I may use that line in the README for the Wiki!

khaosx commented 5 years ago

@donmelton @vr8hub ...

That's great! I may use that line in the README for the Wiki!

And for a brief, shining second, I thought "He's going to write a readme. For the Wiki. The one that expands on the already voluptuous readme. If this doesn't kill us, it will surely break our hearts."

And then I thought "No way. I'm just not great at reading."

And then I remembered that you mentioned making the Wiki a separate repo, and now I am afraid again. I'm going to go get my kids and then go cry in the shower with a big glass of wine.

lisamelton commented 5 years ago

@JMoVS I've been meaning to see if "Prometheus" is available on one of the streaming services just to see how their transcoding looks. :)

after you created the tag, go to "releases" and fill in the history of what has changed there.

What magic is this?!? :) Seriously, I never noticed that. Can I go back and just copy the "History" snippets for the old releases into the old tags?

Anyway, I may try that for the upcoming release. Thanks!

lisamelton commented 5 years ago

@khaosx LOL! :) It will be a very, very small README. Relax. But have the wine anyway. :) And, yes, it will be a separate repo because that's the only I can grant all of your access without going insane.

JMoVS commented 5 years ago

@donmelton Yes, mark some tags as releases and then while doing that you can fill that in. I think I told you about this 1 or 2 years ago once or twice, but it seems like I failed to communicate the idea. :P

lisamelton commented 5 years ago

@JMoVS I'm just slow to learn. That should be obvious by now. :)

JMoVS commented 5 years ago

@donmelton Haha no worries.

BTW: The wiki could be in the main repo and just maintained by the community without you having to give us write rights I think.

https://help.github.com/en/articles/changing-access-permissions-for-wikis

You'd retain ownership and coontrol over the code, we could work and add to the wiki

JMoVS commented 5 years ago

@donmelton BTW: Nice benefit if you add the info to the release tags is: People who watch or subscribed to the releases rss will get meaningful release notes! :D

damorrison commented 5 years ago

I was having some thoughts around the Wiki and what I'd want to get out of it, so poured a glass of wine (or two) and typed some words.. here they are.

Are you transcoding brand new from source, or re-transcoding something old? This is 'what's your goal' question. If direct from the source, then you'll care more to get it as good as you can. I put BR-rips in this category. I want the best visual and audio I can get for the devices I'm using. If re-transcoding, then good-enough will probably be OK. For example, I'll re-transcode my whole TV library into General, HQ and Kids as below. Two reasons, one so that they'll never need to be transcoded on the fly, and also to save space.

What devices will you watch on? Can labels be created in the Wiki for devices, or will this get out of hand? i.e. if you write a section which is relevant to an iPad owner, then label relevant sections with an 'iPad' label in the Wiki. Same goes for Roku, 4K, etc. Multiple tags would be added to each section.

For me, I have the following: • 4K smart tv with Sonos 5.1 running native Samsung TV Plex app. • Plex server on a Synology (mid power DS415+) • iPad - rarely use for transcoded watching as tend to use Netflix downloads
• Kindle fire For kids shows

I'd like to know what characteristics these devices have from a transcoding perspective.

What native video/audio format's do those devices have?

Here we could document what the native capabilities for each of these devices are and what the command line options are for each. For me, I don't want my Plex server to have to transcode anything on the fly, as it's not super powerful, and will buffer, so I want to make sure my converted files fit the native format of the device I want to play them on.

What sources do you have? For me it's:

  1. TV Shows. I'd split these into categories. For me it's:

    • General TV - regular TV shows with a stereo audio track. I like decent picture quality, but 9/10 a simple audio track will be fine
    • HQ TV - some of the shows I have, like Game of Thrones etc. have an immersive soundtrack, and I'd want to keep those 5.1 channels intact
    • Kids TV - my two-year-old doesn't care if his Sarah and Duck episodes are in 5.1, so probably lower quality and a basic stereo track is fine.
  2. Movies (again split into a couple of categories)

    • Downloaded - generally could be in any format with any codec, in any quality and with any/multiple sound tracks. Most difficult to tackle for me to get a consistent library
    • DVD rips format varies but generally good quality and audio
    • Blu-ray rips will always have a Dolby and a 5.1 channel

Given all the above, I'd like to come away armed with the command line switches relevant to what I want to achieve, and some automation hints on how to do them in batch.

lisamelton commented 5 years ago

@JMoVS And to continue documenting the many senior moments I've had today, when I typed "Prometheus" earlier, I meant "The Great Gatsby" instead. D'oh!

lisamelton commented 5 years ago

@JMoVS ...

You'd retain ownership and coontrol over the code, we could work and add to the wiki

Yeah, I've read that before. But, as near as I can tell, granting anyone "collaborator" status grants them access to change the code as well. GitHub doesn't seem to distinguish between the two. And I'm not keen on granting any GitHub user the right to edit the Wiki. There's open and then there's too open.

But when I open the issue with my proposal, y'all can talk me into making it more open.

lisamelton commented 5 years ago

@JMoVS ...

BTW: Nice benefit if you add the info to the release tags is: People who watch or subscribed to the releases rss will get meaningful release notes! :D

Stupid me. I didn't realize you could subscribe via RSS. :)

lisamelton commented 5 years ago

@damorrison Interesting! Keep these notes around somewhere!

vr8hub commented 5 years ago

@donmelton Yeah, I've read that before. But, as near as I can tell, granting anyone "collaborator" status grants them access to change the code as well. GitHub doesn't seem to distinguish between the two. And I'm not keen on granting any GitHub user the right to edit the Wiki. There's open and then there's too open.

Right, he was saying use collaborator for the code and anyone for the Wiki (which would keep the Wiki people out of the code). But I agree with you — I don't think you want "anyone" to be able to edit the Wiki. (Me, for example: I'm liable to tell everyone to use hardware encoders. Or name their multi-disc files gerbil*.) And that looks like it will require another repository, unfortunate as that is.

lisamelton commented 5 years ago

@vr8hub @JMoVS Yes, it is unfortunate but it's not an uncommon solution to the problem. For example, this is exactly what the Ethereum project did:

https://github.com/ethereum/wiki

Which makes the link to their actual wiki in that project kind of amusing :)

https://github.com/ethereum/wiki/wiki


Edit: No idea why the GitHub editor added a "thumbs up" after that smiley emoticon. Weird.

JMoVS commented 5 years ago

I think it is this way because it is assumed that the project owner sets up the basic structure of the wiki and then we work on one level down

lisamelton commented 5 years ago

@JMoVS Yes, although technically anyone granted access to the actual wiki pages has access to the top-level "real" project. Which is a good thing since I plan to create directories like "images" and "scripts" at that top level where contributors can upload files which will then be linked to from the wiki pages.

JMoVS commented 5 years ago

I don‘t understand.

If I read it correctly, you don‘t have to grant anybody specific rights to be able to contribute to the wiki. That is only the case IF you enable „limit to collaborators“.

If you don‘t enable that, my understanding was you don‘t have to grant anybody anything - or am I mistaken?

vr8hub commented 5 years ago

If I read it correctly, you don‘t have to grant anybody specific rights to be able to contribute to the wiki.

Right, but then anyone can edit the wiki. He doesn't want "anyone" to be able to edit the wiki, he wants to limit it. But he wants a wider group of people that can edit the wiki than can contribute to the code. One repository can't have two different contributor lists. Hence two repositories.

lisamelton commented 5 years ago

@JMoVS If you enable "Restrict editing to collaborators only " then only collaborators can create and edit project wiki pages. If you don't enable that setting, then any GitHub user can create and edit project wiki pages. I don't want just any GitHub user to create and edit project wiki pages. So I plan to leave that enabled on the new project that I create. Which means I'll have to manually add collaborators.

lisamelton commented 5 years ago

@vr8hub Well put! Yes, that's it exactly.

JMoVS commented 5 years ago

But why would you not allow any github user to edit the wiki? It‘s a wiki after all with active maintainers, why do you want to vet the wiki-contributors specifically?

If you create a new repo, consider doing an „organisation“ so it is bundled eg under github.com/video_transcoding/x

JMoVS commented 5 years ago

Just read @vr8hub‘s comment: I honestly don‘t get why we wouldn‘t try the „anyone can edit the wiki“ option first. We can still move it to another repo if it needs to be limited. But if we limit it from the get go, we will put up a hurdle for people who might want to contribute but don‘t want to jump through access right hoops

skj-dev commented 5 years ago

This may be crazy talk (/me scans the room ... maybe not a problem 😈), but would it be feasible to make the "wiki" a GitHub Pages (gh-pages) branch? Updates could come in as regular pull requests. Its possible I'm asking why there isn't any ketchup at a Killer Tomatoes cookout, but hey, what's the worst that could happen?

vr8hub commented 5 years ago

@ttyS0 This may be crazy talk (/me scans the room ... maybe not a problem 😈)

Finally, someone gets it.

lisamelton commented 5 years ago

@JMoVS Sorry, I went to bed last night before I saw your last comment!

These are good questions. And I have considered both keeping the Wiki in the current project and allowing access to all GitHub users.

But here's why I would rather not do either of those things. :)

While the Ethereum wiki that I linked to yesterday allows access to all GitHub users, it's clear that their biggest problem, which they identify first in their contribution guidelines, is fixing vandalism.

I do not want to spend my time or our contributors time fixing vandalism.

And vandalism is inevitable because 1) just a GitHub account is too low of a bar to allow entry and 2) this is the Internet we're talking about. :) Trolls, self promoters, con artists and other random assholes are large enough group, even here on GitHub, to be wary of a completely egalitarian strategy no matter how noble that sounds.

There's only one way I would have to stop vandalism if it gets out of hand and that is to restrict access to collaborators only. And if that's likely, why not just start out that way?

The downside of gating access is that it could discourage contribution. This is a real concern. But, then again, it could make more thoughtful contribution likely. And when you're a contributor and you know who your peers are, then I think you're also more likely to step up the quality of your contribution. It's simple social engineering.

There's also the downside that gating access could create an elitist tier for contribution. I am concerned about that. But I think if you've already made a contribution here, with a pull request, feature idea, thoughtful or frequent comment, etc., then I plan to be fairly liberal granting access. Plus, I plan to spell out in the guidelines that you can always ask for access by opening an issue with your idea. Which could be as simple as, "Hey, can I fix some of the formatting problems that I see?"

And as you well know, I might be a control freak but it's pretty easy to talk me into something. :)

If we spell out a code of conduct ahead of time, then policing contribution is simple. Vandalizing the Wiki (or other actions like bullying and such) means that your changes are backed out and your access is revoked.

I don't want to be a totalitarian about this but I've been working in Open Source projects for over 25 years so I know that some level of control is needed or you just have anarchy.

And I'll have some more thoughts to add here later after I walk the dog. :) Apologies.

JMoVS commented 5 years ago

@donmelton Better so! Get all the sleep you need so we can enjoy high quality transcoding!

First I think that I would propose giving the community the benefit of the doubt and that I don't think we need a code of conduct.

I looked more into the wiki-issue and I'm ever more convinced we should do the following:

  1. KISS: Keep It Short and Simple
  2. Avoid the wiki issue by building a documentation repo, but don't use the github wiki feature - as it would be lingering in the void
  3. For the documentation, use ReStructured Text using the Sphinx framework and link to the page from the main donmelton/video_transcoding repo - a nice example would be: https://github.com/zrepl/zrepl and the directly linked documentation there

This way we:

  1. get a nicely formatted documentation
  2. With a nice "fork me on Github and improve me" link on the documentation
  3. The documentation/tipps&tricks aren't lingering in a wiki in a separate repository
  4. Change requests are by definition simple pull requests to the repo of the github page, hence everybody with a github account can propose changes and you/we still have control
  5. I personally don't see a need for a code of conduct then either - we didn't need one in the last x-years here either and I am at least unaware of misbehaviour from contributors our yours truly, so 👍🏼

Optionally you could grant collaborators/people from here access for the documentation page repo, but eg configure it that PR's need to be reviewed by you but we can reduce the workload by managing and guiding people there.

If we go for that solution, I think it's time to create a github organisation then - I'm wondering though if you're gonna call it "donmelton" organisation then haha - just to bundle those 2 repos sensibely.

What do you think of this proposal? ;-)

lisamelton commented 5 years ago

@JMoVS That actually seems more complicated to me. :)

I like the Wiki facility in GitHub because it makes the editing process very simple for even non-technical users, i.e. they can just use a Web browser to contribute and change content.

Also, while my vision for this Wiki is, of course, to start by extending the documentation for the current project, I hope that it eventually becomes a knowledge resource for non-professional video transcoders, perhaps even documenting other tools. And that it contains information about the ripping process before you transcode as well as serving up your content afterwards.

So that seems to me like it should be a separate project from this one. That also solves the problem of content migration should access problems require it since, well, it will already be elsewhere. :)

And you should never break links if you can possibly avoid it.

Also, we could start with non-contributor access (i.e. anyone with a GitHub account) to the Wiki as you originally suggested and see how my fears about vandalism play out. I'm willing to give it a shot if y'all are willing to help me fix shit if breaks. :)

But if we start that Wiki here in this project, then my options are limited.

And by starting small with the Wiki facility we could then decide to expand it using the documentation tools and process as you suggest or GitHub pages as @ttyS0 suggested. (BTW, Sean, that's not a crazy idea! I will investigate it.)

BTW, these days a code of conduct is a necessity in the brutal post-2016 world of the Internet.

And we should consider a license as well. Something very liberal, possibly even more public domain than my current projects are. Which is another good reason to keep them separate.

lisamelton commented 5 years ago

@JMoVS BTW, I realize we don't have a code of conduct in place here, but this project is very different than what I envision for the collaboration in the future Wiki/Knowledge Base project.

lisamelton commented 5 years ago

@JMoVS And this is what I envision using as the basis of our code of conduct:

https://www.contributor-covenant.org/

It's pretty much the gold standard for these things.