Implement film grain

[kylophone]

https://github.com/xiph/rav1e/blob/5102093900ec6c0ef26aa1988c70e50eb5474d1e/src/header.rs#L757

Is anyone working on this? If not I'd like to take a crack. Don't know Rust yet, but I'm curious, and it might be a good introduction for me. Implementation would involve a de-noising of input pictures, and then estimating and signaling of grain parameters. Not sure if the current rav1e architecture makes this a more involved task or not. If anyone has a comment, I'm happy to hear it.

[negge]

Hi @kylophone. No, I don't think anyone is looking at implementing film grain yet. This should be an excellent way to get you introduced to rav1e and Rust, we're glad to have your help. The PR #1263 computes the activity mask over the input frame prior to encoding by accessing the luma plane pixel data through the FrameState. You would want to do something similar to de-noise and estimate the grain parameters before encoding the current frame.

I see you found us on IRC already. Don't hesitate to ask questions about rav1e or Rust there and we will do our best to answer them.

[dwbuiten]

You would want to do something similar to de-noise and estimate the grain parameters before encoding the current frame.

That depends. Is the plan to handle denoising and estimation inside rav1e, or outside? I think there are grain removal and modelling techniques that may be better than what is suitable for placement inside rav1e itself. There's also always the option to allow both, i.e. an 'auto' mode that does, and a mode that allows you to pass in either grain freq params, or your own grain tile + seed.

[kylophone]

That depends. Is the plan to handle denoising and estimation inside rav1e, or outside? I think there are grain removal and modelling techniques that may be better than what is suitable for placement inside rav1e itself. There's also always the option to allow both, i.e. an 'auto' mode that does, and a mode that allows you to pass in either grain freq params, or your own grain tile + seed.

Yeah, I think I'll aim for both usecases. I'll see what I can put together and provide WIP patch sometime soon.

[tdaede]

That depends. Is the plan to handle denoising and estimation inside rav1e, or outside? I think there are grain removal and modelling techniques that may be better than what is suitable for placement inside rav1e itself.

There are some pretty good temporal denoisers outside of rav1e, however rav1e already does a first pass quarter-res and half-res motion search that are an obvious candidate to get the most expensive part of denoising "for free".

I'd suggest splitting this up into 3 steps:

1. Signal manually specified noise parameters. You don't actually have to wire it up to the cli, but getting it to decode properly would be enough.
2. Compute an estimated noise by comparing the reconstructed frame and original frame. You don't actually have to noise filter for this, as encoding AV1 (especially at higher quantizers) already denoises a lot. Signal this estimated noise.
3. Actually implement a denoiser. In theory you can use the same function as (2) after the denoiser to figure out what to actually signal. Presumably it will naturally signal stronger values then.

[q3cpma]

Would be cool if an option to pass the denoised video (pipe or file) was possible. It'd allow some advanced denoising like BM3D or NLmeans via vapoursynth.

[Joachim-Otahal]

is there any codec built-in denoising? I am comparing v3.1 with q75/s4 against x264/veryslow/crf20/film. And I see denoising and a bit more blur on rav1e on Anime (no grain) and Movies (grainy) vs. x264. For x265 I see the same, but many encode with "no-intra-smoothing" to fix this, so maybe there is already some hidden switch in av1/rav1e to enable/disable? Apart from that: Very good progress!