aomenc buffer overflow

[Plamper]

Describe the bug

When encoding a video with av1an the default aomenc encoder always crashes with a bufferflow error. SVT-AV1 works fine with the same file.

WARN [av1an_core::broker] Encoder failed (on chunk 0):
encoder crashed: signal: 6 (SIGABRT) (core dumped)
stdout:

stderr:
        Warning: automatically updating bit depth to 10 to match input format.
        *** buffer overflow detected ***: terminated

Steps To Reproduce

Steps to reproduce the behavior:

1. open nix shell on commit hash 5e0ca22929f3342b19569b21b2f3462f053e497b, because of Rust 1.80 issues
2. av1an -i <any_video_file> -o test.mkv
3. Wait through scene analysis and watch it break when starting encoding.

Expected behavior

The video is encoded without issue.

Additional context

I tried to reproduce it in an arch linux container, but the arch package works without issue. So it seems this is a nix issue. Aomenc itself works fine with y4m generated ffrom ffmpeg.

Notify maintainers

@getchoo

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.10.3-xanmod1, NixOS, 24.05 (Uakari), 24.05.20240806.21cc704`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.5`
 - nixpkgs: `/nix/store/pi3i4rcxl8iadwxpyyx481ycnyxm856b-source`

---

Add a :+1: reaction to issues you find important.

[getchoo]

I'd imagine this is an issue with our build of libaom itself (or an upstream bug) rather than av1an, especially since you mention other encoders work fine and the log here is also blaming the encoder

[Plamper]

I did some further testing and can reproduce the buffer overflow just with aomenc, by just inputting a webm file like so aomenc test.webm -h 2160 -w 3840 -o test_out.webm. This is especially strange as aomenc is not intended to be used that way. You are normally supposed to input y4m files. However, the aomenc inside this arch container will do it anyway, while the nix version crashes with the buffer overflow.

[Plamper]

I should probably add libaom maintainers as well. @primeos @kiloreux @dali99