Segmentation fault when encoding sequence with alpha channel and AOM_USAGE_REALTIME

[fdintino]

How to reproduce:

Find an RGBA png and pass it in twice to avifenc to create a sequence, using aom encoding with speed 7 or greater:

avifenc --speed 8  in.png in.png out.avif

Successfully loaded: in.png
AVIF to be written: (Lossy)
 * Resolution     : 326x268
 * Bit Depth      : 8
 * Format         : YUV444
 * Alpha          : Not premultiplied
 * Range          : Full
 * Color Primaries: 2
 * Transfer Char. : 2
 * Matrix Coeffs. : 6
 * ICC Profile    : Present (544 bytes)
 * XMP Metadata   : Present (439 bytes)
 * Exif Metadata  : Absent
 * Transformations: None
 * Progressive    : Unavailable
Encoding with AV1 codec 'aom' speed [8], color QP [24 (Medium) <-> 26 (Medium)], alpha QP [0 (Lossless) <-> 0 (Lossless)], tileRowsLog2 [0], tileColsLog2 [0], 1 worker thread(s), please wait...
 * Encoding frame 1 [1/30 ts]: in.png
 * Encoding frame 1 [1/30 ts]: in.png
Segmentation fault: 11

And here is the backtrace in gdb:

Program received signal SIGSEGV, Segmentation fault.
0x00005630978e5f7b in aom_sad16x16_sse2.loop ()
(gdb) bt
#0  0x00005630978e5f7b in aom_sad16x16_sse2.loop ()
#1  0x0000563097bc6fdb in set_color_sensitivity (cpi=0x7fd86ef9a020, x=0x7fd86efaf030, bsize=BLOCK_32X32, y_sad=0, source_variance=0, yv12_mb=0x7ffca46eca10)
    at ../av1/encoder/nonrd_pickmode.c:2484
#2  0x0000563097bc9504 in av1_nonrd_pick_inter_mode_sb (cpi=0x7fd86ef9a020, tile_data=0x56309a30c5a0, x=0x7fd86efaf030, rd_cost=0x7ffca46eda10, bsize=BLOCK_32X32, ctx=0x56309a32ecf0)
    at ../av1/encoder/nonrd_pickmode.c:3135
#3  0x0000563097ba33f2 in pick_sb_modes_nonrd (cpi=0x7fd86ef9a020, tile_data=0x56309a30c5a0, x=0x7fd86efaf030, mi_row=0, mi_col=0, rd_cost=0x7ffca46eda10, bsize=BLOCK_32X32,
    ctx=0x56309a32ecf0) at ../av1/encoder/partition_search.c:2249
#4  0x0000563097ba4bed in av1_nonrd_use_partition (cpi=0x7fd86ef9a020, td=0x7fd86efaf030, tile_data=0x56309a30c5a0, mib=0x56309a2e0690, tp=0x7ffca46edfe8, mi_row=0, mi_col=0,
    bsize=BLOCK_32X32, pc_tree=0x56309a32e870) at ../av1/encoder/partition_search.c:2610
#5  0x0000563097ba5930 in av1_nonrd_use_partition (cpi=0x7fd86ef9a020, td=0x7fd86efaf030, tile_data=0x56309a30c5a0, mib=0x56309a2e0690, tp=0x7ffca46edfe8, mi_row=0, mi_col=0,
    bsize=BLOCK_64X64, pc_tree=0x56309a32e990) at ../av1/encoder/partition_search.c:2776
#6  0x0000563097b62078 in encode_nonrd_sb (cpi=0x7fd86ef9a020, td=0x7fd86efaf030, tile_data=0x56309a30c5a0, tp=0x7ffca46edfe8, mi_row=0, mi_col=0, seg_skip=0)
    at ../av1/encoder/encodeframe.c:542
#7  0x0000563097b630ca in encode_sb_row (cpi=0x7fd86ef9a020, td=0x7fd86efaf030, tile_data=0x56309a30c5a0, mi_row=0, tp=0x7ffca46edfe8) at ../av1/encoder/encodeframe.c:931
#8  0x0000563097b63a04 in av1_encode_sb_row (cpi=0x7fd86ef9a020, td=0x7fd86efaf030, tile_row=0, tile_col=0, mi_row=0) at ../av1/encoder/encodeframe.c:1103
#9  0x0000563097b63b7d in av1_encode_tile (cpi=0x7fd86ef9a020, td=0x7fd86efaf030, tile_row=0, tile_col=0) at ../av1/encoder/encodeframe.c:1136
#10 0x0000563097b63d63 in encode_tiles (cpi=0x7fd86ef9a020) at ../av1/encoder/encodeframe.c:1173
#11 0x0000563097b6546d in encode_frame_internal (cpi=0x7fd86ef9a020) at ../av1/encoder/encodeframe.c:1658
#12 0x0000563097b661f7 in av1_encode_frame (cpi=0x7fd86ef9a020) at ../av1/encoder/encodeframe.c:1979
#13 0x000056309766c9a8 in encode_without_recode (cpi=0x7fd86ef9a020) at ../av1/encoder/encoder.c:2498
#14 0x000056309766d536 in encode_with_recode_loop_and_filter (cpi=0x7fd86ef9a020, size=0x7ffca46ee708, dest=0x7fd86eea8010 "\022", sse=0x0, rate=0x0, largest_tile_id=0x7ffca46ee544)
    at ../av1/encoder/encoder.c:2966
#15 0x000056309766f433 in encode_frame_to_data_rate (cpi=0x7fd86ef9a020, size=0x7ffca46ee708, dest=0x7fd86eea8010 "\022") at ../av1/encoder/encoder.c:3635
#16 0x000056309766fadc in av1_encode (cpi=0x7fd86ef9a020, dest=0x7fd86eea8010 "\022", frame_input=0x7ffca46ee760, frame_params=0x7ffca46ee710, frame_results=0x7ffca46ee708)
    at ../av1/encoder/encoder.c:3787
#17 0x0000563097b7e2a9 in av1_encode_strategy (cpi=0x7fd86ef9a020, size=0x7ffca46eeab0, dest=0x7fd86eea8010 "\022", frame_flags=0x7ffca46eeab8, time_stamp=0x7ffca46eeac0,
    time_end=0x7ffca46eeac8, timestamp_ratio=0x56309a1dfc68, pop_lookahead=0x7ffca46eeae0, flush=0) at ../av1/encoder/encode_strategy.c:1580
#18 0x00005630976709fb in av1_get_compressed_data (cpi=0x7fd86ef9a020, cpi_data=0x7ffca46eeaa0) at ../av1/encoder/encoder.c:4489
#19 0x00005630976300bd in encoder_encode (ctx=0x56309a1df5d0, img=0x7ffca46eec80, pts=0, duration=1, enc_flags=0) at ../av1/av1_cx_iface.c:2995
#20 0x0000563097786455 in aom_codec_encode (ctx=0x56309a06bbf8, img=0x7ffca46eec80, pts=0, duration=1, flags=0) at ../aom/src/aom_encoder.c:176
#21 0x00005630975e4ee2 in aomCodecEncodeImage (codec=0x56309a06bb50, encoder=0x56309a06a310, image=0x56309a207f00, alpha=1, tileRowsLog2=0, tileColsLog2=0, encoderChanges=0,
    addImageFlags=0, output=0x56309a06bae0) at /tmp/libavif/src/codec_aom.c:977
#22 0x00005630975de025 in avifEncoderAddImageInternal (encoder=0x56309a06a310, gridCols=1, gridRows=1, cellImages=0x7ffca46eefc0, durationInTimescales=1, addImageFlags=0)
    at /tmp/libavif/src/write.c:1145
#23 0x00005630975de12b in avifEncoderAddImage (encoder=0x56309a06a310, image=0x56309a207f00, durationInTimescales=1, addImageFlags=0) at /tmp/libavif/src/write.c:1175
#24 0x00005630975c0032 in main (argc=6, argv=0x7ffca46ef538) at /tmp/libavif/apps/avifenc.c:1274

If I make it so that it never passes aomUsage of AOM_USAGE_REALTIME by deleting these lines:

https://github.com/AOMediaCodec/libavif/blob/607998755a507b97e9ad4b2a2735cdeb3b1f85ba/src/codec_aom.c#L560-L568

I no longer get a segmentation fault.

I suspect this is a bug in libaom, but it came to my attention because I had CI test failures in pillow-avif-plugin when updating to the latest libavif.

[jzern]

Thanks for the report, this does sound like a libaom issue. Do you see the issue with all content or just a particular image? Could you attach a sample source file?

If you're using the libavif release, then you picked up libaom 3.5.0, correct?

[jzern]

And could you confirm the platform you're seeing the crash on? I see some Ubuntu and mingw-w64 targets in your workflows.

[fdintino]

So far as I can tell, any RGBA image whatsoever triggers it. If you're looking for a specific image, try this one. The "wheels" workflow of pillow-avif-plugin tests against the widest variety of platforms; I've added a test that should fail with this bug so that I can get a full list of platforms that have the issue. (the windows wheels are still not built against the latest libavif and aom, so ignore those for now). I will report back with the results once it finishes running. So far in my local testing I've seen it on every x86 platform I've tried (alpine, ubuntu, centos, and OS X). I can't be sure, but I think that the M1 OS X builds didn't exhibit the issue.

[wantehchang]

Frankie: Thank you for the bug report.

James: Please check if the stack trace shows the crash occurs when we encode the alpha plane. Could you check if the libaom functions in the stack can handle a monochrome image? Thanks.

[jzern]

I submitted a couple of fixes, one found by the source file and the other when adding a test. I didn't do an exhaustive search, so there still may be others. @fdintino if you get a chance, please give the new version a try.

[fdintino]

I can confirm that this fixes the issue I was having. Thanks!

[jzern]

Great, thanks for the update. If you run into any other issues, don't hesitate to reach out.