Open malaterre opened 8 months ago
Thanks!
There was a request to enable libjxl for Gentoo on 32-bit PowerPC: https://bugs.gentoo.org/936241
The failed tests are the blocker there too.
At least following tests are failing on ppc32 with libjxl-0.10.3:
3829/4841 Test #3829: EncoderStreamingTest/EncoderStreamingTest.ChunkedFrame/WithoutContainer_WithSmallBuffers_FastLossless_CannotSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ................................................................***Failed 0.69 sec
3831/4841 Test #3831: EncoderStreamingTest/EncoderStreamingTest.ChunkedFrame/WithoutContainer_WithLargeBuffers_FastLossless_CannotSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ................................................................***Failed 0.66 sec
3833/4841 Test #3833: EncoderStreamingTest/EncoderStreamingTest.ChunkedFrame/WithoutContainer_WithSmallBuffers_WithMultipleFrames_FastLossless_CannotSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ .............................................***Failed 0.74 sec
3835/4841 Test #3835: EncoderStreamingTest/EncoderStreamingTest.ChunkedFrame/WithoutContainer_WithLargeBuffers_WithMultipleFrames_FastLossless_CannotSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ .............................................***Failed 0.76 sec
3845/4841 Test #3845: EncoderStreamingTest/EncoderStreamingTest.ChunkedFrame/WithoutContainer_WithSmallBuffers_FastLossless_CanSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ...................................................................***Failed 0.69 sec
3847/4841 Test #3847: EncoderStreamingTest/EncoderStreamingTest.ChunkedFrame/WithoutContainer_WithLargeBuffers_FastLossless_CanSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ...................................................................***Failed 0.71 sec
3849/4841 Test #3849: EncoderStreamingTest/EncoderStreamingTest.ChunkedFrame/WithoutContainer_WithSmallBuffers_WithMultipleFrames_FastLossless_CanSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ................................................***Failed 0.76 sec
3851/4841 Test #3851: EncoderStreamingTest/EncoderStreamingTest.ChunkedFrame/WithoutContainer_WithLargeBuffers_WithMultipleFrames_FastLossless_CanSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ................................................***Failed 0.71 sec
4085/4841 Test #4085: EncoderStreamingTest/EncoderStreamingTest.ChunkedAndOutputCallback/WithoutContainer_WithSmallBuffers_FastLossless_CannotSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ....................................................***Failed 0.71 sec
4087/4841 Test #4087: EncoderStreamingTest/EncoderStreamingTest.ChunkedAndOutputCallback/WithoutContainer_WithLargeBuffers_FastLossless_CannotSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ....................................................***Failed 0.49 sec
4089/4841 Test #4089: EncoderStreamingTest/EncoderStreamingTest.ChunkedAndOutputCallback/WithoutContainer_WithSmallBuffers_WithMultipleFrames_FastLossless_CannotSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ .................................***Failed 0.79 sec
4091/4841 Test #4091: EncoderStreamingTest/EncoderStreamingTest.ChunkedAndOutputCallback/WithoutContainer_WithLargeBuffers_WithMultipleFrames_FastLossless_CannotSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ .................................***Failed 0.91 sec
4101/4841 Test #4101: EncoderStreamingTest/EncoderStreamingTest.ChunkedAndOutputCallback/WithoutContainer_WithSmallBuffers_FastLossless_CanSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ .......................................................***Failed 0.69 sec
4103/4841 Test #4103: EncoderStreamingTest/EncoderStreamingTest.ChunkedAndOutputCallback/WithoutContainer_WithLargeBuffers_FastLossless_CanSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ .......................................................***Failed 0.71 sec
4105/4841 Test #4105: EncoderStreamingTest/EncoderStreamingTest.ChunkedAndOutputCallback/WithoutContainer_WithSmallBuffers_WithMultipleFrames_FastLossless_CanSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ....................................***Failed 0.54 sec
4107/4841 Test #4107: EncoderStreamingTest/EncoderStreamingTest.ChunkedAndOutputCallback/WithoutContainer_WithLargeBuffers_WithMultipleFrames_FastLossless_CanSeek_WithoutExtraChannels_ColorWithoutAlpha_OneGroup_ ....................................***Failed 0.77 sec
It seems offending tests are been skipped on Debian, but I cannot test if the finally installed libjxl is working ok on those arches (hence only a test issue) or it is broken (in that case it shouldn't be installed probably)
I tried again with v0.11.1 794a5dc
on 6.6.62-gentoo-ppc32-ppc #1 SMP Thu Jan 9 05:40:30 UTC 2025 ppc 7447/7457, altivec supported PowerMac3,1 GNU/Linux
and the failed tests were:
552 - DecodeTest.ProgressionTestLosslessAlpha (Failed)
560 - DecodeTest.FlushTestLosslessProgressiveAlpha (Failed)
3425 - EncodeTest.FrameSettingsTest (Failed)
4273 - JxlTest.RoundtripAlphaResampling (Failed)
4274 - JxlTest.RoundtripAlphaResamplingOnlyAlpha (Failed)
4276 - JxlTest.RoundtripAlpha16 (Failed)
4309 - JxlTest.RoundtripProgressive (Failed)
4310 - JxlTest.RoundtripProgressiveLevel2Slow (Failed)
4399 - ModularTest.RoundtripLossyDeltaPalette (Failed)
4401 - ModularTest.RoundtripLossy (Failed)
4402 - ModularTest.RoundtripLossy16 (Failed)
4438 - RoundtripLossless/ModularTestParam.RoundtripLossless/1bitSqueeze # GetParam() = 8-byte object <00-00 00-01 00-00 00-01> (Failed)
4457 - RoundtripLossless/ModularTestParam.RoundtripLossless/20bitSqueeze # GetParam() = 8-byte object <00-00 00-14 00-00 00-01> (Failed)
4458 - RoundtripLossless/ModularTestParam.RoundtripLossless/21bitSqueeze # GetParam() = 8-byte object <00-00 00-15 00-00 00-01> (Failed)
4461 - RoundtripLossless/ModularTestParam.RoundtripLossless/24bitSqueeze # GetParam() = 8-byte object <00-00 00-18 00-00 00-01> (Failed)
4463 - RoundtripLossless/ModularTestParam.RoundtripLossless/26bitSqueeze # GetParam() = 8-byte object <00-00 00-1A 00-00 00-01> (Failed)
4464 - RoundtripLossless/ModularTestParam.RoundtripLossless/27bitSqueeze # GetParam() = 8-byte object <00-00 00-1B 00-00 00-01> (Failed)
4467 - RoundtripLossless/ModularTestParam.RoundtripLossless/30bitSqueeze # GetParam() = 8-byte object <00-00 00-1E 00-00 00-01> (Failed)
4486 - PassesTest.ProgressiveDownsample2DegradesCorrectlyGrayscale (Failed)
4487 - PassesTest.ProgressiveDownsample2DegradesCorrectly (Failed)
4785 - roundtrip_test (Failed)
Thanks for reporting. I'll try to setup some big-endian environment to check what is going on.
I'd appreciate if you point me to how to repro, e.g. matching qemu
arch / existing docker image with similar system...
(going to start with https://hub.docker.com/r/s390x/debian/ as in original report)
I use qemu-system-ppc -L pc-bios -M mac99 -cpu apollo7 -accel tcg -boot d -m 2048
and I boot from PPC/PPC64 Minimal Installation CD https://www.gentoo.org/downloads/#ppc
Then I roughly followed instructions at https://wiki.gentoo.org/wiki/Handbook:PPC to install other packages.
Running the libjxl tests takes approximately whole day there under QEMU.
Thanks, Daniel. Succeeded with docker.
...with couple extra time-outs:
7116 - JxlTest.RoundtripLargeEmptyModular (Timeout)
7243 - JxlStreamingTest/JxlStreamingEncodingTest.StreamingSamePixels/jxl/flower/flower_alpha.png-e6D4 (Timeout)
Some tests fail fairly quickly, so good chances pinpointing the problem.
TODO(eustas): setup (weekly?) s390x CI tests.
On PowerPC, there is some decoding problem.
Input file: https://github.com/nomacs/formats_testset/blob/main/JXL/cmyka16.jxl
jxlinfo -v cmyka16.jxl
box: type: "JXL " size: 12, contents size: 4
JPEG XL file format container (ISO/IEC 18181-2)
box: type: "ftyp" size: 20, contents size: 12
box: type: "jxll" size: 9, contents size: 1
box: type: "jxlc" size: 306410, contents size: 306402
JPEG XL image, 997x467, (possibly) lossless, 16-bit CMY+Black+Alpha
num_color_channels: 3
num_extra_channels: 2
extra channel 0:
type: Black
bits_per_sample: 16
extra channel 1:
type: Alpha
bits_per_sample: 16
alpha_premultiplied: 0 (Non-premultiplied)
have_preview: 0
have_animation: 0
Intrinsic dimensions: 997x467
Orientation: 1 (Normal)
Color space: 724088-byte ICC profile, CMM type: "ADBE", color space: "CMYK", rendering intent: 1
box: type: "Exif" size: 282, contents size: 274
Uncompressed Exif metadata: 282 bytes
djxl produces strange artifacts.
djxl cmyka16.jxl cmyka16.png
JPEG XL decoder v0.12.0 4d720b88 [Unknown]
Decoded to pixels.
libpng warning: profile '1': 0h: PCS illuminant is not D50
libpng warning: profile '1': 'CMYK': invalid ICC profile color space
997 x 467, 0.052 MP/s [0.05, 0.05], , 1 reps, 1 threads.
the expectation is something like:
I see the corruption in decoded output with some other files too, for example here (only in black channel).
FWIW: compiled in "scalar only" mode, and modular_test
passes =) That narrows down the area of search...
FWIW: here is how OSX preview perceives this file
=)
When I re-built highway and libjxl from git today, I no longer have the cmyka16.jxl
decoding problem,
However, I have a new observation. valgrind on ppc compains about Invalid write of size 4
==6544== Memcheck, a memory error detector
==6544== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
==6544== Using Valgrind-3.24.1.GIT and LibVEX; rerun with -h for copyright info
==6544== Command: djxl cmyka16.jxl cmyka16.png
==6544==
JPEG XL decoder v0.12.0 b3fc5f6 [Unknown]
==6544== Invalid write of size 4
==6544== at 0x4E7340C: fetch_and (atomic_base.h:651)
==6544== by 0x4E7340C: jxl::GroupBorderAssigner::ClearDone(unsigned int) (dec_group_border.cc:59)
==6544== by 0x4F8693B: jxl::LowMemoryRenderPipeline::ClearDone(unsigned int) (low_memory_render_pipeline.h:40)
==6544== by 0x4E0C1A3: jxl::FrameDecoder::ProcessSections(jxl::FrameDecoder::SectionInfo const*, unsigned int, jxl::FrameDecoder::SectionStatus*) (dec_frame.cc:694)
==6544== by 0x4EBC74B: jxl::(anonymous namespace)::JxlDecoderProcessSections(JxlDecoderStruct*) (decode.cc:1118)
==6544== by 0x4EBDD27: jxl::(anonymous namespace)::JxlDecoderProcessCodestream(JxlDecoderStruct*) (decode.cc:1432)
==6544== by 0x4EBFA0B: HandleBoxes(JxlDecoderStruct*) (decode.cc:1901)
==6544== by 0x4EBFFF7: JxlDecoderProcessInput (decode.cc:2052)
==6544== by 0x15D0BF: jxl::extras::DecodeImageJXL(unsigned char const*, unsigned int, jxl::extras::JXLDecompressParams const&, unsigned int*, jxl::extras::PackedPixelFile*, std::vector<unsigned char, std::allocator<unsigned char> >*) (jxl.cc:239)
==6544== by 0x111273: (anonymous namespace)::DecompressJxlToPackedPixelFile(jpegxl::tools::DecompressArgs const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, std::vector<JxlPixelFormat, std::allocator<JxlPixelFormat> > const&, void*, jxl::extras::PackedPixelFile*, unsigned int*, jpegxl::tools::SpeedStats*) (djxl_main.cc:394)
==6544== Thread 2:
==6544== Conditional jump or move depends on uninitialised value(s)
==6544== at 0x4FDAF78: hwy::N_EMU128::Vec128<float, 4u> hwy::N_EMU128::operator/<float, 4u, (void*)0>(hwy::N_EMU128::Vec128<float, 4u>, hwy::N_EMU128::Vec128<float, 4u>) (emu128-inl.h:852)
==6544== by 0x4FDA90B: hwy::N_EMU128::Vec128<float, 4u> hwy::N_EMU128::Div<hwy::N_EMU128::Vec128<float, 4u> >(hwy::N_EMU128::Vec128<float, 4u>, hwy::N_EMU128::Vec128<float, 4u>) (generic_ops-inl.h:7721)
==6544== by 0x4FE0CDB: jxl::N_EMU128::EPF1Stage::ProcessRow(std::vector<std::vector<float*, std::allocator<float*> >, std::allocator<std::vector<float*, std::allocator<float*> > > > const&, std::vector<std::vector<float*, std::allocator<float*> >, std::allocator<std::vector<float*, std::allocator<float*> > > > const&, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int) const (stage_epf.cc:349)
==6544== by 0x4F7F823: jxl::LowMemoryRenderPipeline::RenderRect(unsigned int, std::vector<jxl::Plane<float>, std::allocator<jxl::Plane<float> > >&, jxl::RectT<unsigned int>, jxl::RectT<unsigned int>) (low_memory_render_pipeline.cc:739)
==6544== by 0x4F81113: jxl::LowMemoryRenderPipeline::ProcessBuffers(unsigned int, unsigned int) (low_memory_render_pipeline.cc:907)
==6544== by 0x4F99DFB: jxl::RenderPipeline::InputReady(unsigned int, unsigned int, std::vector<std::pair<jxl::Plane<float>*, jxl::RectT<unsigned int> >, std::allocator<std::pair<jxl::Plane<float>*, jxl::RectT<unsigned int> > > > const&) (render_pipeline.cc:125)
==6544== by 0x4F9A153: jxl::RenderPipelineInput::Done() (render_pipeline.cc:139)
==6544== by 0x4E86E6F: jxl::ModularFrameDecoder::FinalizeDecoding(jxl::FrameHeader const&, jxl::PassesDecoderState*, jxl::ThreadPool*, bool)::{lambda(unsigned int, unsigned int)#1}::operator()(unsigned int, unsigned int) const (dec_modular.cc:775)
==6544== by 0x4E89B67: jxl::ThreadPool::RunCallState<jxl::ModularFrameDecoder::FinalizeDecoding(jxl::FrameHeader const&, jxl::PassesDecoderState*, jxl::ThreadPool*, bool)::{lambda(unsigned int)#1}, jxl::ModularFrameDecoder::FinalizeDecoding(jxl::FrameHeader const&, jxl::PassesDecoderState*, jxl::ThreadPool*, bool)::{lambda(unsigned int, unsigned int)#1}>::CallDataFunc(void*, unsigned int, unsigned int) (data_parallel.h:112)
==6544== by 0x752F3: jpegxl::ThreadParallelRunner::RunRange(jpegxl::ThreadParallelRunner*, unsigned long long, int) (thread_parallel_runner_internal.cc:107)
==6544== by 0x7550F: jpegxl::ThreadParallelRunner::ThreadFunc(jpegxl::ThreadParallelRunner*, int) (thread_parallel_runner_internal.cc:137)
==6544== by 0x76E4F: void std::__invoke_impl<void, void (*)(jpegxl::ThreadParallelRunner*, int), jpegxl::ThreadParallelRunner*, unsigned int>(std::__invoke_other, void (*&&)(jpegxl::ThreadParallelRunner*, int), jpegxl::ThreadParallelRunner*&&, unsigned int&&) (invoke.h:61)
Nice catch. Will investigate soon. (hopefully fixes to skcms will land soon as well)
Describe the bug
Two tests fails on big-endian archs
To Reproduce
Full build log on s390x
powerpc or ppc64 are affected the same way.
Expected behavior
Expected that all tests passes as on little-endian archs
Screenshots
Environment