JXL export: libjxl call failed with err 1

[RafaelLinux]

Describe the bug

Trying to export to lossless JXL ends after a long time (about 35 minutes) with an error. I tried with distinct RAW files and it works when exporting to TIFF.

Steps to reproduce

1. Select a RAW file
2. In "export", select JXL format and 100 quality
3. Click on "Export"

Expected behavior

Create a JXL file after about 7 seconds

Logfile | Screenshot | Screencast

output.txt

Commit

No response

Where did you obtain darktable from?

distro packaging

darktable version

4.6.0

What OS are you using?

Linux

What is the version of your OS?

openSUSE Tumbleweed

Describe your system?

Operating System: openSUSE Tumbleweed 20240202 KDE Plasma Version: 5.27.10 KDE Frameworks Version: 5.114.0 Qt Version: 5.15.12 Kernel Version: 6.7.2-1-default (64-bit) Graphics Platform: Wayland libjxl: 0.8.1-4.1

Processors: 24 × AMD Ryzen 9 3900X 12-Core Processor Memory: 31.3 GiB of RAM Graphics Processor: AMD Radeon RX 7600 Product Name: B550M Phantom Gaming 4

Are you using OpenCL GPU in darktable?

No

If yes, what is the GPU card and driver?

No response

Please provide additional context if applicable. You can attach files too, but might need to rename to .txt or .zip

I can export to lossless JXL with GIMP, Krita, Digikam … however when I try to export to JXL (quality 100, so lossless) with Darktable, it remains exporting forever, but doing nothing about (Darktable is loaded and working, but not exporting). If I cancel export queue and close Darktable window, I can see Darktable loaded in memory thru “system activity” (Plasma, KDE).

[ralfbrown]

With that kind of slowdown, it might also be worth limiting effort to at most 7. Or at least putting up a big flashing warning when selecting 7 or above....

[RafaelLinux]

I'm going to continue the conversation from pixls.us here since I think it is more appropriate. I'm not sure about the error you had, but based on the reading at libjxl, when using lossless, effort values above 4 dont provide too much benefit. Increasing the effort does increase the amount of time to encode. The recommendation was to use 3.

I tried (I answered in Pixl.us) your advice about effort parameter, but in my case, seems not to be related. Anyway, your info let me notice I didn't include my system libjxl version.

Anyway, you have a detailed and interesting explanation about effort parameter here. In fact, in JXL GitHub they wrote "Some updates/additions to the main README, and a new documentation file explaining what encode effort actually means (which is not just "higher effort implies smaller files" as some people would expect), and what exactly the various effort settings are doing."

[kmilos]

OP, what version is libjxl in your system?

libhwy could also play a part in this, if using the system one that is...

[RafaelLinux]

For quality settings less than 100, therefore lossy compression, the recommendation is an effort of 7. From what I read, the effort helps avoiding artifacts. Lossy is the normal use case for jxl when exporting from a raw editor and you keep the original raw. I will test the effort settings later today using q=80.

I would love to be able to use RAW files as a source and perform non-destructive operations on them in a workflow or ecosystem such as Adobe's. But on Linux, unfortunately, we don't have that. Therefore, from a photo manager like Digikam/ShowFoto/Krita/GIMP I can't see in the thumbnails the filters applied by Darktable. Therefore, to follow my usual workflow (e.g. blending images, applying brushes, etc) I need a pre-processed file to work on losslessly, which until now, was TIFF. However, the appearance of JXL and its impressive native lossless and lossy capabilities make it clear to me that it is the format to work on once I'm done with what Darktable offers (which is no small feat). The release version (when all browsers adopt JXL) will be lossy. But it's a matter of opinion.

While this is interesting, it still doesn't address the OP error. I looked into their documentation and API, but I can't find the meaning of error code 1.

OP, what version is libjxl in your system?

I'm sorry. I forgot to put in my first post (I'll update it now) 0.8.1-4.1

[kmilos]

I looked into their documentation and API, but I can't find the meaning of error code 1.

Unfortunately "err 1" does not give a useful hint:

https://github.com/libjxl/libjxl/blob/v0.9.2/lib/include/jxl/encode.h#L87-L89

Does this happen with the nightly AppImage?

[RafaelLinux]

Does this happen with the nightly AppImage?

I'll try in 6 hours.

[RafaelLinux]

OP, what version is libjxl in your system?

libhwy could also play a part in this, if using the system one that is...

1.0.7-1.2 in my case.

[kmilos]

Also, does the error happen if you disable metadata in the export preferences?

[RafaelLinux]

Also, does the error happen if you disable metadata in the export preferences?

I'll try at night too, before trying AppImage. Anyway, I suppose that all the other applications I have mentioned will also use this library to export, and they have not given me any problems, although it is true that in these cases I have not started from a RAW file, but from a TIFF file, which have practically the same metadata (although not the same amount of information).

[RafaelLinux]

On Krita, what are your export settings?

• Save as JPEG-XL checked
• Lossless codification
• Compensation: 7 (default)
• Transcoding speed: 0 (default)

[RafaelLinux]

Also, does the error happen if you disable metadata in the export preferences?

I don't know how to avoid metadata export. Doesn't find an option about that in export settings.

[RafaelLinux]

Does this happen with the nightly AppImage?

It doesn't seems to exist an AppImage in Darktable homepage.

[RafaelLinux]

In the export module, click on the hamburger menu and you will get to the export preferences.

It seems no metadata is exported by default:

The app image is here on GitHub as part of the nightly ci runs. It is the current version of the master branch (includes all bug fixes plus new development features).

Unfortunately, same behaviour :(

This is one of the files Darktable fails to export to JXL https://www.transfernow.net/dl/202402107TQuhCFm

[RafaelLinux]

First, I must say that after using the AppImage, I can't use my distro Darktable version, cause database is altered and can't be read by previous versions.

Did you try with q=80 e1?

I tried that config to export JXL, and no issues!!!. I tried a lot of combinations and finally I found out that error is related to lossless quality. Any effort value (even max) works, but if quality value is 100 (lossless) always fails to export (mentioned error). I don't know why this happens, only with Darktable and only with lossless quality.

[RafaelLinux]

Can you share your xmp?

Renamed to TXT, to get to upload to GitHub _1020827.RW2.xmp.txt

What the profile for export? sRGB? What's the intent for export?

Not sure about "intent" (maybe a translation issue), so this is the screenshot of "export" tab:

And yes we know that it is only happening in dt with lossless.

I didn't know only happens with lossless till yesterday. Never tried in lossy with Darktable, cause I don't want a lossy version ;)

[RafaelLinux]

Again, you got it right. Using lossless compression, with maximum effort and floating point, it manages to export without problems. So, the problem is related to the whole pixel type, I don't know if it's the way Darktable handles it or if it's the library itself.

In view of the results it is worth using JXL as a lossless storage format. TIFF with lossless deflate takes 107M, while JXL takes 51M, almost half the size.

[RafaelLinux]

Thank you. I wish you get to fix it soon. Meanwhile, I'll use the AppImage version till my distro compiles official version.

[RafaelLinux]

I read some more information online and added this JxlEncoderSetCodestreamLevel to 10 in a local branch. I dont notice a difference.

I read this, maybe related with the error showed: "The default value is 5. To use level 10 features, the setting must be explicitly set to 10, the encoder will not automatically enable it. If incompatible parameters such as too high image resolution for the current level are set, the encoder will return an error."

As a test I used other images I had from PlayRaw. Images larger than yours (yields 150MB) and other RW2 are all instant. Your image is the only one that takes a long time.

Error happens with all my RAW files (from my Lumix). Maybe is a Lumix RAW native format complexity

[RafaelLinux]

Usually, I put my camera color space to AdobeRGB, cause I do large photo prints and as you know, AdobeRGB spectrum is wider. Then, in Darktable export to sRGB when I'll use images for web.

[kmilos]

I think during lossless uint16 we explicitly avoid color space conversion. When we select float, dt does provide the option to use the internal profile vs keeping the original.

Correct. However, this has nothing to do w/ Exif data (don't even think we use Exif.Photo.ColorSpace anywhere), we're talking about the color space of the dt colorout module, or color space set by the dt export module itself (if different to colorout).

[RafaelLinux]

Setting to Adobe RGB while shooting raw is not useful. The dt export profile is the one that should matter.

I imagine, because you don't specify it, that you mean that if I only take pictures in RAW, the colour space is indifferent, as it is not taken into account. However, if I take pictures in RAW + JPG (which is what I usually do to be able to manage them immediately), the chosen colour space is important. In fact, my monitors are calibrated for AdobeRGB, so it is quicker for me to discard or accept the photos from the JPG with the AdobeRGB profile because I know that I will be seeing the exact colours on my monitor. It is when I am going to send the photo, after editing, that I choose CMYK or sRGB as the colour space.

[github-actions[bot]]

This issue has been marked as stale due to inactivity for the last 60 days. It will be automatically closed in 300 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.

[RafaelLinux]

Please, after testing with the AppImage version you asked me to test, the database is not the same and I still have to launch the AppImage version. With which version of Darktable from my distribution's repositories will I be able to use Darktable normally?

Thanks

[RafaelLinux]

I can't reproduce this issue in latest stable releases of DT, so I close it. Thanks again