Allow more preview formats

[rullzer]

Right now we serve previews in png or jpg. Which has the advantage of being support on nearly every platform since the late 90s. But of course it is no longer the best solution out there.

I would propose to allow different image formats. The top candidate in my opinion is webp. This allows for smaller images at comparable quality. Of course the backend would need to distinquis what to server but I'm sure we can solve that with some js magic to set the accept headers ;)

This doesn't mean we fully get rid of the jpg an png as older firefox and IE versions do not support it. But I bet this could be performance boost especially when using a lot of images.

[J0WI]

There is also GraphicsMagick as an alternative to ImageMagick. Vips would be a very fast option but it's not yet implemented in Nextcloud (stale effort in #28279).

10-bit support depends on whether the library was compiled to support it (e.g. CONFIG_AV1_HIGHBITDEPTH=1 in aom).

[JanisPlayer]

@J0WI I think it's possible to replace ImageMagick with GraphicsMagick. It's easily usable in PHP with almost the same functionality. Vips also sounds interesting, but it's more complex to use. However, the current functionality of the Preview Generator won't automatically switch all formats to GraphicsMagick. This means you'd not only have to rewrite existing ImageMagick Preview Generators as a fallback for GD but also add new ones. Whether the performance boost is worth it is the next question. Since a new solution for fast generation seems necessary soon, especially considering Imaginary is no longer being developed by the original developer. It might be used for modernization over time. Just a speculation. Thanks for the acceleration tips; I can use those elsewhere.

[DesertCookie]

@JanisPlayer I'm facing an issue with your patch. Running the pre-generate I get PHP Fatal error: Paletter image not supported by webp in /var/www/html/lib/private/legacy/OC_Image.php on line 455. Running with -vvv indicates it's failing for a document preview: 2024-02-24T17:07:27+00:00 Generating previews for /34/files/Vorlagen/Expense report.ods (only other log output before the error).

I am unsure whether it is an issue with your patch at all, or possibly my instance/install: Fresh instance of Nextcloud AIO (less than a week old) with v27.1.7 RC1 (your patch is only for v27.1.2). Preview format is set to WebP with a quality of 30. At least since applying your patch I don't even get previews downloaded (or possibly even generated) to my Desktops via the desktop client for images (usually it does load a preview, even when the file is only present online).

[JanisPlayer]

I took a brief look at the file. At this point, it checks if your PHP version supports GD WebP with imagetypes() & IMG_WEBP. If not, it returns an error, and the generation is aborted. For Nextcloud AIO, I haven't tested the patch yet due to some security-related removal of certain preview functions like those involving ImageMagick. However, a quick fix for this issue is using Imaginary.php for generation, which I've included in older patches, modified for WebP generations. This change is present in the latest Nextcloud version. I using the old patch for version 27.1 multiple times, even for newer versions where not much has changed, could potentially lead to errors. I suspect the issue is related to the PHP version and installed modules. WebP and AVIF are supported under GD only in later PHP versions. It's strange that a new Nextcloud version doesn't use the recommended PHP version. I'll create a simple script soon to automatically add the changes during pattern search. Handling this for apps is a bit more complex since the code needs processing, but it can work with a pattern patch. I'll check later which PHP version is used in the AIO version.

Edit: https://github.com/nextcloud/all-in-one So I looked, the latest Docker version of them uses the libs, a PHP version that should be able to do that and also the modules.

[DesertCookie]

Trying again today, I do not get any log output after running occ preview:pre-generate -vvv. I've tried with the preview format set to webp, avif, and removing it. I also don't get AVIF previews in the web interface.

[JanisPlayer]

The errors may have disappeared because images had already been generated. But since the patch is actually intended more for manual installation, this may also be due to differences compared to all in one installation.Since there is no way to install it, can you give me more information about your installation and how you ran the patch? So it can actually only be due to a missing lib, module or incorrect PHP version. Or that the version is actually incompatible with the patch. But since WebP previews are enough for you, in version 28 works without a patch. You can read about how Imaginary does this in the documentation. But if you really want AVIF or want to create the previews via GD, this is currently not possible without a patch. Unfortunately, I don't have any information about when a review is planned.

[DesertCookie]

WebP would be enough for me. However, Nextcloud AIO doesn't allow me to upgrade to 28 yet.

I simply installed the nextcloud-aio-mastercontainer:nextcloud/all-in-one in unRAID 6.12.6 and allowed it to configure all the containers for me: nextcloud-aio-database:nextcloud/aio-postgresql, nextcloud-aio-redis:nextcloud/aio-redis, nextcloud-aio-nextcloud:nextcloud/aio-nextcloud, nextcloud-aio-apache:nextcloud/aio-apache, nextcloud-aio-docker-socket-proxy:nextcloud/nextcloud-aio-docker-socket-proxy, nextcloud-aio-clamav:nextcloud/aio-clamav, nextcloud-aio-fulltextsearch:nextcloud:aio-fulltextsearch, nextcloud-aio-imaginary:nextcloud/aio-imaginary, nextcloud-aio-talk-recording:nextcloud/atio-talk-recording, nextcloud-aio-talk:nextcloud/aio-talk, nextcloud-aio-notify-push:nextcloud/aio-notify-push, nextcloud-aio-watchtower:nextcloud/aio-watchtower. I also run an OnlyOffice server.
Everything is proxied by Nginx Proxy Manager. Logins are handled via Authentik SAML. Everything is stored in a Docker volume on my unRAID server's cache SSD. The nextcloud-aio-nextcloud runs PHP 8.1.27.

I simply downloaded the v27 patch ZIP and replaced the files in my volume with the ones from the patch. Then I restarted the container (Nextcloud and Apache).

[DesertCookie]

@JanisPlayer the webp_quality does not change the quality of the generated preview files for me. I have tried anything from 30 to 60 and keep getting webp preview files with the same quality (within 2%). I set the option both in the config file and via your provided OCC command.

[JanisPlayer]

At this point, the default value is defined, which is used if no value is set via occ config:app:set preview avif_quality --value="30". https://github.com/JanisPlayer/server/blob/b5fa45a083e7ff302189343965c33015998b7f0d/lib/private/legacy/OC_Image.php#L58 This line checks if a value is set. https://github.com/JanisPlayer/server/blob/b5fa45a083e7ff302189343965c33015998b7f0d/lib/private/legacy/OC_Image.php#L505 After that, your image is generated with the desired quality: https://github.com/JanisPlayer/server/blob/b5fa45a083e7ff302189343965c33015998b7f0d/lib/private/legacy/OC_Image.php#L460 So, at this point, without logs and precise debugging, I can only speculate why it doesn't work for you to change the quality. It might be due to a compatible version with your Nextcloud version. Since I'm not sure if the patch will be accepted anyway. The only information I found is that AVIF support is planned from version 30 onwards. https://github.com/nextcloud/server/pull/37564#issuecomment-2006968272 Unfortunately, the developer no longer feels like working on his PR, although I thought he would add what I built here as just a feasibility test, making it nicer and more versatile for other formats. So I'll probably have to improve this feasibility test later, but so far, I have no information on what will be added and desired. That's why I haven't improved the quickly improvised feasibility test yet, which only makes minimal changes to the preview generator. So, what my PR should actually do is just ensure that AVIF images are also displayed in the photo app, processed correctly in the AI apps, and previews are created from AVIF images. So basically, the feasibility test should also work. I have also considered just writing an app for this or temporarily using a script that simply replaces the code according to a pattern and executes NodeJS for the apps. So I can only suggest applying the changes to your Nextcloud by doing it yourself if I haven't made a patch for your version yet. You can see the changes here: https://github.com/nextcloud/server/pull/40048/files, you can also use a code comparison tool to compare the differences between the patch and your code and incorporate them into your code. So maybe this information will help you solve the problem.

[Forza-tng]

Hi. I just want to double check if this patch series enables using avif or webp as format for the previews, even if the original image is a jpeg? I have just upgraded to NC 29 and thought it would have a go at regenerating my previews.

I have not applied your patches yet. This is current status for my system after removing all previews and regenerating them using occ preview:generate-all. I am not using Imaginary at the moment. I have PHP 8.2.20 and ImageMagick 7.1.1.

# count of preview files 
❯ find -type f -name "*.webp" | wc -l
301
❯ find -type f -name "*.jpg" | wc -l
298441

config.php

'enable_previews' => true,
'preview_format' => 'webp',

[JanisPlayer]

WebP for previews is already supported in the current version of Nextcloud via Imaginary. AVIF support is still in development but already functional; however, the last tested patch I created was for Nextcloud (28.0.1). You can create a new patch by simply applying all changes to the current version. I wanted to automate this process, but for instance, the Viewer app needs to be rebuilt with NodeJS each time to display AVIF images, which became too time-consuming for me. Another issue is that developers haven't communicated to me what else is needed. For example, there was also the idea of incorporating @aentwist's PR into mine to provide more flexibility in selection, or whether to retain a somewhat imperfect PoW concept or focus solely on displaying AVIF images without AVIF or WebP previews. Therefore, my PR is not currently being actively maintained. However, it has generally been kept up-to-date enough to be easily added to the current version. So, if you want to create a patch for version 29, you simply need to use Diff to add the changes from my patch to the files. The Viewer app needs to be rebuilt with my changes to display AVIF images. For the Photos app, it suffices to modify the file without needing to rebuild the app. Don't forget to add AVIF images to the list as well. Nextcloud config example:

'enable_previews' => true,
'preview_format' => 'webp',
# For Imaginary: 'preview_imaginary_url' => 'http://127.0.0.1:9000',
'enabledPreviewProviders' => 
array (
# For Imaginary: 0 => 'OC\Preview\Imaginary',
1 => 'OC\Preview\JPEG',
2 => 'OC\Preview\GIF',
3 => 'OC\Preview\BMP',
4 => 'OC\Preview\XBitmap',
5 => 'OC\Preview\Movie',
6 => 'OC\Preview\PDF',
7 => 'OC\Preview\MP3',
8 => 'OC\Preview\TXT',
9 => 'OC\Preview\MarkDown',
10 => 'OC\Preview\PNG',
11 => 'OC\Preview\WebP',
12 => 'OC\Preview\HEIC',
13 => 'OC\Preview\OpenDocument',
14 => 'OC\Preview\Krita',
15 => 'OC\Preview\AVIF',
20 => 'OC\Preview\Movie',
),

If anyone wants to automate this process, for example, using Bash, they can share the script with me. But I'm not sure if there is enough demand to justify it. Ideally, this PR should have been included in Nextcloud long ago.

[Forza-tng]

WebP for previews is already supported in the current version of Nextcloud via Imaginary.

I see..I might look at Imaginary again. I just didn't see a huge benefit, and it also had problems with some files that otherwise worked. I think, for example, dng/raw files did not work with Imaginary.

I am surprised that Imaginary is a requirement for WebP as PHP natively supports it in its gd library since a while back. I created a ticket for allowing WebP previews via PHP gd. In PHP 8.1 even AVIF is natively supported. https://github.com/nextcloud/server/issues/46611

[jmigual]

Hey! So I'm facing a strange issue. Somehow, even though I enabled webp preview creation, I still see some previews generated as .jpg. I'ved made sure to check whether they were from before by deleting all .jpg images but they appear again after a while (when nextcloud generates them). This is the relevant part of my config:

config.php

```php false, 'default_language' => 'ca', 'default_locale' => 'ca_ES', 'default_phone_region' => 'ES', 'defaultapp' => 'files', 'htaccess.RewriteBase' => '/', 'memcache.local' => '\\OC\\Memcache\\APCu', 'allow_local_remote_servers' => true, 'apps_paths' => array ( 0 => array ( 'path' => '/var/www/html/apps', 'url' => '/apps', 'writable' => false, ), 1 => array ( 'path' => '/var/www/html/custom_apps', 'url' => '/custom_apps', 'writable' => true, ), ), 'memcache.distributed' => '\\OC\\Memcache\\Redis', 'memcache.locking' => '\\OC\\Memcache\\Redis', 'passwordsalt' => 'REDACTED', 'secret' => 'REDACTED', 'trusted_domains' => array ( 0 => 'localhost', 1 => 'REDACTED', 2 => 'REDACTED', ), 'datadirectory' => '/var/www/html/data', 'version' => '28.0.7.4', 'overwrite.cli.url' => 'REDACTED', 'dbtype' => 'pgsql', 'dbname' => 'nextcloud', 'dbhost' => 'nextcloud-postgres', 'dbport' => '', 'dbtableprefix' => 'oc_', 'dbuser' => 'nextcloud', 'dbpassword' => 'REDACTED', 'mysql.utf8mb4' => true, 'installed' => true, 'instanceid' => 'REDACTED', 'maintenance' => false, 'overwriteprotocol' => 'https', 'loglevel' => 0, 'logfile' => '/var/www/html/data/logs/nextcloud.log', 'trashbin_retention_obligation' => '30, 31', 'versions_retention_obligation' => 'auto, 31', 'enabledPreviewProviders' => array ( 0 => 'OC\\Preview\\GIF', 1 => 'OC\\Preview\\Imaginary', 2 => 'OC\\Preview\\PNG', 3 => 'OC\\Preview\\WebP', 4 => 'OC\\Preview\\JPEG', 5 => 'OC\\Preview\\HEIC', 6 => 'OC\\Preview\\BMP', 7 => 'OC\\Preview\\XBitmap', 8 => 'OC\\Preview\\MP3', 9 => 'OC\\Preview\\TXT', 10 => 'OC\\Preview\\MarkDown', 11 => 'OC\\Preview\\Krita', 12 => 'OC\\Preview\\MP4', 13 => 'OC\\Preview\\Movie', 14 => 'OC\\Preview\\MKV', 15 => 'OC\\Preview\\AVI', 16 => 'OC\\Preview\\PDF', 17 => 'OC\\Preview\\SVG', 18 => 'OC\\Preview\\TIFF', ), 'enable_previews' => true, 'preview_max_x' => 4096, 'preview_max_y' => 4096, 'preview_imaginary_url' => 'http://imaginary:9000', 'preview_format' => 'webp', 'preview_ffmpeg_path' => '/usr/bin/ffmpeg', 'webp_quality' => '80', 'jpeg_quality' => 80, 'app_install_overwrite' => array ( 0 => 'facerecognition', 1 => 'extract', ), 'theme' => '', 'mail_smtpmode' => 'smtp', 'mail_smtphost' => 'REDACTED', 'mail_sendmailmode' => 'smtp', 'mail_from_address' => 'nextcloud', 'mail_domain' => 'REDACTED', 'mail_smtpauth' => 1, 'mail_smtpauthtype' => 'LOGIN', 'mail_smtpname' => 'apikey', 'mail_smtppassword' => 'REDACTED', 'mail_smtpport' => '587', 'memories.exiftool' => '/var/www/html/custom_apps/memories/bin-ext/exiftool-amd64-glibc', 'memories.vod.path' => '/var/www/html/custom_apps/memories/bin-ext/go-vod-amd64', 'memories.vod.ffmpeg' => '/usr/bin/ffmpeg', 'memories.vod.ffprobe' => '/usr/bin/ffprobe', 'redis' => array ( 'host' => 'nextcloud-redis', 'password' => '', 'port' => 6379, ), 'memories.gis_type' => 2, 'memories.db.triggers.fcu' => true, 'maintenance_window_start' => 1, 'trusted_proxies' => array ( 0 => 'REDACTED', ), ); ```

[JanisPlayer]

@jmigual There should be an error in the log when dealing with image formats that Imaginary does not support. As a fallback, the GD or Imagemagick generator is used, both of which do not support WebP. It might also be that your Imaginary has too low a limit on the number of images that can be generated simultaneously. This will also appear as an error message in the log. You can increase the limit in the Docker config. With your log and information about the image properties of the files that do not work and the config of your Imaginary container, we might be able to get to the bottom of the problem.

@Forza-tng Yes, that's exactly what my pull request does. The code of Preview Generator in its original version is very limited and only allows a few formats, so it's not PHP's fault. A modification for Imaginary was easier back then than rewriting the entire GD generator. I am currently using WebP with 80 percent quality via Imaginary myself.

[MenooMeow]

Hello. I read above that AVIF is planned for NC 30? Is that still true? I have a rather large collection of images mounted externally but none are really accessible via NC since it ignores them. Has anything changed that now allows NC to read AVIF in the Files section? I'd like to second the request for this capability if there is a way to throw my request in.

[strech345]

Please also think about jxl. Its better in compression and speed Else avif. Apple supports it and hope all others Like chrom and Firefox will follow

[langfingaz]

@strech345

While JPEG XL is another royality free modern image format, I am not sure if it is better in all aspects than AVIF. Proper requirements analysis has to be done to decide if it should be prioritized over AVIF.

Ideally, the architecture of the preview generator should be modular to allow the easy addition of further image formats in the future. By following this issue I think that there is not too much left until we could have AVIF support. So I think it is best to do that first.

[JanisPlayer]

@strech345 JPEG XL is currently supported by Apple, but it has been removed from Chrome and Firefox. It can now only be supported via WebASM or JavaScript libraries. It’s a wonderful image format in terms of versatility, but it also has the same problems as JPEG — 8x8 pixel blocks tend to produce artifacts in small images. However, a major advantage is that JPEG XL generally performs better with details found in the real world, while AVIF tends to blur these or completely remove or misrepresent them. Here's a comparison.

Since JPEG XL needs to be supported by both web browsers and operating systems for smartphones, it's highly unlikely that it will have a future as a preview format. The smartphone's OS, like with AVIF, would occasionally need an extra library to display all images. For example, AVIF has issues with images over 1080p on Android. Preview formats are only concerned with displaying an image with good quality, minimal size, acceptable encoding speed for the system, and wide support. Unfortunately, given the current support, it doesn't meet those criteria. JPEG XL might still have a future if tools like ImageMagick or PHP’s GD library support it. Then I could convert JPEG XL images into previews. I think Nextcloud's standard Photos app will support it eventually, though I've noticed that some app developers don’t seem eager to support all formats. I believe they'll catch up if they see it stick around, but I'm not entirely sure.

Generally speaking, I had the issue where at some point no one provided any feedback, even though everything was finished and needed it. So I have no idea if these new formats even have a future or if you need to ring a bell on GitHub. However, I recently read under a different pull request that supposedly version 30 would be it. I don't really believe that; as you can see, it was closed and bumped up to version 31. So, because of all these issues, I wouldn’t add any more preview formats to my pull request. Instead, I’d add the option to convert JPEG XL into previews if that’s possible through PHP. As I explained above, adding a new preview format is tricky since it requires broader support or extra libraries, and then there are the apps to consider. I also need testers to spot problems and other developers who are actively working on the project to tell me what's possible, like whether I'm allowed to use certain libraries, which could mean significant changes. I think that’s why the developers probably want to handle it themselves. My pull request and the other one may just serve as templates or ideas. That’s why I didn’t want to make my pull request too complicated, so it could be integrated quickly.

So, in conclusion, JPEG XL looks difficult to support, especially since I already had problems with a well-supported format like AVIF.

[Forza-tng]

Well, in IMHO, if the preview and image handling features would be image format agnostic, usersand organisation can choose which format is best for their needs. New formats like JPEG-XL would then be easy to add

If we keep hard-coding formats, it will always be difficult to make changes.

Preview formats are only concerned with displaying an image with good quality, minimal size, acceptable encoding speed for the system, and wide support.

One frequent complaint for home users is the large disk space usage of the thumbnails. AVIF and WebP would be very beneficial for those, as they can save 50% disk space compared to JPEG.

I did a qualitative (SSIM) test of jpeg, avif and webp so it is possible to see for a specific image size and target quality, the resulting file sizes would be for each format. There are many ways to measure quality, and I chose one I could easily script with imagemagic. Feel free to try other methods.

https://mirrors.tnonline.net/misc/img-compare/001_Juvenile_leopard_in_the_Serengeti_National_Park_Photo_by_Giles_Laurent/

[langfingaz]

@Forza-tng

Well, in IMHO, if the preview and image handling features would be image format agnostic, usersand organisation can choose which format is best for their needs.

If we keep hard-coding formats, it will always be difficult to make changes.

There are two things we have to separate here: 1) Conversion of images into a preview format 2) Choosing a preview format and displaying preview images

Regarding (1):

If the preview generator app uses a library on the server that can decode lots of image formats, it would be nice to generate small preview images for these formats. Additionally, there should be an option to enable/disable individual formats. For each image format this requires some (small) amount of code.

Your wish to avoid hard coding formats could be realized this way: Adding an option to enable reading any image format that the photo library supports. This way, as soon as new formats are supported by the library, previews can get generated on the Nextcloud server. However, I don't think that this is an option enterprises would want to use on their Nextcloud instances for stability/security reasons.

Regarding (2): The list of possible formats for the generated preview images is limited and can only slowly adopt. The reason here is that all clients (supported web browsers, mobile Apps, etc.) need to support all these formats. That's why it takes so long to support e.g. AVIF as format for the preview images. As explained avove, JPEG XL is not (properly) supported by all common webbrowsers.

[Forza-tng]

Regarding (2): The list of possible formats for the generated preview images is limited and can only slowly adopt. The reason here is that all clients (supported web browsers, mobile Apps, etc.) need to support all these formats. That's why it takes so long to support e.g. AVIF as format for the preview images. As explained avove, JPEG XL is not (properly) supported by all common webbrowsers.

For my use case with family photo archive of 100k photos it would make a rather big difference for us. All our devices would have no problem with either webp or avif. Why shouldn't there be an option to configure the format? There is no need, at least for this use case to implement some kind of auto detection, browser sniffing, and all that. Just a simple configuration option.

[JanisPlayer]

@Forza-tng yes, it's just not as simple as they imagine. But they can already use WebP today, as I do. The savings are usually only about 10% in most cases. If the visual quality isn't that important, they can lower the quality even further. The encoders in GD, ImageMagick, or Imaginary are usually set for fast encoding speed, which also affects the quality. So, we’re not using the library directly. Therefore, the comparison of perceived visual quality is a bit different. I think I already have close to 50k photos on my Nextcloud, and with WebP, the storage for previews up to 4096px at 80% quality is only about 16.4 GiB.

If they're only interested in displaying JPEG XL, meaning having Nextcloud convert a JPEG XL image into a WebP preview image, that's probably easier to support. The prerequisite is that GD or ImageMagick (Imaginary is unlikely, as it’s no longer being updated, and as a Nextcloud developer told me, they don’t want to take it over) provides support for decoding.

@langfingaz also made an interesting point about using modulo and libraries, which would require significant maintenance, and only the developers themselves can decide if they want to make such substantial changes to the preview generator. There may also be security concerns with maintaining the libraries. So, it's not that simple. They can check out my PR to understand the basic workings of the current preview generator and how the apps need to be modified so they can load the images. But it's not as easy as they imagine.

I think they can also see that JPEG XL, as I explained above, due to the issue with support requiring many libraries, is currently unlikely to be used as a preview format. Converting it to a WebP preview is much simpler if the decoder is supported, and adapting the apps to recognize it as an image and load the preview is doable. But as a preview format, JPEG XL is highly unlikely to be used in productive systems right now due to the lack of native support. My tip is to enable WebP as a preview format, which will save a few percent and, in some cases, even more.