search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.33k stars 4.06k forks source link

Allow more preview formats #13552

Open rullzer opened 5 years ago

rullzer commented 5 years ago

Right now we serve previews in png or jpg. Which has the advantage of being support on nearly every platform since the late 90s. But of course it is no longer the best solution out there.

I would propose to allow different image formats. The top candidate in my opinion is webp. This allows for smaller images at comparable quality. Of course the backend would need to distinquis what to server but I'm sure we can solve that with some js magic to set the accept headers ;)

This doesn't mean we fully get rid of the jpg an png as older firefox and IE versions do not support it. But I bet this could be performance boost especially when using a lot of images.

lapineige commented 5 years ago

:+1: for WebP, it is now supported by the major web browsers, and is more interesting than JPEG, PNG and GIF if we look at the quality/file size.

I think that would be a nice addition, for instance for people who wants to save some storage space on their servers thanks to WebP, for a galery of pictures, and so on. And images in this format would speed up page loading (smaller file size).

MorrisJobke commented 5 years ago

ref #9857

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity and it seems to be missing some essential informations. It will be closed if no further activity occurs. Thank you for your contributions.

J0WI commented 5 years ago

There is a tiny app for WebP support written by @Flow86: https://github.com/Flow86/preview_webp

Would you agree to merge this functionality upstream?

Flow86 commented 5 years ago

yes of course you can merge the functionality to upstream

stefaang commented 5 years ago

Here's an interesting demo with blurred webp preloads: https://jmperezperez.com/webp-placeholder-images/

SimplyCorbett commented 3 years ago

Can we get a status update on this?

mrAceT commented 2 years ago

There is a tiny app for WebP support written by @Flow86: https://github.com/Flow86/preview_webp Would you agree to merge this functionality upstream?

This has been written in 2019.. when I see the code one would think this could be added to the main server software?

Just a thought, when one takes a look at 'oc_filecache' you see that the preview images are generated in the same format as the original (JPG / PNG). Webp is now (version 24.0.3) not really supported. When I upload a webp image, the preview generated is in PNG! Could it be an idea to add the functionality for webp and add the following option to the config:

'preview_format' => 'webp',

This because webp is smaller then PNG and JPG, so when one would want previews in webp, oen would want it for all previews..

J0WI commented 2 years ago

same for AVIF btw.

utack commented 2 years ago

We even regressed on this ticket, heif previews are now disabled

whitephoenix117 commented 1 year ago

Hope this is still active, adding AVIF support would outstanding, its supported in imageMagick.

I did some testing with some of my large (60MP) pictures and its able to reduce ~10MB photos (jpg) to 0.5M-2MB (avif)

aentwist commented 1 year ago

Clearly the frontrunners here are WEBP and AVIF. AVIF for "best" format (with caveats) and WEBP for the (still current, yet quickly shrinking) advantage of browser support it has over AVIF. JPG and PNG are ancient.

https://jakearchibald.com/2020/avif-has-landed/ https://caniuse.com/avif

I am interested in helping with this if it gets more direction.

aentwist commented 1 year ago

I haven't actually used Nextcloud, but from what I can tell after reading the code, previews are generated by simply resizing original images and preserving the format (whether it be JPEG, PNG, WebP, ETC). Some compression is added, at least for JPEGs.

So we want to allow all previews to be of a specific format.

Note that Nextcloud seems to have WebP support [ https://github.com/nextcloud/server/commit/6d4afca7ace0b62f4256b18a0cab60f180ebbc07 ]. It does not seem to have AVIF support.

Let's break this down -

this issue

at this point WebP previews are available

maybe this issue, maybe other issues

I am going to look into the main part first - saving in specific formats. The rest is chores.

aentwist commented 1 year ago

I turned on AVIF quality 1 and now my Nextcloud is screaming fast. Also, if you wear glasses but forget to put them on, there is no disadvantage. This is the way.

On a more serious note go grill my initial pass at this. I look forward to it. Still needs polished. Next steps for me, put up a draft PR for the documentation.

aentwist commented 1 year ago

From #30118

At this time using AVIF is *not* trivial due to system dependencies. TLDR - compile a current version of libaom for use on Debian bullseye. [docker-php-extension-installer](https://github.com/mlocati/docker-php-extension-installer) does this for you. The package manager's version should work in bookworm. https://github.com/juliushaertl/nextcloud-docker-dev/issues/182

I can provide my doctored Docker image (which uses docker-php-extension-installer) for the time being. In ~ 6 months - 1 year it probably won't be an issue whatsoever.

darkBuddha commented 1 year ago

AVIF is essential

JanisPlayer commented 1 year ago

I once conducted a test and added AVIF to the preview generator in the following files:

To identify all the places where I needed to make changes, I simply searched for WebP. Here's the link to my search: GitHub WebP Search.

In the end, I only modified the most important parts. Now, AVIF images are also being converted into preview thumbnails. If there's still interest and someone would like to assist me, I would create a pull request.

The only issue I'm facing is that the images are not being accessed within the APP file on the web and observed with the image viewer, although the thumbnail generation is working. However, based on my test, it's feasible and even relatively straightforward to implement, as you essentially need to copy everything from WebP and adapt it for AVIF.

lapineige commented 1 year ago

I can't help with the implementation part, but there is an interest, yes !

JanisPlayer commented 1 year ago

I've gone ahead and created a PR based on my personal experimentation, incorporating what has worked relatively well so far. However, there are still a few issues that need to be addressed, so the work is not yet complete. Nonetheless, it's a starting point that demonstrates the feasibility of the concept. I'll see later how I can add contributors to the patch, so that you might contribute the parts I couldn't address quickly.

aentwist commented 1 year ago

See also my PR, which never gained any traction for some reason despite being fully working afaict. Maybe if you swirl them together you'll get something good. But you need to work with someone who will review properly and merge, and I am not sure who that would be.

JanisPlayer commented 1 year ago

PR

Ah, I see you've even taken it a step further and wanted to fully integrate AVIF as an alternative to JPEG. I think that was the issue. If I focus on AVIF support for decoding and conversion, they might accept it. It's not like it's a rare, poorly supported format anymore, as JPEG XL unfortunately turned out to be, where you now need WebAssembly to display images in the browser and even GIMP doesn't fully support it.

I mean, AVIF has really gained excellent support today, and owning an AVIF image is no longer a rarity. So, from a pure support perspective, I don't foresee any problems. I think the concern was that your idea didn't go over well due to the high overhead that the encoder requires. A developer from a Nextcloud image application actually demonstrated to me that anything above 1080p takes an extremely long time and can lead to disruptions. Therefore, the best approach might be to decouple something like this from Nextcloud. That's what I'm also doing with my WebP images, which causes no inconvenience whatsoever. However, WebP images also have significantly lower overhead, so with AVIF, it's inevitable to impose restrictions on resolution and quality. This limitation, though, would suffice for preview images without significant zoom, as is common with Nextcloud.

Thank you; I'll take some inspiration from your approach, and perhaps I can resolve a few issues this way.

darkBuddha commented 1 year ago

I converted all my documents and images to AVIF, and all my videos to AV1. To have first-class support for AV1 + AVIF is an absolute necessity. It will be everywhere very soon.

JanisPlayer commented 1 year ago

@darkBuddha Yeah, the browser support for AVIF has greatly improved. Only Edge, Opera Mini, QQ Browser, KaiOS Browser, and of course IE don't support it anymore. It's questionable why Edge doesn't have support yet, considering it's based on Chromium. Microsoft seems to only make amazing modern browsers that can do everything. ;D

But there are enough user-friendly and widely-used options to display and create this format. I've been quite surprised that Nextcloud still doesn't have support for previews.

In the meantime, I've also looked at the PR from @aentwist. I think it was well-structured and had interesting ideas included. However, the issue is that the code needs to be updated to the current state. I'm currently facing a few challenges to implement the previews correctly.

For example, deciding whether to use a PNG or WebP with an alpha channel. And ensuring that the images are displayed correctly within the Nextcloud Apps interface. I just can't seem to find the point in the interface where the app decides whether to use the viewer or to download the file.

Edit: It has turned out that adding to the Viewer app is also possible. There was also a PR for WebP and HEIF back then. However, doing this today is apparently easier than it was at the time of the PR. Nowadays, you only need to add an image.avif.cy.ts file to cypress/e2e/images/, and in /src/models/images.js, you'll need to decide what to add AVIF for, as many browsers support it, but not all of them. That's a good question.

As of now, the following is possible:

So, anyone who has solutions to these problems could contribute them. I can't say for sure whether AVIF will actually be added to Nextcloud, because @aentwist's PR intended to do so, but somehow it wasn't accepted.

There seemed to be a lack of review and interest, despite there being a milestone. There was hardly any feedback afterwards, regarding improvements or functionalities.

@aentwist, do you know what your PR still needs in order to be merged?

JanisPlayer commented 1 year ago

I've added AVIF support experimentally with an extra option to DG. However, I've designed it in a way that makes it easy to remove, and by default, it will use JPEG without this option for AVIF images.

One could consider converting AVIF images to lower-resolution AVIF images in the standard settings, but I believe JPEG is sufficient for this purpose. As mentioned earlier, due to browser support, this might not be a feasible option.

If you set 'preview_format' => 'avif' in the configuration, all images, except GIFs, will be converted to AVIF by DG Preview Generator. This seems to work acceptably up to a resolution of 1080p, even though it heavily taxes the CPU. So, the question is, would AVIF support as a preview format be useful to you, even though it might only yield reasonable performance at lower resolutions on fast systems?

Additionally, some browsers, as mentioned above, won't display the images anymore because of the lack of native support. When you open the image, the original is loaded, and the thumbnail is missing because all images, regardless of resolution, are converted to AVIF with this option.

I didn't want to add extra code to handle different resolutions for small thumbnails in JPEG and large previews in AVIF.

I'm considering fully supporting AVIF through Imaginary, but this project needs some adjustments. The significant advantage is that it's separate from Nextcloud and can be better optimized. Although there are forks of Imaginary, they are not entirely finished, and you can't create PRs for changes in Nextcloud. So, I would need to ask a developer there first.

But first, I have a question for you all:

If all of these conditions are met, I can try adding AVIF experimentally with this setting as a preview format. I've also created a benchmark to give you a sense of what it's like when generating 8 images compared to JPEG and WebP.

Since my goal is to make as few changes as possible and the highest priority is to convert AVIF to a preview initially, this is just an experiment. I would even suggest that someone with more experience completely rewrite the Preview Generator, and we first add AVIF in a way that can be easily removed. I've already adjusted most of the Nextcloud apps for AVIF support. If I missed any, please let me know, or feel free to make the adjustments yourself as a PR.

DesertCookie commented 1 year ago

To answer your survey:

In my experience too, AV1 is mostly efficient at higher resolutions. 1080p is about the lowest I'd ever go with it (at lower resolutions it's usually beaten by well-tuned VP9 and HEVC). I'm mainly looking towards native AVIF support for originals; those currently don't even produce a preview image. However, working with 6-8k images that AVIF can reduce to 25% of their JPEG 80% size while also using 10bit just makes it very hard to not want to move all my images to it.

Glutamat42 commented 1 year ago

This is only relavant for my personal small instance of Nextcloud as this is the only one where the photos app is used.

For me the relevant factor is bandwidth. Nevertheless, support by/for thumbnail pregeneration would be a nice feature to reduce the impact of the longer preview generation time.

vid-bin commented 1 year ago
JanisPlayer commented 1 year ago

Thank you for the feedback. I won't impose a limit on this experimental feature, but I'll include a warning in the documentation that resolutions above 1080p may lead to increased resource usage.

Regarding browser support, it does seem a bit challenging. It's not straightforward to set an alternative source since that would require rewriting every client app. The client would need to inform the server of the user agent it's using. The server would then have to send the appropriate preview. To make this work, I'd also need to change how images are stored in the database and how client apps request previews. So, while it's technically possible, whether the core developers would appreciate such a significant overhaul just for optimal AVIF preview support for an experimental feature is questionable.

I checked, and most browsers will soon support AVIF, including Edge, which can already handle it if activated. However, whether you trust this for a third-party user is another matter. Since some seem to want to control the encoder efficiency, it might be possible to make it configurable via a config command. However, this will likely slow down your system, although the images would be smaller. I'm not entirely sure if this approach will be well-received, as it was hastily implemented with the hope that another developer might revamp the preview generator, as there is already a pull request for that.

I'll soon test the code on the latest version of Nextcloud and provide a patch for you to test by replacing the files.

If AVIF as a preview format doesn't come to fruition, you might consider building an app as a temporary solution. What does work well, in any case, is converting AVIF to JPEG previews. I've done the same with WebP, using it as a preview format instead of PNG since it doesn't rely on ImageMagick, which is currently used for HEIC, for example.

I'll need to consult the developers about the priority of the preview generator for each format. There's GD from PHP and ImageMagick. My assumption was that it doesn't matter much and is more about which formats are supported and practical.

I also inquired about using Imaginary (an external tool), but Nextcloud is reluctant to create a genuine fork or use another existing one, which would be necessary to support AVIF. Unfortunately, this option is off the table. We'll have to wait for the developer to add AVIF support to Imaginary, which would have made things much simpler.

Regarding CPU priority, it's also a bit tricky, especially since I'm currently using GD. Changing the script's priority to run with a different one would likely require too much code for an experimental feature. I added AVIF support as a preview format in a way that it can be easily removed if it's not accepted, as, ideally, it would require a complete rewrite of the generator for proper support.

vid-bin commented 1 year ago

@JanisPlayer While this may be outside of the norm it would be nice if AVIF previews included HDR support for images that have HDR data. My understanding is the format does support HDR content.

JanisPlayer commented 1 year ago

@vid-bin Yes, AVIF can support 12-bit color depth, but I'm not sure if PHP's GD library can create such images. Whether it's practical for previews depends on the user's preferences.

I have tested whether GD can handle 12-bit AVIF images in the sense of opening them, but I don't know if it can actually generate them.

I use the Photo App Memories, which allows loading original images when zooming, and it has saved me a lot of storage.

I've just built a patch for version 27.1.2 of Nextcloud. You can test it by finding the patch under the Pull Request. To use it, simply replace the files, and your Nextcloud will be patched. Please use it only for version 27.1.2 to avoid potential errors.

I'm currently using this version, but I can't rule out the possibility of missing errors during testing. If you encounter any issues, discover new formats or settings causing problems, feel free to report them or even post code suggestions.

Regarding the Viewer App, I had difficulties building it, so I made adjustments directly in the viewer-main.js file. If you encounter any issues, please let me know. Note that replacing this file in a version newer than 2.2.0 might lead to problems. Also, keep in mind that this is not a final solution and is intended for testing purposes only. This patch will be removed automatically with the next update of your Nextcloud or apps.

vid-bin commented 1 year ago

It works and it looks like it also works on nextcloud iOS app. I haven't tested extensively and don't know if say the recognize app will function properly though.

JanisPlayer commented 1 year ago

Yes, exactly, the issue with the Recognize app is something I've also noticed when using webp or avif as the preview format. This seems to lead to an error since this app also relies on Nextcloud's Preview Generator to recognize images, and it doesn't support those formats. The current workaround is to use standard preview formats, but at least you can still create AVIF files as JPEG previews, which doesn't cause problems. However, I believe this issue is solvable.

It's good to know that the Nextcloud iOS app can handle AVIF – thanks for the information. I couldn't test it myself because I don't have an iPhone, so I'm not sure if I still need to provide screenshots for the Pull Request.

vid-bin commented 1 year ago

@JanisPlayer

Yes, so the problem is that tesseract only supports jpeg. It doesn't support any other format. I looked into this a while ago.

Recognize is going to have to modify the preview generation to force jpeg temporary files if it wants to work.

I wish fullfilesearch with elastic search backend did the same because as it currently stands even on a base nextcloud install without your modifications none of my heic images get processed.

Edit: Or maybe it does support other image formats but certainly not heic and likely not avif. It relies on the leptonic library to read images.

JanisPlayer commented 1 year ago

Yes, I'm seeing it now. The ideal solution would be to rewrite the Preview Generator so that app developers have a better way to set the output format and other options like quality. In fact, it might be better to use a custom function here rather than relying on Nextcloud's generator, especially because some programs may not handle its output properly. I mean, if a user has the wrong setting here, it results in the AI receiving a lower-quality image, which is almost a waste of resources.

Yes, it seems that HEIC and AVIF can be problematic for AI applications. It might be necessary to adjust the Preview Generator for app developers, so they can make better use of it in these cases.

vid-bin commented 1 year ago

Yes, the AI apps should be designed in such a way to generate temporary preview files of images in jpeg instead of just using the original file. This would solve a lot of problems with apps not being able to process files because of incompatible formats.

It would introduce more load onto the server but I think people who are running such apps have powerful servers anyway and they can handle the load.

JanisPlayer commented 1 year ago

I just did an experiment for fun with GD and 100% JPEG images. Yes, you can significantly improve recognition. Creating the JPEG images is extremely fast, but what bothers me is that they are written to my temp folder instead of briefly using the RAM disk. However, I think that would be an idea for the developer.

In addition, I find 100% quality to be a bit excessive; it only makes them unnecessarily large. A reduction of about 5% makes hardly any difference. :D

With this solution, all images will be regenerated, I think that would be a good idea for an experiment.

But I'm in favor of modifying the Nextcloud Preview Generator so that it can also be used for such a function. As you mentioned, most AI apps use this generator, and the quality and format cannot be truly fixed through the API.

darkBuddha commented 1 year ago

Please add AVIF support. My ImageMagick supports it but NC doesn't recognize .avif images as image so no preview is generated.

JanisPlayer commented 1 year ago

@darkBuddha this is a patch for Recognize app. Its work with AVIF and WebP previews. ;) But this is straight from the pull request, it may still be changed, but it works for me. wget https://raw.githubusercontent.com/JanisPlayer/recognize/patch-2/lib/Classifiers/Classifier.php -O /var/www/nextcloud/apps/recognize/lib/Classifiers/Classifier.php

strech345 commented 1 year ago

so what you guys think, how long will it might take until release of avif support (supported file format and preview)?

darkBuddha commented 1 year ago

or AV1?

JanisPlayer commented 1 year ago

@darkBuddha 20 => 'OC\\Preview\\Movie', AV1 videos are already supported as long as they are stored in a supported container format. Previews are created using FFmpeg in this case.

As for the support of AVIF, I can only say that I received an email stating that the pull request is in the beta list. However, I still need to clean it up and make it more efficient, or maybe even delete some things that are not currently relevant to the main version. But in essence, adding pure AVIF support for JPEG, WebP, and PNG previews is a straightforward task.

@strech345 Regarding the AVIF preview format, it appears to be more challenging. I have made an effort to adapt as many apps as possible for support, although there may still be some AI apps missing, and I'm not sure if all these apps will be willing to work with it. But there's only so much I can do. WebP faces similar difficulties with AI apps, so I think once the apps are adapted for it, AVIF should function as a preview format as well. The images also look better than WebP images at the same file size. Browser support is also on the rise, and the performance is acceptable, especially for non-high-resolution images. However, I believe this is more of an experimental feature, suitable for administrators who are familiar with it.

I might occasionally submit a test patch under the pull request, which you can apply in the meantime. This way, any errors and issues that I may have overlooked with my configuration could be discovered.

darkBuddha commented 1 year ago

Thanks Janis!

Do you know which containers are supported? I don't even get H265 thumbnails. You are referring to vanilla Nextcloud or some plugin that generates the thumbnails? I would have my doubts that FFmpeg can't read all containers... it will take also take a .mov

JanisPlayer commented 1 year ago

.MP4, .MKV, and .MOV files work for me. The crucial thing is that FFmpeg is installed, and the settings are configured in the Config. And yes, this works with the standard Nextcloud version.

darkBuddha commented 1 year ago

Then I am missing just the settings, thank you. Will check it.

Edit: It works, even AV1. Amazing. Thank you @JanisPlayer !! Can AVIF also be thumbnailed?

JanisPlayer commented 1 year ago

Can AVIF also be thumbnailed?

Yes, the patch enables that as well, at least in the current version. I've also provided instructions on how to use it in the PR under the Experimental section: To use AVIF as the preview format for all images (GD), add the following line to the configuration: 'preview_format' => 'avif', You can change the quality with the following command: occ config:app:set preview avif_quality --value="30" To enable AVIF support, you also need to add OC\Preview\AVIF to the configuration. Please be aware that using AVIF as the preview format may lead to issues with apps and browsers that do not support AVIF. Additionally, this won't work if you want to use Imaginary, as I've mentioned in the PR.

darkBuddha commented 1 year ago

I am wondering why Nextcloud isn't creating previews for my .AVIF files. Support is enabled in config. PHP has AVIF support. Nextcloud logs don't show anything. Do you maybe have an idea @JanisPlayer? 🙏🏻

JanisPlayer commented 1 year ago

This is because the standard Nextcloud does not yet allow this. You have to install the patch from me under the pool request. All you really have to do is replace all the files.Then check file permissions and restart server. So, as I said, i still working on parts of it being included in the main project.

DesertCookie commented 9 months ago

How about adding an option to allow for 10bit AVIF previews?

Compatibility-wise, since the era of H.265 and VP9, devices that supported 8bit usually also started supporting 10bit, not going the route H.264 went where 10bit was unsupported on many devices for a long time. 10bit can save bandwidth over 8bit, even with 8bit sources (at least in video streaming). Of course, encoding time for 10bit would be about 10-20% longer. If at all, there probably should be three options for high-bit-depth AVIF previews: always 8bit, always 10bit, only 10bit if source is also >8bit.

I ran some tests that clearly showed a reduction in file size for my 2048x2048 test image (downsampled to 8bit, as the 16bit TIFF messed with the image quality metrics):

Format Quality Bit-depth Size (KB) SSIM
PNG lossl. 8 7,116 -
JPEG 40 8 342 0.947
JPEG 50 8 402 0.954
JPEG 60 8 468 0.960
JPEG 70 8 568 0.967
WEBP 20 8 117 0.923
WEBP 30 8 144 0.935
WEBP 40 8 172 0.945
WEBP 50 8 196 0.951
AVIF 20 8 42 0.881
AVIF 20 10 37 0.875
AVIF 30 8 65 0.909
AVIF 30 10 60 0.906
AVIF 40 8 97 0.929
AVIF 40 10 92 0.928
AVIF 50 8 143 0.942
AVIF 50 10 138 0.943

Both WEBP and AVIF absolutely annihilated the natural grain of the test image, leading to very low scores compared to JPEG, which does a better job, but also adds horrible artefacts. This is a sort of worst-case scenario. AVIF definitely should be encoded with some grain synthesis here, which I can't do by simply exporting from GIMP; grain synthesis with AOM requires the images to be encoded in 4:4:4 though. This can only be circumvented by first building the grain profile and exporting it as a text file, then adding it back in during encoding.

Download the test images

lars-becker commented 9 months ago

@DesertCookie AVIF 8bit is already poorly supported; but 10bit creates more compatibility issues. For example baseline 8bit works on new Samsung devices out of the box, while 10 bit does not. For now, it creates probably much more support requests – especially on systems with a large user base – than the few bytes saved will be worth. Qualitywise it makes no real difference because it can't be spotted and previews should give an impression, but don't need high fidelity graphics.

JanisPlayer commented 9 months ago

@DesertCookie, yes, those are interesting results. However, it is not yet clear whether AVIF will make it into the PR as a preview format developer feature; that decision still needs to be made. You used a different encoder in your test than what is currently available to me.

For Nextcloud, there are only two options to use AVIF: GD or ImageMagick and Imaginary (eliminated), as anything else is considered too ambitious and labor-intensive for the PR. I opted for GD because it was faster and simpler, requiring less rewriting.

The encoder quality of GD is different from what you are currently accustomed to, and the available options are fewer. The documentation doesn't even specify whether 10-bit is supported. Therefore, it is interesting for the future and for my personal use, but unfortunately, it is currently challenging to implement for the PR.