search

nextcloud / previewgenerator

Nextcloud app to do preview generation in the background.
https://apps.nextcloud.com/apps/previewgenerator
GNU Affero General Public License v3.0
455 stars 57 forks source link

Large HEIC files not generating preview. #472

Closed maxi1134 closed 1 month ago

maxi1134 commented 5 months ago

I see this in the logs:

ImagickException Failed to read the file File: /maxi1134/files/InstantUpload/Camera/2024/05/20240502_192302.heic Imagick says:

Which is odd; Since the lower res pictures taken work perfectly.

The reason I use HEIC is that live motion pictures don't seem to work with jpeg/mp4 on my S24 ultra.

Thanks for reading!

st3iny commented 5 months ago

It seems like the error message was truncated. Was there any more content after Imagick says:?

Also, would it be possible for you to provide a failing test image file?

maxi1134 commented 5 months ago

Here is the complete log for this event:

{
  "reqId": "azXvJxodos9FkXs8nmx9",
  "level": 3,
  "time": "2024-05-03T14:37:24+00:00",
  "remoteAddr": "192.168.0.1",
  "user": "maxi1134",
  "app": "core",
  "method": "GET",
  "url": "/core/preview?fileId=1354415&x=1920&y=1080&a=true&etag=4ad6b185b10c282a2dccfb1f765ae137",
  "message": "File: /maxi1134/files/InstantUpload/Camera/2024/05/20240502_192302.heic Imagick says:",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:125.0) Gecko/20100101 Firefox/125.0",
  "version": "28.0.2.5",
  "exception": {
    "Exception": "ImagickException",
    "Message": "Failed to read the file",
    "Code": 1,
    "Trace": [
      {
        "file": "/var/www/html/lib/private/Preview/HEIC.php",
        "line": 116,
        "function": "readImage",
        "class": "Imagick",
        "type": "->",
        "args": [
          "/var/www/html/data/maxi1134/files/InstantUpload/Camera/2024/05/20240502_192302.heic[0]"
        ]
      },
      {
        "file": "/var/www/html/lib/private/Preview/HEIC.php",
        "line": 76,
        "function": "getResizedPreview",
        "class": "OC\\Preview\\HEIC",
        "type": "->",
        "args": [
          "/var/www/html/data/maxi1134/files/InstantUpload/Camera/2024/05/20240502_192302.heic",
          8192,
          8192
        ]
      },
      {
        "file": "/var/www/html/lib/private/Preview/GeneratorHelper.php",
        "line": 64,
        "function": "getThumbnail",
        "class": "OC\\Preview\\HEIC",
        "type": "->",
        "args": [
          [
            "OC\\Files\\Node\\File"
          ],
          8192,
          8192
        ]
      },
      {
        "file": "/var/www/html/lib/private/Preview/Generator.php",
        "line": 361,
        "function": "getThumbnail",
        "class": "OC\\Preview\\GeneratorHelper",
        "type": "->",
        "args": [
          [
            "OC\\Preview\\HEIC"
          ],
          [
            "OC\\Files\\Node\\File"
          ],
          8192,
          8192
        ]
      },
      {
        "file": "/var/www/html/lib/private/Preview/Generator.php",
        "line": 337,
        "function": "generateProviderPreview",
        "class": "OC\\Preview\\Generator",
        "type": "->",
        "args": [
          [
            "OC\\Files\\SimpleFS\\SimpleFolder"
          ],
          [
            "OC\\Files\\Node\\File"
          ],
          8192,
          8192,
          false,
          true,
          "image/heic",
          ""
        ]
      },
      {
        "file": "/var/www/html/lib/private/Preview/Generator.php",
        "line": 143,
        "function": "getMaxPreview",
        "class": "OC\\Preview\\Generator",
        "type": "->",
        "args": [
          [
            "OC\\Files\\SimpleFS\\SimpleFolder"
          ],
          [],
          [
            "OC\\Files\\Node\\File"
          ],
          "image/heic",
          ""
        ]
      },
      {
        "file": "/var/www/html/lib/private/Preview/Generator.php",
        "line": 110,
        "function": "generatePreviews",
        "class": "OC\\Preview\\Generator",
        "type": "->",
        "args": [
          [
            "OC\\Files\\Node\\File"
          ],
          [
            [
              1920,
              1080,
              false,
              "fill"
            ]
          ],
          "image/heic"
        ]
      },
      {
        "file": "/var/www/html/lib/private/PreviewManager.php",
        "line": 187,
        "function": "getPreview",
        "class": "OC\\Preview\\Generator",
        "type": "->",
        "args": [
          [
            "OC\\Files\\Node\\File"
          ],
          1920,
          1080,
          false,
          "fill",
          null
        ]
      },
      {
        "file": "/var/www/html/core/Controller/PreviewController.php",
        "line": 173,
        "function": "getPreview",
        "class": "OC\\PreviewManager",
        "type": "->",
        "args": [
          [
            "OC\\Files\\Node\\File"
          ],
          1920,
          1080,
          false,
          "fill"
        ]
      },
      {
        "file": "/var/www/html/core/Controller/PreviewController.php",
        "line": 141,
        "function": "fetchPreview",
        "class": "OC\\Core\\Controller\\PreviewController",
        "type": "->",
        "args": [
          [
            "OC\\Files\\Node\\File"
          ],
          1920,
          1080,
          true,
          true,
          "fill",
          false
        ]
      },
      {
        "file": "/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",
        "line": 230,
        "function": "getPreviewByFileId",
        "class": "OC\\Core\\Controller\\PreviewController",
        "type": "->",
        "args": [
          1354415,
          1920,
          1080,
          true,
          true,
          "fill",
          false
        ]
      },
      {
        "file": "/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",
        "line": 137,
        "function": "executeController",
        "class": "OC\\AppFramework\\Http\\Dispatcher",
        "type": "->",
        "args": [
          [
            "OC\\Core\\Controller\\PreviewController"
          ],
          "getPreviewByFileId"
        ]
      },
      {
        "file": "/var/www/html/lib/private/AppFramework/App.php",
        "line": 184,
        "function": "dispatch",
        "class": "OC\\AppFramework\\Http\\Dispatcher",
        "type": "->",
        "args": [
          [
            "OC\\Core\\Controller\\PreviewController"
          ],
          "getPreviewByFileId"
        ]
      },
      {
        "file": "/var/www/html/lib/private/Route/Router.php",
        "line": 315,
        "function": "main",
        "class": "OC\\AppFramework\\App",
        "type": "::",
        "args": [
          "OC\\Core\\Controller\\PreviewController",
          "getPreviewByFileId",
          [
            "OC\\AppFramework\\DependencyInjection\\DIContainer"
          ],
          [
            "core.Preview.getPreviewByFileId"
          ]
        ]
      },
      {
        "file": "/var/www/html/lib/base.php",
        "line": 1069,
        "function": "match",
        "class": "OC\\Route\\Router",
        "type": "->",
        "args": [
          "/core/preview"
        ]
      },
      {
        "file": "/var/www/html/index.php",
        "line": 39,
        "function": "handleRequest",
        "class": "OC",
        "type": "::",
        "args": []
      }
    ],
    "File": "/var/www/html/lib/private/Preview/HEIC.php",
    "Line": 116,
    "message": "File: /maxi1134/files/InstantUpload/Camera/2024/05/20240502_192302.heic Imagick says:",
    "exception": [],
    "CustomMessage": "File: /maxi1134/files/InstantUpload/Camera/2024/05/20240502_192302.heic Imagick says:"
  },
  "id": "6634f6a461a22"
}

And here a direct download link, to avoid any re-encoding on another platform:

https://nuage.purgatoire.ca/s/9WeZ9CMZJRXintQ

joshtrichards commented 4 months ago

Unfortunately I think that's a relatively generic error from the underlying tool (ImageMagick).

What Imagick version do you have installed?

You might try seeing what happens when using ImageMagick to access that file+layer directly from the command line.

maxi1134 commented 4 months ago

You might try seeing what happens when using ImageMagick to access that file+layer directly from the command line.

I tried running the command on the file but it says "command not found".

I am using the Apache docker container, any idea where the executable is stored? I cannot find the information!

edit: Upon trying to install the imagemagick. I see that this version is already installed: (8:6.9.11.60+dfsg-1.6+deb12u1


root@f7140ea1e7e9:/var/www/html# sudo apt install imagemagick
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
imagemagick is already the newest version (8:6.9.11.60+dfsg-1.6+deb12u1).

Edit2:

The command seems to be "convert"

Here a test with the command and an image presenting the bug


root@f7140ea1e7e9:/var/www/html/data/maxi1134/files/InstantUpload/Camera/2024/05# convert  -identify 20240502_114349.heic

convert-im6.q16: no images defined `20240502_114349.heic' @ error/convert.c/ConvertImageCommand/3229.

root@f7140ea1e7e9:/var/www/html/data/maxi1134/files/InstantUpload/Camera/2024/05#

Edit3:

I tried modifying the policy.xml for ImageMagick-6 under `/etc/ImageMagick-6/' in the latest docker image with unreasonably large limits, still not dice!


<policymap>
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <policy domain="resource" name="memory" value="256MiB"/>
  <policy domain="resource" name="map" value="512MiB"/>
  <policy domain="resource" name="width" value="160KP"/>
  <policy domain="resource" name="height" value="160KP"/>
  <!-- <policy domain="resource" name="list-length" value="128"/> -->
  <policy domain="resource" name="area" value="1000MP"/>
  <policy domain="resource" name="disk" value="1GiB"/>
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="coder" rights="none" pattern="MVG" /> -->
  <!-- <policy domain="module" rights="none" pattern="{PS,PDF,XPS}" /> -->
  <!-- <policy domain="path" rights="none" pattern="@*" /> -->
  <!-- <policy domain="cache" name="memory-map" value="anonymous"/> -->
  <!-- <policy domain="cache" name="synchronize" value="True"/> -->
  <!-- <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/>
  <!-- <policy domain="system" name="max-memory-request" value="256MiB"/> -->
  <!-- <policy domain="system" name="shred" value="2"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- <policy domain="system" name="font" value="/path/to/font.ttf"/> -->
  <!-- <policy domain="system" name="pixel-cache-memory" value="anonymous"/> -->
  <!-- <policy domain="system" name="shred" value="2"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- not needed due to the need to use explicitly by mvg: -->
  <!-- <policy domain="delegate" rights="none" pattern="MVG" /> -->
  <!-- use curl -->
  <policy domain="delegate" rights="none" pattern="URL" />
  <policy domain="delegate" rights="none" pattern="HTTPS" />
  <policy domain="delegate" rights="none" pattern="HTTP" />
  <!-- in order to avoid to get image with password text -->
  <policy domain="path" rights="none" pattern="@*"/>
  <!-- disable ghostscript format types -->
  <policy domain="coder" rights="none" pattern="PS" />
  <policy domain="coder" rights="none" pattern="PS2" />
  <policy domain="coder" rights="none" pattern="PS3" />
  <policy domain="coder" rights="none" pattern="EPS" />
  <policy domain="coder" rights="none" pattern="PDF" />
  <policy domain="coder" rights="none" pattern="XPS" />
</policymap>
maxi1134 commented 4 months ago

I seem to have fixed it.

Either this previous modification in the previous comment; Or lowering 'preview_max_memory' from 8000 to 1024

maxi1134 commented 4 months ago

image It seems like the issue is back :/


2024-05-16T16:19:12+00:00 Generating previews for /maxi1134/files/InstantUpload/Camera/2024/05/20240515_133650.heic
2024-05-16T16:19:12+00:00 Generating previews for /maxi1134/files/InstantUpload/Camera/2024/05/20240515_133658.heic
2024-05-16T16:19:12+00:00 Generating previews for /maxi1134/files/InstantUpload/Camera/2024/05/20240515_133659.heic
2024-05-16T16:19:12+00:00 Generating previews for /maxi1134/files/InstantUpload/Camera/2024/05/20240515_133700.heic
2024-05-16T16:19:13+00:00 Generating previews for /maxi1134/files/InstantUpload/Camera/2024/05/20240515_133705.heic

No errors, but I get " ImagickException Failed to read the file File: /maxi1134/files/InstantUpload/Camera/2024/05/20240515_133705.heic Imagick says: " when trying to open one

maxi1134 commented 4 months ago

It seems like the nextcloud update overwrote the "/etc/ImageMagick-6/policy.xml" file.

Changing the values to


  <policy domain="resource" name="memory" value="2560MiB"/>
  <policy domain="resource" name="map" value="5120MiB"/>
  <policy domain="resource" name="width" value="160KP"/>
  <policy domain="resource" name="height" value="160KP"/>
  <!-- <policy domain="resource" name="list-length" value="128"/> -->
  <policy domain="resource" name="area" value="2280MP"/>
  <policy domain="resource" name="disk" value="10GiB"/>

Is indeed the fix.

It seems like the default value (128MP) is limiting new Samsung phones!

It would be perhaps wise to increase those values by default to at least 250MP, as 200MP cameras are a thing on phones now.

joshtrichards commented 4 months ago

It seems like the nextcloud update overwrote the "/etc/ImageMagick-6/policy.xml" file.

I think you're running a custom Docker image. You'll have do something like either mount that file into your container and/or handle it in your Dockerfile if you want to retain it.

It would be perhaps wise to increase those values by default to at least 250MP, as 200MP cameras are a thing on phones now.

I presume the default is coming from the package, not by us. But when you update your image all that gets replaced unless you do the prior thing.

That said, it's entirely possible some doc additions (https://github.com/nextcloud/documentation) would be appropriate somewhere to suggest some reasonable numbers for people to use.

AndyXheli commented 1 month ago

Same issue on NC 30 RC1 Not running docker Ubuntu 22.04 PHP 8.3 Redis Apache2 imagemagick is already the newest version (8:6.9.11.60+dfsg-1.3ubuntu0.22.04.5+esm1). 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

{"reqId":"n5ZKVfuUurXDFGb6OnQF","level":3,"time":"2024-08-16T08:33:14-05:00","remoteAddr":"","user":"","app":"core","method":"GET","url":"/core/preview?fileId=1575189&x=250&y=250","message":"File: /username/files/Photos/2024/24-08-14 21-58-05 0598.heic Imagick says:","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36","version":"30.0.0.8","exception":{"Exception":"ImagickException","Message":"Failed to read the file","Code":1,"Trace":[{"file":"/var/www/nextcloud/lib/private/Preview/HEIC.php","line":99,"function":"pingImage","class":"Imagick","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/HEIC.php","line":56,"function":"getResizedPreview","class":"OC\\Preview\\HEIC","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/GeneratorHelper.php","line":44,"function":"getThumbnail","class":"OC\\Preview\\HEIC","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":337,"function":"getThumbnail","class":"OC\\Preview\\GeneratorHelper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":313,"function":"generateProviderPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":119,"function":"getMaxPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/Preview/Generator.php","line":86,"function":"generatePreviews","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/lib/private/PreviewManager.php","line":167,"function":"getPreview","class":"OC\\Preview\\Generator","type":"->"},{"file":"/var/www/nextcloud/core/Controller/PreviewController.php","line":159,"function":"getPreview","class":"OC\\PreviewManager","type":"->"},{"file":"/var/www/nextcloud/core/Controller/PreviewController.php","line":123,"function":"fetchPreview","class":"OC\\Core\\Controller\\PreviewController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":208,"function":"getPreviewByFileId","class":"OC\\Core\\Controller\\PreviewController","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php","line":114,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/AppFramework/App.php","line":161,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Route/Router.php","line":309,"function":"main","class":"OC\\AppFramework\\App","type":"::"},{"file":"/var/www/nextcloud/lib/base.php","line":996,"function":"match","class":"OC\\Route\\Router","type":"->"},{"file":"/var/www/nextcloud/index.php","line":24,"function":"handleRequest","class":"OC","type":"::"}],"File":"/var/www/nextcloud/lib/private/Preview/HEIC.php","Line":99,"message":"File: /username/files/Photos/2024/24-08-14 21-58-05 0598.heic Imagick says:","exception":[],"CustomMessage":"File: /username/files/Photos/2024/24-08-14 21-58-05 0598.heic Imagick says:"},"id":"66bf552f2e150"}

image

maxi1134 commented 1 month ago

It seems like the nextcloud update overwrote the "/etc/ImageMagick-6/policy.xml" file.

Changing the values to


  <policy domain="resource" name="memory" value="2560MiB"/>
  <policy domain="resource" name="map" value="5120MiB"/>
  <policy domain="resource" name="width" value="160KP"/>
  <policy domain="resource" name="height" value="160KP"/>
  <!-- <policy domain="resource" name="list-length" value="128"/> -->
  <policy domain="resource" name="area" value="2280MP"/>
  <policy domain="resource" name="disk" value="10GiB"/>

Is indeed the fix.

It seems like the default value (128MP) is limiting new Samsung phones!

It would be perhaps wise to increase those values by default to at least 250MP, as 200MP cameras are a thing on phones now.

@AndyXheli Have you tried this?

AndyXheli commented 1 month ago

Also nothing chnaged on my system only nextcloud update so why are we getting this all of a sudden never had isues with HEIC photos before

Also my files are not large

image

when i open the photo image

image

maxi1134 commented 1 month ago

Also my files are not large

I invite you to read the title of this issue.

AndyXheli commented 1 month ago

Hi @maxi1134,

Your comment seems unrelated and not necessary for this issue. The error I'm experiencing appears to be the same as the one described in the existing ticket, so creating a duplicate ticket wouldn't be helpful. If you don't have any relevant information on how to resolve this issue, please refrain from posting unrelated comments.

Thank you!

maxi1134 commented 1 month ago

Changing the values inside the "/etc/ImageMagick-6/policy.xml" file to something like


  <policy domain="resource" name="memory" value="2560MiB"/>
  <policy domain="resource" name="map" value="5120MiB"/>
  <policy domain="resource" name="width" value="160KP"/>
  <policy domain="resource" name="height" value="160KP"/>
  <!-- <policy domain="resource" name="list-length" value="128"/> -->
  <policy domain="resource" name="area" value="2280MP"/>
  <policy domain="resource" name="disk" value="10GiB"/>

Fix for those having this issue with large files above.