SHA1 value for images wrong in neos_flow_resourcemanagement_persistentresource db table

Weissheiten commented 6 years ago

Description

When trying to import a site exported on a live environment on a dev environment the import breaks with the following error:

Exception #1300360480: Error: During import an exception occurred: "Could not convert target type "Neos\Media\Domain\Model\Image", at property path "resource": The source SHA1 did not match the SHA1 of the imported resource.".

Steps to Reproduce

I did not find the time yet try reproducing this on another setup at the moment, for the current environment (live and dev environment both with Neos 3, PHP 7.2)

Export the current site (does not matter if with --package-key flag or as specific file destination)
Transfer the file (does not matter if transfer happens via FTP client or wget in dev container)
Import the site with the site:import command

Expected behavior

A working import of the site

Actual behavior

The import breaks on the first file (Resource) which is over a certain file size (approximately 2 MB). Usually the SHA1 of the file equals the file name AND the information in the sites.xml json information for said image. By running a foreach loop on all files in the Resources directory which was created by the export command I was able to filter which files do not match the expected behavior (expected: sha1_file($filename) is equal to the filename) The only pattern I was able to identify so far is the file size.

Other tests which I performed to which the problem does NOT seem to be connected with:

to the transfer
to the PHP version (only difference on my latest test was the bugfix version)
to choosing an export via --package-key or --filename (same result)
to exporting tidy or non-tidy
to shared directories in docker (shared/non-shared directories for the import source did net the same result - I even tried disabling all sharing)
to specific Node Types using it (error occured for a Node based on "Image" - ResponsiveImage, but also for a News Node after removing the formerly from the XML)
to the filename, a new Test with the filename "insänely stupid fülenaîme.jpg" did result in the correct hash

Affected Versions

Neos: tested with 3.3.11

Flow: 4.3.6

You can get this information by running composer show or using the package management module within Neos.

Weissheiten commented 6 years ago

Further investigation lets me think that this is not related to the export/import, but to a wrong SHA1 entry in the database for certain images. I was unable to reproduce the issue with a 5.5 mb file however, the issue title has been updated accordingly though.

Weissheiten commented 6 years ago

The problem seems to be related to some edge case, even when trying to reproduce the error by downloading the exact same image from the media library and re-uploading it - Neos saves it under the "correct" SHA1. My best guess is that there was "something wrong with SHA1 creation for images" in a former Neos version or on my shared host and I am now stuck with images in the media library possessing the wrong SHA1

dfeyer commented 6 years ago

@Weissheiten Did you use any package for image optimization ? Like the one provided my MOC ? I think that with a wrong configuration you can run the optimization on the original image and in this case you change the sha1 of the file, and break the export/import as a side effect.

Weissheiten commented 6 years ago

@dfeyer I thought about this and it sounds feasible, however this would not explain why this only occurs with certain images (about 40 %) for which I didn't find a pattern yet. In addition some of the images (not all) which have been uploaded before using the MOC package are affected. (they could have been re-rendered but then again - I would expect that this then hits all of them?) I did check the file types - the wrong SHA1 occurs and also does not occur for JPGs (does not seem related to file types)

Weissheiten commented 6 years ago

Current config:

MOC:
  ImageOptimizer:
    useGlobalBinary: true
    globalBinaryPath: /path/bin
    formats:
      image/jpeg:
        enabled: true
        useGlobalBinary: false
        library: jpegtran
        binaryPath: jpegtran
       arguments: '${''-copy none -optimize '' + (progressive ? ''-progressive '' : '''') + ''-outfile '' + file + '' '' + file}'
        parameters:
          progressive: true
      image/png:
        enabled: true
        useGlobalBinary: false
        library: optipng
        binaryPath: /path/optipng
        arguments: '${''-o'' + optimizationLevel + '' -strip all -out '' + file + '' '' + file}'
        parameters:
          optimizationLevel: 2
      image/gif:
        enabled: false
        useGlobalBinary: false
        library: gifsicle
        binaryPath: path/gifsicle
        arguments: '${''--batch -O'' + optimizationLevel + '' '' + file}'
        parameters:
          optimizationLevel: 2
      image/svg+xml:
        enabled: false
        useGlobalBinary: false
        library: svgo
        binaryPath: path/svgo
        arguments: '${(pretty ? ''--pretty '' : '''') + file}'
        parameters:
          pretty: false

Weissheiten commented 6 years ago

Still experiencing the problem, after an update to the latest Neos 3 release. The SHA1 of some files editors upload is "wrong" (saved sha1 by Neos != sha1 of the file), they use the ReactUI to upload the images, the image is part of an own nodetype. When downloading the image in the media browser and replacing it, the SHA1 is correct. (File SHA1 == stored sha1 in the db)

neos / neos-development-collection