Craft 4 beta, GQL: "Malformed UTF-8 characters, possibly incorrectly encoded"

[tremby]

I'm running Craft 4 beta and querying via GraphQL.

My assets are in as S3 volume. I also have Imager-X installed, which is set up to use Imgix.

When trying to select blurhashUri: url @assetToBlurHash on an asset, I am getting this:

{
  "error": "Malformed UTF-8 characters, possibly incorrectly encoded",
  "exception": "yii\\base\\InvalidArgumentException",
  "file": "/srv/my-project/vendor/yiisoft/yii2/helpers/BaseJson.php",
  "line": 147,
  "trace": [
    {
      "file": "/srv/my-project/vendor/yiisoft/yii2/helpers/BaseJson.php",
      "line": 88,
      "function": "handleJsonError",
      "class": "yii\\helpers\\BaseJson",
      "type": "::"
    },
    {
      "file": "/srv/my-project/vendor/yiisoft/yii2/web/JsonResponseFormatter.php",
      "line": 133,
      "function": "encode",
      "class": "yii\\helpers\\BaseJson",
      "type": "::"
    },
    {
      "file": "/srv/my-project/vendor/yiisoft/yii2/web/JsonResponseFormatter.php",
      "line": 112,
      "function": "formatJson",
      "class": "yii\\web\\JsonResponseFormatter",
      "type": "->"
    },
    {
      "file": "/srv/my-project/vendor/yiisoft/yii2/web/Response.php",
      "line": 1095,
      "function": "format",
      "class": "yii\\web\\JsonResponseFormatter",
      "type": "->"
    },
    {
      "file": "/srv/my-project/vendor/craftcms/cms/src/web/Response.php",
      "line": 275,
      "function": "prepare",
      "class": "yii\\web\\Response",
      "type": "->"
    },
    {
      "file": "/srv/my-project/vendor/yiisoft/yii2/web/Response.php",
      "line": 339,
      "function": "prepare",
      "class": "craft\\web\\Response",
      "type": "->"
    },
    {
      "file": "/srv/my-project/vendor/yiisoft/yii2/base/Application.php",
      "line": 390,
      "function": "send",
      "class": "yii\\web\\Response",
      "type": "->"
    },
    {
      "file": "/srv/my-project/web/index.php",
      "line": 12,
      "function": "run",
      "class": "yii\\base\\Application",
      "type": "->"
    }
  ]
}

I have the same error if I pass (asUri: false).

Side question: are the results cached anywhere? Or is the full-size image going to be pulled from S3 and reprocessed every time I query for the blurhash of a specific asset?

[tremby]

I had a dig and it looks like it's burning through the memory limit. The image I'm testing on is large, but not ridiculous for a source image -- it's 6000x4000. I tried on a much smaller image (2727x1818) and I get the same thing. I increased memory limit from 128M to 512M and I get the same thing.

[tomkiss]

Thanks for the issue @tremby - will take a look when I can.

[tomkiss]

Sorry, to answer your question - yes, the results are cached. A cache hash is generated using some of the images properties, so it might be regenerated if the dimensions change, for example.

I have one follow-up question- is this happening with just one image, or are you returning many results in your GQL?

If you're able to share a link to the image at all, it might be helpful.

[tremby]

It's when I query for just one at a time. I always prototype with kittens -- how else to stay happy -- so the two images I'm using are the full res versions of these two:

• https://unsplash.com/photos/RCfi7vgJjUY
• https://unsplash.com/photos/lpgAlv8I7V8

Whichever one I query for, I get an out of memory error.

[tomkiss]

Thanks for that!

That does seem like way too much memory. The image itself will use up about 120MB when loaded for the first time, which is a bit chunky but I'm not sure if there's anything that can be done about that - it needs to be loaded in order to be read.

It's hard for me to test though, as I don't have the Imager X, S3 and Imgix setup at my end. So, I'm assuming that it's the combination of those plugins that's maybe causing Craft blurhash to fail somewhere.

Ideally I'd fix this. However, Imager X supports (the same!) blurhash - have you tried that? Perhaps it's worth giving that a spin and seeing what results you get.

[tremby]

Thanks for pointing that out -- I hadn't noticed it. That does seem to give a blurhash back.

I was actually hoping to get a data URI (Nextjs's image component doesn't directly support blurhash and I'd rather the BE decode the blurhash rather than add yet another client lib), but seemingly by design Imager X won't let me get a data URI when Imgix is enabled, even though it still does have access to the original via S3. But that's not a you problem -- I'll likely open a ticket with Imager X.

I'd still say there's a bug here. If I find time I'll try disabling the various other plugins one by one and see if I still get the issue.

[tomkiss]

Hmmm. Ok, I've reproduced the memory error at my end. On an empty Craft 4 Beta install running in nitro, I did this by:

• Installing imager-x
• Disabling craft-blurhash
• Running the following imager-x markup on an asset:

{% set transformedImage = craft.imagerx.transformImage(image, { width: 1200 }) %}
{% set blurhash = transformedImage.getBlurhash() %}

...I then get a PHP error of memory usage exceeded at about 530MB (with the blurhash kornrunner lib being the last thing to try and use memory).

• If I don't use imager-x and I just use craft-blurhash, I don't get any memory errors and I can generate the blurhash image.
• If I try to pass the transformedImage value to my blurhash function or filter, it doesn't work, as the plugin only accepts Craft Assets and not imager-x objects.

I notice that, searching for blur hash in the imager-x GitHub issues, the same malformed utf-8 characters error was reported. That user doesn't say if they had imgix in use - although there is another issue recently asking about support for imgix's own blurhash feature. Seems like blurhash is available at each layer here, but the crux of the issue may lie with imager-x?

[clieee]

I can confirm it is not working as expected on Craft Pro 3.7.38. If I use a rather "low resolution" image it will work, but if I pick a larger dimensions it will fail, and thereby give me the same error as stated before.

EDIT: Im only using your plugin, and the uploaded images are stored on AWS.

[tomkiss]

Hi @clieee.

Hmmm, I can't test on S3, but I have tested with images stored on GCP, running Craft 4 in nitro locally with no apparent issue. Testing with a chunky 120 megapixel image (requiring approx 12,000 x 10,000 x 4 channels = ~480MB to load).

What is your memory limit? If you try to process an image that requires more memory than you have available, then it will fail and I'm not sure there is anything I can do about that.

[clieee]

Okay, thanks for testing. I've managed to get it to work on the remote server, but locally using Docker it seems to fail. Not sure if I need to adjust some settings on docker in order to get it to work?

[tremby]

It could well be that memory limit. PHP's default memory limit is 128M, and unless you jump through some hoops that's what it'll be in your docker container.

Does the entire decompressed bitmap really need to load into memory for this operation? If resize transformations work without hitting the memory limit, those algorithms must somehow avoid doing that.

[tremby]

If I'm right that resize transformations work without hitting the memory limit, that'll be the solution. Beyond some size threshold, do a resize and then run the blurhash operation. The documentation specifically says to do this, in fact.

[tomkiss]

Yeah, if you've got 128MB memory you will struggle. Craft's minimum required memory is 256MB with 512MB being the recommended amount.

At present, the "entire" image does need to be loaded in to memory (even for it to then be immediately resized to a tiny sample, which is all that's needed).

Craft image transforms work in a different way, as they are queued and performed as tasks in the background. This plugin parses the images "on demand", although it does cache the result.

One way to improve this would be to create the blurhash on upload, and to store that with the image.

[tremby]

Yeah, if you've got 128MB memory you will struggle. Craft's minimum required memory is 256MB with 512MB being the recommended amount.

At present, the "entire" image does need to be loaded in to memory (even for it to then be immediately resized to a tiny sample, which is all that's needed).

I'm also running in Docker, and everything was fine with the default 128M limit until I started trying out craft-blurhash. I only increased it once I started seeing this error.

My 6000x4000 image should take up no more than about 90M decompressed (assuming it uses 4 bytes per pixel even if there's no transparency). If I'm still hitting a 512M limit surely something is going wrong. Perhaps the blurhash algorithm is memory-hungry? Is the image being resized before being run through blurhash?

[tomkiss]

My 6000x4000 image should take up no more than about 90M decompressed (assuming it uses 4 bytes per pixel even if there's no transparency). If I'm still hitting a 512M limit surely something is going wrong. Perhaps the blurhash algorithm is memory-hungry? Is the image being resized before being run through blurhash?

The first thing this plugin does is resize the image to 64x64 sample, before it does anything else. But to do that it does need to load the image in to memory. It does this on its own, independently so if there are other plugins loading the image or doing anything else to the image, then that is going to easily multiply the memory being used (at least, on the first hit).

Did you see my note above about how I managed to reproduce the same error with imager-x? If you can, it might be worth removing imager-x on an instance to see if you get the error with running craft-blurhash alone. I can't quite fathom if there's some sort of a conflict with the two running together.

[clieee]

So, about the transform. Is this something you could work with? https://docs.craftcms.com/api/v3/craft-elements-asset.html#public-methods

$generateNow ([boolean (opens new window)](https://php.net/language.types.boolean), [null (opens new window)](https://php.net/language.types.null)) – Whether the transformed image should be generated immediately if it doesn’t exist. If null, it will be left up to the generateTransformsBeforePageLoad config setting.

Since my thought is, if when query for an image url (if we forget about your plugin for a bit), with a specific transform, then I get back a url for that specific image transform size which means then it needs to exist, right? Thereby you should probably be able to use an image transform before you start loading the image to hash it?

[tomkiss]

This would be the way to go about getting a transform from the built-in api (albeit the newer getTransformUrl). Although you will get a URL returned, even if it's generated immediately, it doesn't necessarily mean the image exists (it could still be a redirection URL). afaik "Immediately" simply means "put this to the front of the queue".

I spent a bit of time previously trying to use native transforms but wasn't able to do it. I can generate the transformed image fine, but loading it and reading the data proved more difficult, hence the direct approach of working with the source asset and reading that. I can take another stab at it, I'm just not that confident on success, at least based on my previous attempts.

[clieee]

Got it. I can tell you now it's working on my local machine, it was a matter of libjpeg installation order within my docker image.

Anyhow - would it be possible you think to specify the size of the generated base64 image? It can default to 64, but maybe if you could add that as a parameter, if I would like to have it even smaller?

[tomkiss]

Great, glad to hear the plugin is working now for you.

The size of the image is limited to largest dimension 64 (with aspect ratio determining the other dimension). I played around with other sizes but in the end chose 64 because I felt it was the best quality / smallest size. I could add an override fairy easily enough.

[clieee]

Okay cool. It would be nice to add an override. Please let me know once you have time to do so!

[tomkiss]

Hey @clieee - OK so this is done now in 1.2.5 for Craft 3 and 2.0.1 for Craft 4, which adds support for settings config - please check config.php for details. 😀

Based on the discussion above, I'm now going to close this issue.