Support HEIC images

[sumnerevans]

Steps to reproduce

1. Send a HEIC image to a room. (Mimetype should be "public.heic".)

Example event source (some parts redacted):

``` { "room_id": "!redacted:beeper.com", "type": "m.room.message", "content": { "body": "ima_355e641", "info": { "size": 1182567, "mimetype": "public.heic", "w": 3024, "thumbnail_file": { "iv": "REDACTED", "hashes": { "sha256": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" }, "key": { "k": "REDACTED", "ext": true, "key_ops": [ "encrypt", "decrypt" ], "alg": "A256CTR", "kty": "oct" }, "url": "mxc://beeper.com/0000000000000000000000000000000000000000", "v": "v2" }, "h": 4032 }, "file": { "iv": "REDACTED", "hashes": { "sha256": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" }, "key": { "k": "REDACTED", "ext": true, "key_ops": [ "encrypt", "decrypt" ], "alg": "A256CTR", "kty": "oct" }, "url": "mxc://beeper.com/0000000000000000000000000000000000000000", "v": "v2" }, "msgtype": "m.image" } } ```

Outcome

What did you expect?

I expect that when I click on the thumbnail, the full image is shown.

What happened instead?

When I click on the thumbnail, the full image is not shown (blank screen)

Operating system

NixOS

Application version

Element version: 1.9.2, Olm version: 3.2.3

How did you install the app?

From Nixpkgs

Homeserver

nevarro.space

Will you send logs?

No

[turt2live]

Unfortunately we're limited to browser support on this (the desktop app is essentially a Chrome): https://caniuse.com/heif

[sumnerevans]

Are you open to solutions that use https://github.com/strukturag/libheif or similar?

[turt2live]

assuming it can be included in the web app with minimal performance overhead, sure.

[Pecacheu]

Why is this closed? Is there anything wrong with the above suggestion of using one of now numerous open-source JS libraries available to convert (perhaps at time of upload) or draw (pre-render it on a Canvas as the chat loads then cache locally) HEIC images, such as the one suggested above? Here's another suggestion I found myself, a converter in this case: https://github.com/alexcorvi/heic2any

And this isn't some obscure format by the way. All Samsung flagships now use HEIC out of the box. I recently encountered this problem when trying to upload an HEIC pic I just took on my phone. Without HEIC support (Which, by the way, all other major chat platforms seem to have full support for, including Telegram Web) there is no way for most mobile users of newer phones (default Samsung, default iPhone, optional on LG/Pixel, probably others) to upload photos without using a clunky JPG converter app.

Personally I have done something similar in apps I've worked on but we were doing it server-side using ImageMagick, probably not feasible for E2EE of course lol

[t3chguy]

Telegram Web doesn't really count given that due to lack of encryption the server can do the conversion.

[Pecacheu]

Perhaps. It would be weird if there were features supported in a normal chat that aren't supported in an E2E chat though. Guess I will have to test if uploading HEIC is possible in an E2E Telegram chat, don't know if I've tried that before.

EDIT: Yeah it totally worked, just gave it a test with a friend. Private chat seems to be fully featured. Tested sending videos (w/ video editor, which is also ALL done client-side as is obvious by the fact that it only uploads after you're done editing), music, gifs, photo albums, HEIC and WEBP, stickers, and reactions.

And according to Telegram they can't read attachments in a private chat and their E2EE is fully client-side (it can't even sync across devices). So yes, they must be doing the conversion client side.

[ara4n]

reopening as this is a pain, and it could be fudged clientside using libheif

[turt2live]

I don't think we can load libheif in the browser though?

When I last looked it either increased our bundle size dramatically or didn't work.

[t3chguy]

https://www.npmjs.com/package/libheif-js looks to be a thing for the browser

[turt2live]

LGPL-2.0 though, which iirc is a license incompatibility.

[Pecacheu]

I'm not a license expert but isn't LGPL meant to be more inclusive of integration into projects with other open-source licenses than regular GPL?

(I wouldn't be the one to ask about the specifics of if it's compatible with Apache 2 though. I've just heard the "Linux heads" in my area complain about Apache licenses a lot, so I avoid them for my projects.)

[turt2live]

it's mostly compatible, but the "same license" clause causes issues. We'd be forced to look at re-licensing or dual licensing, neither of which are fun.

[Pecacheu]

Ah gotcha. The other library I linked earlier, heic2any is under MIT instead, not sure if that is any easier.

[t3chguy]

"mimetype": "public.heic",

This isn't the right mime type ftr, this is the Uniform Type Identifier, the mime types are image/heif, image/heif-sequence; image/heic, image/heic-sequence; image/avif, image/avif-sequence https://mimetype.io/image/heic

[tbertels]

LGPL-2.0 though, which iirc is a license incompatibility.

It's actually LGPL-3.0 (was the npmjs website wrong?), which is supposedly compatible.

[shmerl]

HEIC is pretty much DOA because browsers don't support it and it's not freely licensed format. Use AVIF instead (AV1 image in HEIF container).

[Pecacheu]

@shmerl No need to be a nay-sayer, there's a difference between "It's not easy" and "It's completely DOA, not possible." Frankly, sounds like the licencing is the biggest issue so far rather than a technical challenge, but there are multiple options for libraries under various different licences.

And don't ask us as to why major device vendors chose this format, write to them to complain lol, cause I'd love if they used a more open format too, but that's not the case yet.

Fact is, HEIC is used by many major smartphones, and every major OS has native support now (Windows, MacOS, Ubuntu by default too now, probably other distros, and of course Android and iOS have native support and native conversion APIs.) AV1 native support on the other hand is still... Extremely spotty at best.

For common formats that work natively in image viewers, users expect to upload them without any extra steps, and will be confused when it doesn't work. Funnily enough, I recently was asked to implement HEIC support in an app I'm working on too (an inventory catalogue system for work, which needs to support image uploading). Granted, I did that server-side with ImageMagick.

But as we've discussed, client side isn't an impossibility. In fact it seems to be a decently common solution in existing web apps. There's various articles like this about it.

As far as I can tell, seems like heic2any may be an option, as it's licenced under MIT. According to this thread it should be compatible with Apache 2.0.

[shmerl]

Problem is that it's patent encumbered. Even using the open source decoder opens you to liability to patent trolls who control said patents.

[Pecacheu]

So is H.264 and H.265 (which is what HEIC uses), both of which have wide browser support at this point. I believe Chromium can only do the latter in hardware but Firefox can do both in software. I.e. the decoder is built into an open source project directly. Plus, there's projects like FFMPEG and ImageMagick, which contain directly bundled decoders and encoders for TONS of proprietary formats, including HEIC, so this is nothing unprecedented.

Now from what I've read, the reason these projects are able to get away with this is that the parents only seem to cover hardware implementations. As long as software implementations avoid certain specific patented features, and don't straight up steal code, they are 100% free to licence their decoders/encoders however they wish. It's especially true for decoders because there's already less grounds to sue over a decoder than an encoder just due to the nature of how they work.

[shmerl]

I think H.264 is backed by Cisco for browsers to use, so they provide defense. No one does it for H.265 however. That's why it will never be supported let's say in Firefox.

Projects like ffmpeg implement it, yes. But they aren't offering any defense against patent aggressors.

[Pecacheu]

Hmm, you're right on Firefox. I don't use it normally, and I recall H.265 working when I did, as it does in Edge on my main PC... But if I open an H.265 MP4 from my phone on my older Linux PC (Intel CPU old enough for no hardware decoder), it just plays the audio... Video is just black. VLC on the same machine however, along with the default video player that came with my distro, plays it fine, so just be using software decoding. I can also view HEIC pictures on that machine.

Also I dunno where you get the Cisco thing, but H.264 doesn't seem to be any more protected than H.265 from anything I can find. Similar patent pools exist, though only one of them as they all combined (which makes it easier to license). But there's also no precedence for lawsuits over H.265 for purely software, only hardware/chipset vendors have been sued or threatened and they were using closed-source encoders/decoders.

Theoretically it could maybe happen, but the same is the case for H.264. I checked numerous sources and the answer seems to be, no one knows if x264 is patent infringement, not even the devs of it. x265 is the same way, and is even built in x264's source code.

But the minimum threshold for licencing fees is quite high in both cases, such as a subscription service needs to have subscriber count >100,000 before fees apply. Also both the patent pools for AVC and HEVC have specific exceptions for "free internet broadcast applications," which will never incur a licence fee. I'm not a legal expert so I dunno what they mean by that, but that's specifically something the x264 devs cite as applying to them, so the same should be the case for x265 because the same stipulation exists.

Also since the images would be converted on the client end and servers wouldn't be storing HEVC files, I don't think the subscription service thing applies even if you count the element paid plan.

P.S. apparently the official reason the Firefox dropped HEVC, which they apparently did support at one point, is just "it's not in the web standards." I'm sure they also didn't like having a proprietary format, but that's their official reason, simple as that.

[shmerl]

About Cisco, see: https://blogs.cisco.com/collaboration/ciscos-openh264-now-part-of-firefox

[G2G2G2G]

@Pecacheu h.265 doesn't actually work on edge. it falls back to the OS, because microsoft thinks it's a good idea to open the browser up since they <3 viruses https://apps.microsoft.com/store/detail/hevc-video-extensions-from-device-manufacturer/9N4WGH0Z6VHQ?hl=en-us&gl=us which is why https://caniuse.com/avif is broken unless you install their av1 extension

h.264 can be used for non profitable web use, h.265 can't and has lots of more royalties as they basically had the industry on lock with h.264.. which is why so many opted for alternate solutions and we had vp8, vp9, av1, and alliance for open media all because of h.265 royalty issues, no other reason.

Also I dunno where you get the Cisco thing

have you maybe tried looking at your browser plugins in the past 20 years? It has always been there. Before this was a different one, provided by cisco still. They seemingly are the only ones that gave a shit to allow one codec to be used everywhere.

anyway heif isn't going to happen. neither is jpeg xl from the looks of it, sadly. It'll probably happen down the road in 5+ years though if all the other image industry keeps pushing it. Heif won't though, it has no steam at all.

so avif and av1 webm is all we got.

[Tronic]

Element gets stuck with a loading spinner on a HEIC image. I suppose it attempts some conversion but my browser supports the image directly. Could you just give the original image in img?

[shmerl]

I'd say users shouldn't be encouraged to use HEIC images in the first place - they'll never get browser support. Use AVIF instead if you want modern compression and quality. HEIC is basically DOA.

[t3chguy]

I suppose it attempts some conversion

Element does no image conversion as it is purely an app that runs in your browser and doesn't bundle any of the rather huge image decoding libraries available.

but my browser supports the image directly.

Only Safari claims to support HEIC https://caniuse.com/?search=HEIC

Could you just give the original image in img?

That is exactly what we do, you can inspect element and see for yourself.

[Tronic]

Okay, sorry for the misguided report. Despite the forever spinner, I could download and then open the image (after replacing the plain uuid name with a proper extension).