search

element-hq / element-meta

Shared/meta documentation and project artefacts for Element clients
75 stars 12 forks source link

[Story] EX: Resize media before sending #2543

Closed manuroe closed 2 weeks ago

manuroe commented 1 month ago

It will also improve everyone bandwidth usage.

Acceptance criteria

Summary

Image Preset Resolution File size JPEG Quality Format
Optimised 2048 ~200-650KB 0.78 JPEG/GIF/PNG
Optimised off Original Original N/A Original
Video Preset Output Format
Optimised 720p MP4
Optimised off 1080p MP4

Leads

Time sheeting

ER-97 EX: Persistent Event Cache & Upload Improvements

Size estimate

None

Dependencies

Out of scope

Open questions

### Questions
- [x] Ok for a global setting?
- [x] What should be the different quality levels and their associated resolutions?
- [x] Should we resample the audio too? --> No, it is a very specific case

Subtasks

### Android
- [ ] https://github.com/element-hq/element-x-android/issues/3651
### iOS
- [ ] https://github.com/element-hq/element-x-ios/issues/3398
### Rust
### Other
### Bugs

Sign-off

Android

iOS

pixlwave commented 1 month ago

Dumping the various Image/Video qualities I've found testing apps today. However this isn't quite as simple as these tables make it seem.

Taking signal as an example, some of the apps seem to have a selection of image presents and the user setting maps to a default preset. However, when the default preset exceeds the target file size, resizing is attempted again with a slightly worse preset until the target size is achieved. This seems somewhat overkill without properly validating a bunch of size+quality combinations to me.

Signal

Image Preset Resolution File size JPEG Quality
High 4096 3MB 0.55
Standard 2048 1.5MB 0.6
Standard (share extension) 1600 1MB 0.6
Video Preset Output
All 480p

iMessage

Image Preset Resolution File size JPEG Quality
Normal Original Up to 25MB? 0.93
Low 1280 300KB 0.86
Video Preset Output
All 720p

Whatsapp

Image Preset Resolution File size JPEG Quality
HD 5120 3MB 0.75
Standard 2048 550KB 0.8
Video Preset Output
HD 😂 720p
Standard 480p

Slack

Image Preset Resolution File size JPEG Quality
Non-optimised Passthrough Passthrough Passthrough
Optimised Original Varies 0.78
Video Preset Output
Original Passthrough
Optimised 720p

Discord

Image Preset Resolution File size JPEG Quality
All Original Varies 0.63
Video Preset Output
All 480p
f35f0ef9d0e827dae86552d3899f78fc commented 1 month ago

Instead of targeting a byte size, how about the way the older Element iOS app behaved? I believe it used a constant calculation (simpler) and offered Small, Medium, or Large choices to the user. Each size choice corresponded to a relatively static percentage reduction of the image's dimensions.

I am not a developer, more like a guru, but I think this linked code is related.

pixlwave commented 1 month ago

Instead of targeting a byte size

Yeah don't worry, this seems like an unpredictable "quality" setting to me. The post above is only illustrative of how other apps work as some background research.

ara4n commented 1 month ago

while i'm very happy to see that we may let users choose whether to optimise or not... i'm worried that particularly for video, we're giving the option of 480p (awful) and original (which could well be gigantic 4K@60fps on an iPhone 16).

Can i make a plea to offer both "low", "high" and "original" or something? So 480p, 720p and original in the case of video (similar to what WhatsApp is doing)? Otherwise I can see us being drowned in a 50/50 mix of "why is Element's video quality so crap?" and "why are Element's videos so big?" bugs.

pixlwave commented 1 month ago

Just to clarify, we don't upload 4K video, it has always been processed using Apple's AVAssetExportPreset1920x1080 preset before being handed to the SDK.

mxandreas commented 1 month ago

Otherwise I can see us being drowned in a 50/50 mix of "why is Element's video quality so crap?"

I agree with this; I think these days most users value/expect high-quality more than their bandwidth. I would not add a 3rd option - just use 720 instead of 480. We can always add 3rd option later if this becomes necessary.

@ara4n It is worth mentioning though that 480 was picked because this is what everybody else was doing.

fibcom-bp commented 3 weeks ago

@pixlwave thanks for the comparrision to other messengers. What is the reason for EX having the optimization turned off by default although the others have it turned on by default? Personally i would like to have optimization turned ON by default. This saves a lot of storage/bandwidth and is what the users are used to from their personal messenger apps.

ricardo-duarte-av commented 3 weeks ago

A few questions if i may:

1. Won't it be a bit deceiving to users that, with optimize OFF a video will still be tampered with ? Worse, It's stated that it will be resized to 1080p MP4. MP4 is a container, if i record in h265 i'll have my files being converted (silently) to h264. Worse still, we might not even see that great of file size reduction:

image

(this is highly subjective to encoding settings and the actual media being encoded)

  1. Having this be a global setting isn't too limiting ? Perhaps a toggle before upload to Disable optimization (if it's enable globally) or Enable optimization (if disabled globally) would be preferable ?

pixlwave commented 3 weeks ago

@pixlwave thanks for the comparrision to other messengers. What is the reason for EX having the optimization turned off by default although the others have it turned on by default? Personally i would like to have optimization turned ON by default. This saves a lot of storage/bandwidth and is what the users are used to from their personal messenger apps.

It's been decided to default this to on (the initial plan was mine as iMessage (and Slack iirc) defaults to off and it seemed undesirable for a user to be sending reduced quality images without knowing this, but as I was the only person who thinks this, we changed the plan 😢)

Won't it be a bit deceiving to users that, with optimize OFF a video will still be tampered with ? Worse, It's stated that it will be resized to 1080p MP4. MP4 is a container, if i record in h265 i'll have my files being converted (silently) to h264. Worse still, we might not even see that great of file size reduction:

The TL;DR on this is that h.264 is essentially universally compatible, whereas h.265/VP9/AV1 don't have such ubiquitous support. So by always running the video through a 1080p encoding, we ensure reasonable file size (as compared to 4K), along with much wider device compatibility. So yes, we might end up with increased file sizes in some instances, but this comes with the benefit that everybody should be able to play the video. (We have been sending videos like this from EXI since we added media uploads and we've not had any complaints about it. As an aside, it is also an easy way to make sure that any GPS location data is removed from uploaded videos without having to handle all the individual video containers individually).

Having this be a global setting isn't too limiting ? Perhaps a toggle before upload to Disable optimization (if it's enable globally) or Enable optimization (if disabled globally) would be preferable ?

This is very much a first pass at supporting smaller file uploads and we've already discussed internally that we're likely to come back to it to look at such enhancements in the future as there are clearly various tradeoffs being made here.

daedric7 commented 3 weeks ago

It's been decided to default this to on (the initial plan was mine as iMessage (and Slack iirc) defaults to off and it seemed undesirable for a user to be sending reduced quality images without knowing this, but as I was the only person who thinks this, we changed the plan 😢)

You aren't. I'm also not against compression/transcoding, just agains the silent type :)

The TL;DR on this is that h.264 is essentially universally compatible, whereas h.265/VP9/AV1 don't have such ubiquitous support. So by always running the video through a 1080p encoding, we ensure reasonable file size (as compared to 4K), along with much wider device compatibility. So yes, we might end up with increased file sizes in some instances, but this comes with the benefit that everybody should be able to play the video. (We have been sending videos like this from EXI since we added media uploads and we've not had any complaints about it. As an aside, it is also an easy way to make sure that any GPS location data is removed from uploaded videos without having to handle all the individual video containers individually).

Solid facts, h264 has become the jpeg of video. Yes there are better formats, but support is not so great. As for no complains in EXI, i assure you there were at least observations on EXA (i did notice that my screen recordings became... different). The removing of private info from EXIF and others is a must have so no arguments there.

This is very much a first pass at supporting smaller file uploads and we've already discussed internally that we're likely to come back to it to look at such enhancements in the future as there are clearly various tradeoffs being made here.

I think i should say that i didn't feel that this was decided by the flip of a coin. Discussion must have been held, arguments on the pros and the cons must have been put forward and the best decision, at the time, with the info that was available, and the time available to decide was set in code.

As a server owner, i should be preoccupied with the amount of space media will take, and as such compression/transcoding is a must have. The media upload (and download) screens are currently very bland and optionless, i hope in the future we get more info transmited to the user during the preview and before the upload, as well as some options regarding how media should be handled localside (oh please let me draw on images!!) before being uploaded :)

manuroe commented 2 weeks ago

All done 🎉 . It is going to be available in the next release of EX.