Media compression on upload

[zeerooth]

Currently kitsune's upload backend just takes the file and (if it's an allowed filetype) places it in the specified storage backend. This behavior is of course correct and acceptable, but I'd be nice to implement an optional feature of image/video/audio compression on upload so as to save on storage space and bandwidth.

At this moment AV1 is probably the most effective encoding format that's suitable both for image and video and supported by (almost) all major browsers + there are a few crates in the rust ecosystem that deal with AV1, with the most notable being rav1e.

Edit: for more clarification, I'm referring here to both AV1 and AVIF (which is based on AV1). rav1e crate is only for video, so for image encoding we'd need to use a different crate like cavif-rs.

[aumetra]

That's something we could shove into the AttachmentService and run on our rayon threadpool we already utilize for password hashing.

(Maybe we could create a separate threadpool in the future, so that password hashing and verification aren't impacted by large attachments?)

[aumetra]

But I think Safari is still lacking AVIF support, so maybe we should convert into something like WebP for now?

Edit: Reference

[zeerooth]

But I think Safari is still lacking AVIF support, so maybe we should convert into something like WebP for now?

Edit: Reference

Safari supports AVIF but supports AV1 video decoding only on devices that have the hardware decoder, unfortunately, but that means that the newest Apple devices should work. Edge is more of a problem I think, cause there it only works if you enable feature flag, which is not on by default. Anyhow, I think that in both cases it's just a matter of time that the support gets added, but the question is just whether it's a matter of weeks or months?

[zeerooth]

WebP would also work, most definitely, but in the end AVIF is just about superior in just about every measure, though we could implement different backends for now and later switch the default, that's fine, with proper abstractions this shouldn't be an issue. Or just wait for some time and add media compression at the time that Safari and Edge fully embrace AV1/AVIF (hopefully not too long from now lol).

[aumetra]

Yeah, should be doable. We could probably throw it into another crate with a nice abstraction over a Bytes stream and just process it via a rayon threadpool and some Tokio oneshots to preserve asynchronicity

[aumetra]

Update about the threadpool situation: I added the kitsune-blocking crate with separate threadpools for cryptographic work and general CPU work (PR: #382).

The internals make it easy to define another threadpool, so we can just define a separate threadpool for media compression (if desired; otherwise we can just use the general purpose pool).

[aumetra]

And about the format question: Not sure if we should even wait (or use WebP), we could also just say "Screw Edge" and convert to AVIF unconditionally. Then again, I'm also not sure how well certain Mastodon clients support AVIF?

I'm assuming the support there is OS dependent?

[zeerooth]

Depends on the client, any web client will be dependent on browser compatibility, clients based on electron use chromium under the hood so AVIF should work but for others? FediLab for example, the client for Android that I use seems like it doesn't support AVIF yet but others might.

I may be a bit biased as I'm a big fan of the AV1/AVIF project in general but I think that it's better to already focus on this objectively better format as we'll have to make a switch anyway and it's going to be some time until kitsune gets a stable release probably. Also I think it'd be a cool feature that differenciates kitsune from most other fedi software and potentially help owners who have limited storage capabilities.