search

fluttercandies / flutter_photo_manager

A Flutter plugin that provides images, videos, and audio abstraction management APIs without interface integration, available on Android, iOS, macOS and OpenHarmony.
https://pub.dev/packages/photo_manager
Apache License 2.0
669 stars 302 forks source link

[BUG] Video AssetImage.file loads extremely long on iOS even though isLocal == true. AssetImage.originFile loads instantly #704

Closed timcreatedit closed 2 years ago

timcreatedit commented 2 years ago

Describe the bug We're creating an app where we need a custom video picker, however loading the video files via photo_manager is extremely slow on iOS. When obtaining the file for a video for the first time, a 3s video can load for upwards of 5 seconds. For longer videos the loading time is even worse (>1min in some cases), making our app almost unusable on iOS at the moment. Obtaining files through AssetEntity.originFile is almost instant, however some videos are unusable or have unexpected behaviour (HDR, slow motion, etc.) Caching is not really a valid option for us, since we need to provide access to different videos quickly and can't afford the storage that the cache would take up in that case.

To Reproduce

Expected behavior Since other apps, like every messenger for example, can access gallery videos almost instantly, even on first load where caching couldn't have taken place, we expected that the same should hold true for this package as well.

Flutter version [✓] Flutter (Channel stable, 2.10.0, on macOS 12.2 21D49 darwin-arm, locale en-DE) • Flutter version 2.10.0 at /Users/tim/flutter • Upstream repository https://github.com/flutter/flutter.git • Framework revision 5f105a6ca7 (13 days ago), 2022-02-01 14:15:42 -0800 • Engine revision 776efd2034 • Dart version 2.16.0 • DevTools version 2.9.2

Smartphone:

AlexV525 commented 2 years ago

Sorry about the poor experience, but can you provide more conditions when you're running into this issue? Such as, is the file getter always taking so much time? Why would the original file of an HDR photo fail to use?

I'd like to understand the detailed behavior in order to produce more thoughts on how to fix/improve this.

timcreatedit commented 2 years ago

Hi, thank you for your help. Sorry that I didn't provide enough details.

I'm specifically talking about video files. The .file getter is taking that much time for all video files on the first load if isLocal == true. As soon as the video has been loaded once and is cached, it's basically instant, however we can't cache all videos forever due to space constraints.

Original files for HDR video are not really usable, because the colors are completely off, at least when displaying them using video_player, or processing them with FFMPEG. We also can't use the original files for slow motion video, because it just returns the video with high framerate and original speed and ignores the slow motion parts that users selected in the Photos app.

I hope I made it a little clearer, I'm available for questions or any tests that I could run

AlexV525 commented 2 years ago

When loading videos, can you use getMediaUrl instead when isLocallyAvailable?

I just took a glance at the file getter, it'll request an export session that produces a playable media file on the device, which can be I/O related.

https://github.com/fluttercandies/flutter_photo_manager/blob/e688fa2a27aedd0724b3ede971547a18c0088c26/ios/Classes/core/PMManager.m#L555-L558

timcreatedit commented 2 years ago

I'll give that a try and report back with the results. Thank you!

timcreatedit commented 2 years ago

I can confirm that using getMediaUrl() produces the same results as .originFile. The videos load instantly but are unusable in some cases like HDR or slow motion. I have no idea about how the native iOS part works so forgive me if I sound stupid, but I feel like there should be a solution since picking a video in other apps works quickly, and preserves slow motion for example. Thanks again for your help and time

AlexV525 commented 2 years ago

I don't have iOS devices nearby until tomorrow, did you try with the image_picker yet? Does it work well with the video_player? Which picker is available?

timcreatedit commented 2 years ago

I've investigated further using other packages, here are my results:

image_picker 0.8.4+8

https://pub.dev/packages/image_picker

images_picker 1.2.10

https://pub.dev/packages/images_picker

I was wondering if the exact match between the video duration and the time it takes to load was a coincidence, so I measured some other times: 00:24 HDR (95 MB) - 14s to load 00:11 HDR (52 MB) - 6s to load

So it seems to be a coincidence.

AlexV525 commented 2 years ago

So, from your investigations, can you try with a lower quality here to see if it helps?

https://github.com/fluttercandies/flutter_photo_manager/blob/e688fa2a27aedd0724b3ede971547a18c0088c26/ios/Classes/core/PMManager.m#L555

timcreatedit commented 2 years ago

Switching to AVAssetExportPresetLowQuality reduces the export time for the 1:40 video to 15 seconds, which is still as long as the compression step from image_picker, however the resulting video clip is much, much worse to a point where it's basically completely unusable.

AlexV525 commented 2 years ago

Thanks for the proof, sound like we can learn from the compressing process of the image_picker. Are you able to indicate where did the plugin (image_picker) do the compress?

timcreatedit commented 2 years ago

I can try looking into that. However I don't really understand why a compression step is even necessary. If native iOS apps can display a video in full quality and with all edits instantly, why can't we use the same method for this package (might be a stupid question)?

AlexV525 commented 2 years ago

I've filed a PR for this, please verify if it brings improvement in this case.

TL;DR, no compression was brought here, just copying the asset from private to the sandbox without export sessions when the asset already matched the condition. Also, exporting QuickTime movies instead of MPEG4 might cost less time since they're typically QuickTime movies originally.

AlexV525 commented 2 years ago

Tested with below contents on iPhone 12, and all of them are successful to play with video_player.

You can see the pre-check improved instantly reply with the copied file in the sandbox, without any compressions.

Duration Type Resolution Format Pre-check improved Type improved Not improved Size
00:48 HDR 1920x1080 HEVC 0:00:00.046232 0:00:02.161513 0:00:09.608286 68.6 MB
01:36 Network 1080x1920 H.264 0:00:00.034561 0:00:01.113288 0:00:01.078738 8.9 MB
00:03 Slo-mo 1080x1920 HEVC N/A (AVComposition) 0:00:01.894947 0:00:04.578076 23.7 MB
AlexV525 commented 2 years ago

The conclusion here is when videos are special, here only with Slo-Mo videos, are composited as AVComposition. When a video is composited, it can only be fetched with at least an export request, otherwise, the original video will be obtained.

Other videos (when they're AVURLAsset) can be delivered directly without any compressions, but still require a sandbox copy. It only involves with I/O request, so it behaves like instantly obtained without seconds of waiting.

The PR will determine when to obtain the path from AVURLAsset directly, and when to export a composited video with medium quality (since the medium is most balanced so far).

Sandbox is still required, no luck with that.

AlexV525 commented 2 years ago

FYI @Tom3652, I think the improvement can be also related to some long-retrieving issues you've mentioned.

timcreatedit commented 2 years ago

Thank you for your effort and explanation. For the pull request however, unfortunately the file getter now matches the behavior of originFile for me, meaning that HDR video colors are completely off in video_player, and slow motion videos play in original speed. I've tried the passthrough setting in my investigation yesterday as well with the same results.

AlexV525 commented 2 years ago

unfortunately the file getter now matches the behavior of originFile for me, meaning that HDR video colors are completely off in video_player, and slow motion videos play in original speed.

I can't identify the HDR issue, but the Slo-Mo should be fixed since I didn't push my commit when the above comments were posted. Try 28c68d7.

Tom3652 commented 2 years ago

Hi @AlexV525 thanks for mentioning me here.

It was indeed my requirement as well (load videos as fast as native iOS apps), i have not tested with HDR videos or slow motion actually, and @timcreatedit i am using ´getMediaUrl()´ to fetch the video instantly, then if the user wants to upload it, i load it during my upload process with ´.file´ so there is no bad user experience but the upload is much longer depending on the video.

I will give a try with the HDR / slow motion to see if the video_player is playing them normally based on your comment, and if the URL file also plays the correct file in the browser.

I agree however that native iOS apps seem to be able to fetch videos (without cache for the first time) instantly or almost instantly and use it (edited or not)

AlexV525 commented 2 years ago

If the HDR can be solved by commenting out these, then we might require all assets to be exported.

https://github.com/fluttercandies/flutter_photo_manager/blob/28c68d706b747b972a4aa55728d6878e0dbfa25e/ios/Classes/core/PMManager.m#L564-L589

AlexV525 commented 2 years ago

I agree however that native iOS apps seem to be able to fetch videos (without cache for the first time) instantly or almost instantly and use it (edited or not)

I can't guarantee any solid usage here, since we're a cross-platform plugin. :)

timcreatedit commented 2 years ago

I can't identify the HDR issue, but the Slo-Mo should be fixed since I didn't push my commit when the above comments were posted.

I can confirm that slow-motion plays correctly now, but the quality is abysmal, I think a higher quality preset should be considered, especially if users are unable to choose the export quality themselves.

Regarding HDR, I don't know if it is within the scope of this plugin, since the actual responsibility would be for video_player to be able to play HDR content correctly see this issue. It will also be weird if video_player starts supporting HDR at some point and then all the files obtained from the .file getter are SDR I guess?

timcreatedit commented 2 years ago

It was indeed my requirement as well (load videos as fast as native iOS apps), i have not tested with HDR videos or slow motion actually, and @timcreatedit i am using ´getMediaUrl()´ to fetch the video instantly, then if the user wants to upload it, i load it during my upload process with ´.file´ so there is no bad user experience but the upload is much longer depending on the video.

I've considered this approach as well, however this doesn't work for slow motion videos since the displayed video would vary greatly from the one that would be used in export. Even more so since my app requires selecting a specifically timed clip from a video.

AlexV525 commented 2 years ago

If you're good with the highest preset's duration, I think we should use it for all. Can you try it by modifying the code?

file getter could be hard to extend since it'll introduce a breaking change. From the overall API design, I don't have a better idea about a customizable parameter for the quality.

Tom3652 commented 2 years ago

I've considered this approach as well, however this doesn't work for slow motion videos since the displayed video would vary greatly from the one that would be used in export. Even more so since my app requires selecting a specifically timed clip from a video.

I will give it a try today with my implementation and see this, i also have this kind of feature in my app so thanks for pointing it out

timcreatedit commented 2 years ago

I think it's more an issue of predictability than of what I need personally, since users of this plugin will be confused if the video quality suffers so significantly. So imo if you're forced to use the export session, we should go with the highest quality.

I think overall this is a good compromise since slow motion videos also tend to be shorter (and more rare) and therefore the export times shouldn't matter as much as for a 1min 4k HDR video for example.

AlexV525 commented 2 years ago

Although the control flow is the same for Slo-Mo videos, it's still optimized by a more accurate file type for export. So I guess we still got improvements for them.

timcreatedit commented 2 years ago

Although the control flow is the same for Slo-Mo videos, it's still optimized by a more accurate file type for export. So I guess we still got improvements for them.

It is noticably quicker! Also it will make it easier to create a good UX since I think users will at least understand why slow motion videos take longer to load. Thank you very much, now we only need HDR support in video_player 😬

Tom3652 commented 2 years ago

So with the version i have got without the above PR i confirm that getMediaUrl doesn't retrieve the file in Slo-Mo or at least the video_player doesn't play the Slo-Mo i can't tell... However i have no problem playing HDR videos, with which camera do you record these videos @timcreatedit ?

timcreatedit commented 2 years ago

However i have no problem playing HDR videos,

It's not about playing them, it's about the colors being noticeably off. This is an issue with the video_player package, it could be that you're using better_player, which does play HDR on iOS correctly. There is an open issue for the video_player package about the topic, you can go there for details :) https://github.com/flutter/flutter/issues/91241

timcreatedit commented 2 years ago

So with the version i have got without the above PR i confirm that getMediaUrl doesn't retrieve the file in Slo-Mo or at least the video_player doesn't play the Slo-Mo i can't tell...

I think the PR won't fix that, you'll have to use the .file getter for those. It seems like for providing native-like video performance on iOS, we would need to build a video player that can read the metadata of those files to play them correctly.

AlexV525 commented 2 years ago

The PR contains the Slo-Mo composition part when exporting, which means they can be played like a Slo-Mo with file and getMediaUrl (if I tested it right). But the originalFile doesn't.

For a better UX when integrating with iCloud, durations, etc, I still prefer developers to use the file with a progress handler at first, then use the getMediaUrl for something like Live Photos.

Tom3652 commented 2 years ago

I think the PR won't fix that, you'll have to use the .file getter for those. It seems like for providing native-like video performance on iOS, we would need to build a video player that can read the metadata of those files to play them correctly.

Yes you are right, getMediaUrl has not changed (doesn't play the Slo-Mo) but the .file yes.
-> Edit : Actually it's not true, i have cleared my phone's cache and the getMediaUrl is working for Slo-Mo as well :) and it took 5sec instead of 7 to fetch the 3sec video still under isLocallyAvailable = true.

With my performances test, a 3sec Slo-Mo video is loaded in 7 seconds using .file which is much better than before but still quite long while isLocallyAvailable = true.

This is only true the first time the asset is fetched though, after it's instantly done.

I have also got a new error but i should make a new issue i think ?

flutter: Error flutter ZonedGuarded : PlatformException(Error Domain=AVFoundationErrorDomain Code=-11823 "Cannot Save" UserInfo={NSLocalizedRecoverySuggestion=Try saving again., NSLocalizedDescription=Cannot Save, NSUnderlyingError=0x280c100f0 {Error Domain=NSOSStatusErrorDomain Code=-12101 "(null)"}}, null, null, null) 
StackTrace : #0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:607:7)
#1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:177:18)
<asynchronous suspension>
#2      AssetEntity._getFile (package:photo_manager/src/types/entity.dart:624:26)
<asynchronous suspension>
AlexV525 commented 2 years ago

I have also got a new error but i should make a new issue i think ?

Indeed.

timcreatedit commented 2 years ago

This is only true the first time the asset is fetched though, after it's instantly done.

This is due to caching but be careful with the size of the cache. If you don't clean up after yourself, the cache can easily grow to gigabytes in size if you load lots of videos, at least that's what happened for us.

AlexV525 commented 2 years ago

This is due to caching but be careful with the size of the cache. If you don't clean up after yourself, the cache can easily grow to gigabytes in size if you load lots of videos, at least that's what happened for us.

Thought about this today, I think it'll be a good practice to point out in the README.

It's actually indicated already: https://github.com/fluttercandies/flutter_photo_manager/blob/8afba2745ebaac6af8af75de9cbded9157bc2690/README.md#L435-L436