search

gyroflow / gyroflow-plugins

Gyroflow OpenFX, Adobe and frei0r plugins
GNU General Public License v3.0
24 stars 1 forks source link

Bad performance when used with DJI Avata 2 videos #13

Open Shadetail opened 6 months ago

Shadetail commented 6 months ago

I've been using gyrfolow ofx plugin with GoPro 11 mini for like a year now. Recently I started using it with my new Avata 2 drone, but whenever I do, I get severe lag, something that I never experienced with GoPro footage, despite GoPro footage being significantly higher resolution at 5.3k vs 4k for Avata 2.

How this manifests, is that when clicking on clip, davinci freezes for a few seconds while it loads the clip, then it works normally unless I try to change a value of parameters in realtime with the mouse, for example if I try to move the horizon roll slider i get 1-2 second lags as the gyroflow is trying to rotate the video accordingly, while with GoPro footage all this has always been smooth and realtime. With GoPro I could have hundreds of gyroflow ofx enabled clips in the timeline, and it would all work smoothly, but with Avata 2 clips, if I click on a clip that I haven't been seeking through in the past few minutes, I get a freeze of up to 7 seconds as I wait for it to load.

I first expected this might be a davinci issue, but it really only happens with gyroflow enabled, and my guess is that it is somehow connected to the lens profile, if not gyro data itself, rather than with the video. As I understand it, this profile has been provided by DJI. Maybe they messed it up somehow?

This issue bad enough that it's making me consider just sticking a GoPro on Avata 2 despite Avata 2 having a larger senzor size, just because it makes editing and previewing the videos so much more difficult.

Here is a link to unedited video file of one of my flights that includes gyro data that can be used for repro: https://drive.google.com/file/d/1Cs7lhTqp2cEgoSkHwKH9l2WLCj8QyySE/view?usp=sharing

AdrianEddy commented 6 months ago

Have you tried the dev build of the plugin? https://gyroflow.xyz/devbuild/

AdrianEddy commented 6 months ago

It was indeed a bug, just updated the nightly build with the fix

Shadetail commented 6 months ago

Hi! Wow that was fast! I downloaded nightly just now of both ofx plugin and standalone gyroflow to test the fix, but found the issue is still happening.

Is it possible that the nightly still didn't finish building the version that contains the fix and I need to wait 24h? I looked around, but couldn't find any way to confirm which version I'm on.

Just in case, here's the video of the bug in action, as I noticed CPU usage graphs behaving very differently with Avata 2 footage vs GoPro footage so I recorded it: https://drive.google.com/file/d/1T2yj8MAZUbVlCAp5xSTPt_VajSjvS9ey/view?usp=sharing

AdrianEddy commented 6 months ago

Hmm, yes that was exactly the bug I fixed. Can you send me gyroflow.log file which is created in C:\Program Files\Common Files\OFX after you close davinci?

Shadetail commented 6 months ago

Here you go: Gyroflow.ofx.log

AdrianEddy commented 6 months ago

Weird, everything looks good in the log. What hardware do you have, is it pc or laptop? Do you have latest GPU drivers?

Shadetail commented 6 months ago

I just updated the nvidia driver from 552.44 to 555.85, but that didn't help. I'm running on a PC in 4k:

AdrianEddy commented 2 months ago

Try this one and let me know if it's any better https://nightly.link/gyroflow/gyroflow-plugins/workflows/release/main/Gyroflow-OpenFX-windows.zip

Shadetail commented 2 months ago

Thank you for attempting to fix it! ♥ I just tried it and the issue is still present though. image Here's a screenshot of how different the cpu usage graphs look when trying to rotate Avata 2 footage vs GoPro11 footage, same as demonstrated in the video before.

AdrianEddy commented 2 months ago

do you have the video files on HDD or SSD?

AdrianEddy commented 2 months ago

Try this updated version now https://nightly.link/gyroflow/gyroflow-plugins/workflows/release/main/Gyroflow-OpenFX-windows.zip

It should now parse each file only once and cache the result in memory

Shadetail commented 2 months ago

I keep all videos on an SSD (Sabrent Rocket Q NVME, plenty fast). I tried the latest version now, and I noticed something that I didn't notice before: video length drastically affects this bug. I'm not sure if one of your updates fixed this bug for short videos only, or if this is just something I haven't noticed before. I recorded a video for you demonstrating this: https://www.youtube.com/watch?v=0UetRT6jpLc

AdrianEddy commented 2 months ago

so looks like this has nothing to do with Avata or GoPro, but is only an issue with clip length. In your video, the GoPro footage is still shorter than the long Avata one

Well, this is expected, when the effect is first applied, Gyroflow has to parse the entire video file and gyro data from it and do all the initial calculations

When changing any parameter, all the smoothing, zooming and distortion calculations have to be done for the entire video, and this is inevitable

it could be possible to limit some calculations to the selected clip instead of entire video, but it's not trvial. I'll look into it

Shadetail commented 2 months ago

GoPro video used as example is 11 min, and that was perfectly usable performance, and the performance was exactly the same as the much shorter 1 min 23 sec GoPro clip.

Here's media info for the longer GoPro clip: File size: 13.9 GiB Duration: 11 min 3 s Overall bit rate: 179 Mb/s Frame rate: 59.940 FPS Width: 5 312 pixels Height: 2 988 pixels Format: HEVC Format profile : Main 10@L6@Main Codec ID: hvc1

So it appears that for GoPro, performance does not scale with video duration, at least not in a way that is noticeable here.

But for Avata2 videos, it went from usable to very laggy when we reached the video length of 5 min 43 sec, which is less than half of the length of the GoPro video that didn't have any issues.

Here is mediainfo for that Avata 2 clip: File size: 5.21 GiB Duration: 5 min 43 s Overall bit rate: 130 Mb/s Frame rate: 59.940 FPS Width: 3 840 pixels Height: 2 880 pixels Format: HEVC Format profile: Main 10@L5@High Codec ID: hvc1

So Avata 2 clip here is significantly shorter and overall lighter, but performs much worse than the much longer heavier GoPro clip.

I noticed now that I had a .gyroflow file next to the gopro file, and removing it made "Load current file" use defaults which did negatively affect performance compared to high performance in the youtube video I shared, but the lag still never exceeded one second, and it's consistent.

While for the 5 min 43 s Avata clip with default settings lag is inconsistent, sometimes the response in instant, but also easily lags up to 6 seconds.

For these two videos, all the settings that are exposed in the plugin are the same, as both are at defaults due to no .gyroflow files present. It makes me suspect that DJI's lens profile might be to blame, as that's definitely different, but I'm not sure if that's something that could even in theory affect performance. And I'm also not sure what else could be different in between them with regards to gyro data or gyroflow settings which might not be exposed in the plugin.

I opened both in gyroflow and compared default settings. I found that "Rolling Shutter Correction" was enabled in for Avata 2 video but disabled for GoPro video: 2024_09_03-16_22_03-Gyroflow_BHygfMitIc 2024_09_03-16_22_44-Gyroflow_lEkO8HGmhZ

I tried disabling it for Avata 2 video and saving that as .gyroflow file and loading that in the plugin, but it didn't make the performance any better. I also tried to load GoPro lens profile into Avata 2 video, but that didn't help the performance either.

AdrianEddy commented 2 months ago

ah yes, Rolling shutter correction does add to the calculations quite a bit. Also, the "zoom limit" feature is quite heavy and depending on the type of video, it may be kicking in in the Avata case, but not the GoPro case. You can set the zoom limit to 300 to disable it. You can also try to disable rolling shutter correction and save that in the project file and compare the performance then

Shadetail commented 2 months ago

Rolling shutter correction didn't really make a difference for me, but zoom limit did! Setting it to 300 eliminated the lag entirely, it made even the longest Avata 2 video perform consistently with around 0.25 seconds required for the rotation change.

Too bad that's like the best new feature Gyroflow has.

It's possible that "caching the result in memory" feature you added fixed the large practical aspect of this issue where using many clips in a larger project without any changing of gyroflow settings made jumping around the timeline painfully slow just because all clips have active Gyroflow effect on them. I'll need to set up a project with many clips to test this out since the updated plugin broke backwards compatibility with my previous projects that used gyroflow plugin. (With small number of clips Resolve keeps them all in memory and issue is not apparent). I'll do that soon and let you know what the situation is like.

Slightly offtopic, but speaking of the zoom limit feature, my prefered way of using Gyroflow is to disable dynamic zooming entirely and instead use static zoom that slightly crops into the frame to give Gyroflow something to work with to make the video smoother. It's my understanding that zoom limit feature can still be used here to achieve dynamically changing smoothness value, and some parts of the video still benefit greatly from very large smoothness, while some require very low. "Max smoothness at high velocity" is a binary way of toggling between two smoothness values to somewhat address this, but this new zoom limit feature should allow a full spectrum of smoothness values, in fact it should allow for Gyroflow to select the largest possible smoothness value for every point in time that wouldn't result in going outside of the frame (given that zoom/fov is locked). Am I correct in thinking that this is how this should work in theory? I was playing around with it and was getting somewhat confusing results so far, so I thought I'd check with you if I'm understanding this correctly and just need to play with settings more or if I'm trying to do the impossible for some reason.

Shadetail commented 2 months ago

I'll do that soon and let you know what the situation is like.

I got around to testing this, and unfortunately the change did not significantly improve the situation of moving around the timeline when the timeline consists of short clips (3 second each) from 28 different Avata 2 videos, all of them with gyroflow applied. I thought the only way of currently working like this would be to not use the zoom limit by setting it to 300 on all clips, but even doing that didn't really change much, I still got up to 6 seconds lag spikes when trying to play the timeline and 90% of playback time would be spent frozen. This seems like a hard thing to debug, I'll let you know if I make any further progress on it.

Oh, and btw I just published the video in question, in case you're curious what the use case for this was.

rattkin commented 1 week ago

I am also experienced heavy slowdowns when working with long files, I have interview longer than 1 hour and it is very laggy.

Shadetail commented 1 day ago

@AdrianEddy I figured it out!

This is due to how the IMU data is handled differently between the two cameras.

Since compiling Gyroflow from source comes with having console output, looking at what the console shows when each video file is loaded, I was able to notice the difference in sampling rates:

Since in timeline we observe 1 sample/frame, this means GoPro gyro data has been downsampled to optimize performance, and since having more than 1 wouldn't be useful for stabilization anyways assuming synchronised timings.

Digging further, I found that GoPro data gets processed through get_avg_sample_duration in telemetry-parser which effectively downsamples the data: https://github.com/AdrianEddy/telemetry-parser/blob/master/src/gopro/mod.rs

pub fn get_avg_sample_duration(samples: &Vec<SampleInfo>, group_id: &GroupId) -> Option<f64> {
    let mut total_duration_ms = 0.0;
    let mut first_tsus = None;
    let mut last_tsus = None;
    let mut count = 0;
    let mut last_len = 0;
    // ... calculates average duration between samples
}

But DJI data uses raw sampling rate from metadata (imu_sampling_rate: 2000): https://github.com/AdrianEddy/telemetry-parser/blob/master/src/dji/mod.rs

if let Some(v) = clip.imu_sampling_rate.as_ref().map(|h| h.imu_sampling_rate as f64) {
    sample_rate = v;
}

This explains why GoPro footage processes so much faster - it's working with 33.3x less data points for the same duration of video, because telemetry parser further optimizes the already sparse GoPro data, while not optimizing the 10x denser DJI Avata 2 gyro data.

It makes sense that this especially impacts longer videos and iterative features like Zoom Limit and Auto Keyframing, where the entire set of IMU data gets reprocessed on each iteration.

The fix would likely involve implementing similar downsampling for DJI footage as is already done for GoPro footage in the telemetry parser.

AdrianEddy commented 1 day ago

get_avg_sample_duration(samples doesn't downsample the data, it just calculates the sample duration to normalize it, but the amount of data stays the same. We can't downsample the DJI data because we use rolling shutter correction. GoPro fixes rolling shutter in camera so we can have less data in post production

Shadetail commented 1 day ago

Ouch! So I'm guessing it's fundamentally impossible to somehow process the Avata 2 gyro data in an offline manner once in order to bake in the rolling shutter correction which would allow it to be downsampled?

But maybe it can optionally be downsampled anyways by trading having the rolling shutter correction for performance?

I'm looking at my fast flying footage and toggling the checkbox, and while I see the difference of the slight skew being applied or not as I tick or untick the checkbox, I'm not noticing the difference in the quality of the smoothing or the way video feels to me as a result. I definitely couldn't pass the blind test of guessing correctly whether the rolling shutter correction was enabled or disabled.

Since get_avg_sample_duration doesn't downsample the GoPro data. Would you be able to tell me where the GoPro data is being downsampled? I'd like to at least attempt to do this for DJI data to see how bad the result is so I might as well do it properly, as the alternative is not getting to use Zoom Limit and Auto Keyframing features at all, which both make a big difference to me, way bigger than the difference of how large the effect of rolling shutter correction seems to be, at least to my eyes when applied to my FPV footage.

AdrianEddy commented 1 day ago

GoPro data is simply downsampled on the camera, as far as I remember it's either 1000Hz or 500Hz internally before it's stored as 200Hz in the metadata. We don't do it in Gyroflow. I think it's possible to introduce a "Low quality" checkbox

Shadetail commented 1 day ago

Thanks for the input! But how does Gyroflow then go from reading 200Hz metadata to having 1 sampling point per frame, rather than having 3.33 as would be expected for 60 fps? I just did a 30 and 24 fps tests, and in both cases there was 1 sampling point per frame, and not 6.66 or 8.33. This suggests that number of sampling points is somehow adjusted from 200 sampling points per second to 60, 30 or 24 sampling points per second, depending on video's framerate, and also at the same time the downsampling points seem to be chosen such that they are synchronized to the frame timings, since 200 isn't divisible by 60, 30 or 24.

For DJI Avata I do see the exact number of sampling points per frame that I would expect by doing the math: 2000Hz / 60fps = 33.3 samples/frame.

I'm guessing that the "low quality" checkbox that would fix Avata performance at the cost of not having rolling shutter correction, should do the same thing that is being done for GoPro.

Or if there is no such thing, if nothing is being done to downsample or synchronize GoPro gyro data, is GoPro metadata then falsely reporting 200Hz and it is in fact 60 30 or 24Hz? Given that the console outputs: (1) qml: [in onTelemetry_loaded] Telemetry additional data: {[...]"sample_rate":197.61622687786996} and that zooming in on the graph only shows a single perfectly timed point per frame regardless of framerate.