search

fifonik / FFMetrics

Visualizes Video Quality Metrics (PSNR, SSIM & VMAF) calculated by ffmpeg.exe
570 stars 22 forks source link

can not restart after modifying "Duration" option #90

Closed windowsair closed 2 years ago

windowsair commented 2 years ago

If we have already finished calculating a parameter and then modify the option "Duration" and click "Start" again, the parameter in the new time duration can not be calculated.

Is this the expected behavior? Thank you!

fifonik commented 2 years ago

Thanks for the report. I will check it out.

fifonik commented 2 years ago

The current behaviour is not good as if skip/duration changed, metrics' values are not relevant any longer for the new skip/duration. So calculated metric values must be reset.

However, for some people this might be not the best behaviour as calculation is quite long process.

So I'm thinking about other ways. Possible options I have in mind atm:

  1. Simply reset metrics;
  2. Show confirmation dialog on skip/duration change ('Metrics will be reset, would you like to continue?');
  3. Disable skip/duration dropdowns. They will be re-enabled when 'Reset' pressed.

Please let me know if you have other ideas. Thanks.

windowsair commented 2 years ago

I might prefer two buttons "Start ALL metrics" and "Start selected metric".

In this case, when we change the options, we don't have to explicitly reset them, but just take the new calculated value (one metric or all) instead of the old one.

Well, different users certainly can't agree on their opinions. Just choose the one you like.

fifonik commented 2 years ago

It would be possible to have on the same screen:

I cannot think of a situation when any of the above can be useful in real life (other than to confuse someone on purpose).

windowsair commented 2 years ago

Oh, there is indeed confusion.

By "selected metric" I actually mean a certain type of metric, such as calculating the SSIM or PNSR between two groups.

What I encountered was probably a scenario where I had finished calculating SSIM and PSNR, and I tried to calculate VMAF for 30s to initially check the effect. At this point I would change the duration to calculate the full VMAF, but I would have to use "Reset", which would cause my previously obtained SSIM and PSNR results to be reset together.

fifonik commented 2 years ago

You scenario is reasonable indeed. Another option I shoud consider -- somehow 'mark' (font/background colour? italic?) calculated metrics' values, that are not matched to currently selected options.

ls-milkyway commented 2 years ago

Oh, there is indeed confusion.

By "selected metric" I actually mean a certain type of metric, such as calculating the SSIM or PNSR between two groups.

What I encountered was probably a scenario where I had finished calculating SSIM and PSNR, and I tried to calculate VMAF for 30s to initially check the effect. At this point I would change the duration to calculate the full VMAF, but I would have to use "Reset", which would cause my previously obtained SSIM and PSNR results to be reset together.

I guess ..you will have to restart FFmetrics ....to calculate VMAF at different frame interval (say 30 seconds)....I understand you need to compare 2 encodes based on single or 2 quality metrics (PSNR OR SSIM OR VMAF).....instead of comparing all 3 metrics. Just calculate PSNR & SSIM ...save the graphs ...launch FFmetrics again ...deselect PSNR & SSIM ...& select VMAF with the needed interval....as most of the users want frame by frame comparisons of all the metrics ....say at a particular frame where SSIM is low but VMAF is good or average ....I mean how well all the 3 algorithms states on all the frames used for evaluation and comparing the 3 to other encodes ...otherwise need for 3 different algo becomes sort of pointless...

windowsair commented 2 years ago

Oh, there is indeed confusion. By "selected metric" I actually mean a certain type of metric, such as calculating the SSIM or PNSR between two groups. What I encountered was probably a scenario where I had finished calculating SSIM and PSNR, and I tried to calculate VMAF for 30s to initially check the effect. At this point I would change the duration to calculate the full VMAF, but I would have to use "Reset", which would cause my previously obtained SSIM and PSNR results to be reset together.

I guess ..you will have to restart FFmetrics ....to calculate VMAF at different frame interval (say 30 seconds)....I understand you need to compare 2 encodes based on single or 2 quality metrics (PSNR OR SSIM OR VMAF).....instead of comparing all 3 metrics. Just calculate PSNR & SSIM ...save the graphs ...launch FFmetrics again ...deselect PSNR & SSIM ...& select VMAF with the needed interval....as most of the users want frame by frame comparisons of all the metrics ....say at a particular frame where SSIM is low but VMAF is good or average ....I mean how well all the 3 algorithms states on all the frames used for evaluation and comparing the 3 to other encodes ...otherwise need for 3 different algo becomes sort of pointless...

Yes. That's what I've been doing since I discovered this behavior

ls-milkyway commented 2 years ago

Yes. That's what I've been doing since I discovered this behavior.

It's is indeed normal sort of behavior...as once a frame/time interval is selected ...then for that particular time/frame interval ffmpeg or ffplay or ffprobe (frankly I don't know which one is called) ,,,is locked (for one or 2 or all the metrics) ...I guess what you want can be done...but it sort of can complicates the logic required:

Release FFmpeg (say if it used) if previous calculations are complete and relaunch it in the background if different frame interval is selected. If frame interval is selected in between of a ongoing calculation then prompt "Calculations on going ...please wait till it's over & saved"

That's is to be decided by the developer based on complexity & difficulty of it's implementation .

windowsair commented 2 years ago

Yes. That's what I've been doing since I discovered this behavior.

It's is indeed normal sort of behavior...as once a frame/time interval is selected ...then for that particular time/frame interval ffmpeg or ffplay or ffprobe (frankly I don't know which one is called) ,,,is locked (for one or 2 or all the metrics) ...I guess what you want can be done...but it sort of can complicates the logic required:

Release FFmpeg (say if it used) if previous calculations are complete and relaunch it in the background if different frame interval is selected. If frame interval is selected in between of a ongoing calculation then prompt "Calculations on going ...please wait till it's over & saved"

That's is to be decided by the developer based on complexity & difficulty of it's implementation .

The world of GUI is such that you never know in what way the user will use your software.

So I'm just reporting this one potentially confusing behavior, it's up to the developer to decide what to do 😀

fifonik commented 2 years ago

At this stage I feel that I have feeling that I'm missing something.

I can:

Now I can (without restarting program):

As I mentioned earlier, in this case the metrics values are not consistent (PSNR calculated for whole file, while VMAF is for the specified duration only).

At this stage it is not possible to reset only some metric values. For example, one may decide to calculate everything as above (PSNR for full, VMAF for 5 seconds only), then reset VMAF metric value only, change Duration to full and calculate VMAF so VMAF and PSNR would be consistent.

I think resetting one metric values actually could be useful in some situations and it is very easy to implement: it could be menu items "Action | Reset PSNR results", "Action | Rest SSIM results", "Action | Reset VMAF results", "Action | Reset ALL results".

P.S. One can press Reset button and start over if needed. No program restart required so I'm not sure why program restart was mentioned at all.

ls-milkyway commented 2 years ago

one may decide to calculate everything as above (PSNR for full, VMAF for 5 seconds only),

This usually does not happen when somebody is doing quality analysis....I mean for a single encode ...frames are sometimes checked for all the 3 metric values ...to arrive at conclusion ....say: What's the reason PSNR & SSIM is good whereas VMAF is average for set of frames within the same video ......etc. etc. and make changes in the encoding parameters .....allot more bitrate etc. to that set of frames...otherwise what was the need for all the 3 metrics?

Whereas to simply compare 2 encodes ...which one is better.... the story is different. But still you simply don't select the better one .....the better one can also have tremendous scope of improvement.... For eg. comparing watermelon (reference) to an orange & an apple (2 distorted)....you simply don't pick orange ...because it' size is bigger than the apple...but the big question is: how close is it to the watermelon?

I can understand VMAF takes time & it's better to use short interval for quick 2 or more encodings comparisons. But still it is partial and rough estimate (specially if interval is huge & video is big).

I think resetting one metric values actually could be useful in some situations and it is very easy to implement: it could be menu items "Action | Reset PSNR results", "Action | Rest SSIM results", "Action | Reset VMAF results", "Action | Reset ALL results".

That's a good solution provided graphs from previous calculations are saved.

fifonik commented 2 years ago

it is very easy to implement: it could be menu items "Action | Reset PSNR results", "Action | Rest SSIM results", "Action | Reset VMAF results", "Action | Reset ALL results".

Implemented in my dev version.

fifonik commented 2 years ago

Per metric results reset available in v1.3.0 beta 1

fifonik commented 2 years ago

I've just published v1.3.1 beta 1. In the version, if metric value was calculated with options that are not matched the current options, are crossed out. In addition, options for all metrics are written into result csv file.

For me, this issue can be closed. Please let me know if you still have some concerns about it. Thanks.

windowsair commented 2 years ago

Everything is working fine! Let's close it.