bkauler commented 2 years ago

First, just a note about a message when start lives from a terminal, about an invalid effect:

# lives

LiVES 3.2.1-pre
Copyright 2002 - 2021 Gabriel Finch (salsaman+lives@gmail.com) and others.
LiVES comes with ABSOLUTELY NO WARRANTY
This is free software, and you are welcome to redistribute it
under certain conditions; see the file COPYING for details.

LiVES info: Invalid effect farneback_analyserfarneback_analysersalsaman1 found in compound effect image_stabilizer, line 4

LiVES info: Invalid effect farneback_analyserfarneback_analysersalsaman1 found in compound effect motion_analyser, line 4

flv_decoder: info - frame size=198 x 154, pixel size=192 x 154

That last line is after opening the small video. In the File Open dialog window, the video can be previewed, and plays both audio and video correctly. After opening, all the video frames look scrambled like this:

lives-broken-video

The video file is only 127KB, so I should be able to attach it. I don't know whether can attach a .flv file, so have appended a false ".gz" onto the filename:

20051210-w50s_56K.flv.gz

bkauler commented 2 years ago

Note, I am still using the code from Jan 2, haven't tested your latest Jan 4 commits.

salsaman commented 2 years ago

The effect message is just for info. farneback is only built if opencv-dev is found at build time. It's quite an interesting effect, - it divides the frame into a grid and produces a motion vector at the centre of each grid area, using an optical flow method. The output (x,y) vectors are delivered as two float alpha channels The first out channel holds the x values, the second, the y values, with dimensions of both equal to the grid size. These alpha channels can then be chained to other data effects, or used in a compound effect such as image_stabilizer or motion_analyser.

The info message is just informing that this component of the compound effects was not available, thus the compound will not be loaded.

salsaman commented 2 years ago

For the flv issue, can you please confirm something - the values indicated suggest there is a 3 pixel vertical border on the left and a 3 pixel border on the right. Can you confirm whether or no that is the case, e.g. by playing the clip in another player.

salsaman commented 2 years ago

It's possible the calculation is wrong and the 3 pixel border should be inside the 192 pixel frame width, rather than around it.

salsaman commented 2 years ago

You may also try going into Preferences / Decoding and enabling "Ignore Blank Borders when Possible" to see if it makes any difference.

bkauler commented 2 years ago

Yes, enabling "Ignore blank borders when possible" fixes the video.

From the video frames viewed in lives, it looks like vertical borders on both sides, inside the frame.

I also opened the video in Avidemux and it plays perfectly, doesn't show anything about there being vertical borders. This is the info Avidemux shows about the file:

===================================================== Video

Codec 4CC: VP6F Image Size: 192 x 154 Aspect Ratio: 1:1 (1:1) Frame Rate: 10.000 fps Average Bitrate: 41 kbps Total Duration: 00:00:17.000

===================================================== Video Codec Extradata

Size: 0

===================================================== Audio (1 active track)

Codec: MP3 Channels: Mono Bitrate: 8000 Bps / 64 kbps Frequency: 11025 Hz Total Duration: 00:00:16.822

Whereas, lives is showing the frames as 198x154.

salsaman commented 2 years ago

Yes, it seems to be finding a wrong caclulation somewhere, adding offset to pixel size to get frame_size, instead of subtracting offset from frame_size to get the pixel size.

salsaman commented 2 years ago

It would be useful if you can also upload this clip.

bkauler commented 2 years ago

It would be useful if you can also upload this clip.

I did, see above, file 20051210-w50s_56K.flv.

salsaman commented 2 years ago

OK, missed that too. Can confirm what you reported, in this case as with the bigbunny clip, the audio is speeding ahead of the video. In this case it is mono audio up mixed to stereo. Need to figure this one out too.

salsaman commented 2 years ago

Fixed: a0909c5ecc92902d3b7e3682cb64ad0e138fe19e

Related audio issues fixed too: 0fa428e162f388f2b82ca03f2ace73f56191dcc0

salsaman commented 2 years ago

Still needs a minor update, however. If "ignore blank borders" is switched on, then decoded frames will be saved without the borders, however if it is later enabled, they will be saved with the borders, leading to differences in the clip.

I need to figure out how to handle this.

salsaman commented 2 years ago

Ah yes, I see what it is now. In this case for example, the original frame is 186 X 154, however the border is added during encoding to make the width a multiple of 16, (22 * 16 = 192), which seems to be a requirement for the vp6(a) format used by .flv.

Since that is not a general requirement for LiVES, what should actually happen is:

frame size should be set to 186 X 154 for the clip (i.e we should always use pixel_size, rather than frame_size)
when decoding a frame, the blank borders should be stripped (handled in the decoder plugin)
The option to NOT ignore blank borders should be disabled in Prefs.
the ffmpeg_encoder has an option to encode to flv format. The restriction hstep=16 for this format, which will trigger LiVES to prompt the user with the choice of resizing the frames or adding a blank border. I'll have to check ffmpeg encoding options to see if it's possible to pass a frame_width, which would remove the need to physically add a blank border in the letterbox option.

salsaman commented 2 years ago

Done, except for the final point: faa9b2197c7fc9f8facfef506956a0d862a0c813

I think it maybe possible to set the frame size by passing -s option to ffmpeg, will need to test.

bkauler commented 2 years ago

I have included the latest LiVES in EasyOS version 3.2:

https://bkhome.org/news/202201/easyos-version-32-released.html

When I untick "Automatic trimming/padding of audio when possible" and open that flv file, video shows as 17.00 seconds but audio is still shorter, 5.95 seconds. So when play, it only plays until the audio ends, doesn't play the entire video.

When that checkbox is ticked, only get silence. Using your latest source, Jan. 8.

I just tested again in Avidemux, it plays the entire video.

salsaman commented 2 years ago

Are you still using mpv ? Is the audio playing at the correct rate, or does it seem speeded up ? Please try again with setting $DEBUG_OPEN in smogrify, and check the format of the commandline being passed.

salsaman commented 2 years ago

Great that you added it to EasyOS. Just one small favour to ask...would you mind changing the menu description to "Video Editor / VJ tool" ?

bkauler commented 2 years ago

Great that you added it to EasyOS. Just one small favour to ask...would you mind changing the menu description to "Video Editor / VJ tool" ?

OK, done.

bkauler commented 2 years ago

Are you still using mpv ? Is the audio playing at the correct rate, or does it seem speeded up ? Please try again with setting $DEBUG_OPEN in smogrify, and check the format of the commandline being passed.

No, it is not speeded up, the audio is truncated to 5.96 seconds, and that's where it stops playing.

EasyOS tries to be the smallest possible download, currently 581MB, so cannot have both mpv and mplayer builtin, too much bloat. If I have to use lives with mplayer, then will not be able to include it builtin, it will have to be installable via the package manager.

I'm not sure if the source I downloaded has your very last commit yesterday. Perhaps that is important.

bkauler commented 2 years ago

Here is the result opening that flv with $DEBUG_OPEN=1:

onMetaData:mixedarray
duration:16.900000
width:192.000000
height:154.000000
videodatarate:40.000000
framerate:10.000000
videocodecid:4.000000
audiodatarate:16.000000
audiodelay:0.000000
audiocodecid:2.000000
canSeekToEnd:1.000000
creationdate:Fri Feb 03 12:25:53 2006

:video type is vp6f 192 x 154 (0 x 0 +3 +0)
using palette 512, size 186 x 154
fps is 10.0000 170
flv_decoder: info - frame size=192 x 154, pixel size=186 x 154
vid open command for 685640289 is: 
aud open command for 685640289 is: LANGUAGE=en LANG=en /usr/bin/mpv --no-config --quiet --no-video --vd-lavc-threads=4   --frames=171   --ao=pcm --ao-pcm-waveheader=no --audio-format=s16 --audio-channels=1 "/mnt/sda1/temp8/20051210-w50s_56K.flv"

Have still got "Automatic trimming/padding of audio when possible" unchecked. Also "ignore blank borders when possible" is unchecked, but video looks ok now.

bkauler commented 2 years ago

And attached is the audiodump.pcm, with false ".gz" appended: audiodump.pcm.gz

salsaman commented 2 years ago

I would be quite interested to see what dependencies you have for the package. For reference:

1) Mandatory dependencies include at least: Build

GTK+ (>=3.0.0) + development libs [>= 3.24.0 preferred] gdk-pixbuf-loaders + development libs gettext + development libs

Runtime

mplayer (preferred), mplayer2, or mpv with jpeg/png output support ffmpeg with jpeg/png input support (preferably with x264 support) imageMagick perl sox (with libsox-fmt-all if applicable !)*

2) Highly Recommended build-dependencies include (devel versions of): libpulse, libjack, ffmpeg, mjpegtools, libogg, libtheora, libdv, libraw1394, libavc1394, libunicap, (libavcodec, libavformat, libavutils, libswscale) or (ffmpeg-libs), libpng, libasound2, libX11, libXrender, libGL, libfftwf3.

(For compiling you should have development versions of these libraries installed. Check the output of ./configure carefully for missing optional dependencies.)

Optional: openCV, LADSPA headers, frei0r-devel, projectM.

3) Highly Recommended runtime dependencies include: python, libtheora-bin, ffprobe, xwininfo, wmctrl, mkvtoolnix, vorbis-tools, libmatroska, ogmtools, dvgrab, frei0r-plugins, libdv2-apps/libdv-bin, cdda2wav/icedax, lame, ffmpeg, x264.

If you like I can add you to the packagers list, so you will receive an email whenever a new version is released. Just let me know what email address you prefer.It's also a good means to communicate critical packages following each release.

salsaman commented 2 years ago

If you still have an option "Ignore blank borders..." in prefs, then you are not using the latest code. Please update and test again.

bkauler commented 2 years ago

If you still have an option "Ignore blank borders..." in prefs, then you are not using the latest code. Please update and test again.

Oh, OK, will update tonight.

These are dependencies of ffmpeg, lives and mpv, from the package database. They are all there, with development libs at compile-time, and run-time component at run-time:

ffmpeg-4.3.1-r6
+alsa-lib,+bzip2,+faac,+faad2,+fontconfig,+freetype,+gcc-runtime,+glibc,+jack,
+lame,+liba52,+libbluray,+libcdio,+libcdio-paranoia,+libdc1394,+libdrm,
+libmad,+libmng,+libmodplug,+libogg,+libsdl,+libsdl-image,+libsdl-mixer,
+libsndfile1,+libtheora,+libva,+libvdpau,+libvorbis,+libvpx,+libwebp,
+libxcb,+libxv,+mesa,+mpeg2dec,+opencore-amr,+openjpeg,+openssl,+orc
,+schroedinger,+speex,+taglib,+unzip,+v4l-utils,+wavpack,+x264,+x265,
+xvidcore,+xz,+zip,+zlib

lives-20220108-r6
+alsa-lib,+bzip2,+cups,+cups-filters,+ffmpeg,+fftw,+gcc-runtime,+ghostscript,+glib-2,
+glibc,+gtk+3,+imagemagick,+libdv,+libebml,+libjpeg-turbo,+libmatroska,+libogg,
+libpng,+libsdl,+libsdl-gfx,+libsdl-mixer,+libsdl-ttf,+libtheora,+libtirpc,+libx11,
+libxrender,+mesa,+mjpegtools,+mpv,+orc,+pulseaudio,+schroedinger,+sox,
+unicap,+v4l-utils,+zlib

mpv-0.34.1-r6
+alsa-lib,+desktop-file-utils,+faac,+faad2,+ffmpeg,+flac,+fontconfig,+freetype,+gcc-runtime,
+gdk-pixbuf,+glibc,+jack,+lame,+lcms,+liba52,+libarchive,+libass,+libavc1394,
+libbluray,+libcddb,+libcdio,+libcdio-paranoia,+libdc1394,+libdrm,+libdvdcss,
+libdvdnav,+libdvdread,+libglu,+libjpeg-turbo,+libmad,+libmng,+libmodplug,
+libogg,+libpng,+libraw1394,+librsvg,+libsamplerate0,+libsdl,+libsdl-image,
+libsdl-mixer,+libsdl-ttf,+libtheora,+libva,+libvdpau,+libvorbis,+libvpx,+libwebp
,+libx11,+libxcb,+libxext,+libxinerama,+libxscrnsaver,+libxv,+libxvmc,+lua,
+luajit,+mesa,+mpeg2dec,+mpg123,+openssl,+pulseaudio,+schroedinger,
+speex,+taglib,+v4l-utils,+vcdimager,+wavpack,+x264,+x265,+xvidcore,+zlib

bkauler commented 2 years ago

Here is a list of every run-time package in latest EasyOS:

bkauler commented 2 years ago

And sox:

sox-14.4.2-r6
+ffmpeg,+flac,+gcc-runtime,+glibc,+lame,+libao,+libid3tag,+libmad,+libpng,
+libsndfile1,+libvorbis,+opencore-amr,+wavpack

bkauler commented 2 years ago

Now using the latest lives.

With "Automatic trimming/padding of audio when possible" checked, that 2005*.flv file now shows video 17.00sec and audio 11.04sec, and only plays the first 11.04 sec of the video, so truncated, completely silent.

With that setting unchecked, it behaves as before, 17.00 sec video, 5.96sec audio, and only plays 5.96sec of the video, so truncated, this time with sound. Sound is not compressed, it is truncated.

salsaman commented 2 years ago

Must be an mpv issue. With mplayer it works as expected. I just installed mpv to test and it is hanging, not producing any audio output at all. I'll have to dig a bit further and figure out what is going wrong. Last time I tested with mpv, probably about 1 year ago, it worked with no issues, so something must have changed since then.

salsaman commented 2 years ago

Yep, seems to be a bug in mpv. If I try the following: /usr/bin/mpv --no-video --ao=pcm "/home/salsaman/Downloads/20051210-w50s_56K.flv"

I get the following error: mpv: ../ta/ta.c:335: ta_dbg_check_header: Assertion `h->canary == CANARY' failed. Aborted (core dumped)

Closing the issue as it is an mpv problem, nothing to do with LiVES.

bkauler commented 2 years ago

Earlier on in one of these issues, I reported that same "canary" fail, which was solved by updating mpv to the latest version, 0.34.1. Trying same test as you did above:

# mpv --no-video --ao=pcm 20051210-w50s_56K.flv 
     Video --vid=1 (vp6f 192x154 10.000fps)
 (+) Audio --aid=1 (mp3 1ch 11025Hz)
[ao/pcm] File: audiodump.wav (WAVE)
[ao/pcm] PCM: Samplerate: 11025 Hz Channels: 1 Format: float
AO: [pcm] 11025Hz mono 1ch float

Exiting... (End of file)
#

I played the file 'audiodump.wav' in mhwaveedit, it is 16.8 sec long.

Well, if you really want to chase down what is going on, why not bootup EasyOS?

Download the image file easy-3.2-amd64.img.gz from: https://distro.ibiblio.org/easyos/amd64/releases/dunfell/3.2/easy-3.2-amd64.img.gz
Expand and write it to a usb-stick. Need a usb stick at least 8GB, good quality (for example Sandisk Ultra or Sandisk Extreme), say it is /dev/sdb, do as root:
```
# gunzip easy-3.2-amd64.img.gz
# dd if=easy-3.2-amd64.img of=/dev/sdb bs=1M
# sync
```
Boot from the usb-stick
Install dev pkgs: Click "sfsget' on the desktop, choose file 'devx_3.2_amd64.sfs' Install it to main desktop, not in a container. Reboot.
You are now ready to compile. Absolutely every dev pkg is installed and all tools such as autotools, git, etc. There is even nemiver gui debugger.

salsaman commented 2 years ago

Apologies then. When I have some time I will for sure give Easy a try out. Thanks for confirming that the audio file is actually being generated correctly. That makes it far easier to track down, as it must be a communication error between the front end and the back end.

salsaman commented 2 years ago

Apologies, still very busy with my home move. but I just had another look at this. I am pretty convinced the bug is outside of LiVES. If I switch off "automatic trim / pad", then open the file, the final size of the "audio" file is 744192, which is correct. The audio is mono in the original but forced to 2 channels. Sample size is 16 bit, and rate is 11025, so this is correct: 744192 / (2 2 11025) = 16.8751, which is more or less the correct time.

In the file you uploaded, the size is 131328, which turns out be EXACTLY 744192 / 17 * 3

so it is as if somewhere in mplayer / mpv, the time is being rounded up to 17 seconds (or maybe the faulty calculation is done using video length), and then only the first 6 seconds are written out (the number here is 6 rather than 3, because your file must be mono, hence the time is doubled). If this is then adjusted to scale back from 17.0 to 16.8751, then the result is the 5.96 which you see there.

I can see absolutely no reason in LiVES why we should be getting just the first 6 or so seconds, nor why seconds appears to be calculated from video length, rather than audio length.

salsaman commented 2 years ago

Just to be sure can you try: LANGUAGE=en LANG=en /usr/bin/mplayer -quiet -noconfig all -novideo -lavdopts o=threads=8 -fps 1000000. -frames 170 -alang eng -ao pcm:fast:nowaveheader -format s16le -channels 1 "/20051210-w50s_56K.flv" </dev/null

then check size of audiodump.pcm

bkauler commented 2 years ago

Sorry for slow response, have been busy elsewhere. I tried the above test, mhwaveedit reports that audiodump.pcm is 13.5 seconds. The sound is completely distorted.

Interesting, can open the 2005*flv file in mhwaveedit, and the sound is 16.8 seconds and plays perfectly. Ditto for Celluloid (which is a frontend to mpv).

salsaman commented 2 years ago

Strange indeed. Maybe you can try changing the options passed to mplayer. Try first with just: /usr/bin/mplayer -ao pcm "/20051210-w50s_56K.flv" , which I am pretty sure will work, then try putting the other options back in one by one and see at what point it breaks. Once that is established, it should be possible to come up with a workaround which produces the correct audio file.

salsaman / LiVES

Small video sound works but video scrambled #47

===================================================== Video

===================================================== Video Codec Extradata

===================================================== Audio (1 active track)

1) Mandatory dependencies include at least: Build

Runtime