Sound visualization freezing after one frame

gxgani commented 2 years ago

Steps to reproduce

Enable Audio Capture Card in Effects

Try out Music visualization in Remote Control

Capture Card: https://www.easycoolav.com/products/usb30-video-capture

What is expected?

Music Visualization Overlay on Video Preview

What is actually happening?

Noticed one frame with audio reaction before visualization itself freezes

System

HyperHDR Server: 
- Build:           (HEAD detached at 8d314bc) (Awawa-5b8ed09/8d314bc-1654356765)
- Build time:      Jun  4 2022 15:37:08
- Git Remote:      https://github.com/awawa-dev/HyperHDR
- Version:         18.0.0.0beta3
- UI Lang:         auto (BrowserLang: en-US)
- UI Access:       default
- Avail Capt:      Linux (V4L2)
- Database:        read/write

HyperHDR Server OS: 
- Distribution:   Debian GNU/Linux bookworm/sid
- Architecture:   x86_64
- CPU Model:      AMD Ryzen 5 3400G with Radeon Vega Graphics
- CPU Type:       24
- Kernel:         linux (5.18.0-1-amd64 (WS: 64))
- Qt Version:     5.15.2
- Browser:        Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0 
-

Logs

-- truncated
2022-06-30T13:33:10.572Z [HYPERHDR] Sound device is starting
2022-06-30T13:33:10.572Z [HYPERHDR] Opening device: default:CARD=Capture
2022-06-30T13:33:10.573Z [HYPERHDR] Sound period size = 2048
2022-06-30T13:33:10.574Z [HYPERHDR] Sound buffer size = 4096
2022-06-30T13:33:10.591Z [HYPERHDR] Sound device has started
2022-06-30T13:33:10.591Z [EFFECT0(Music:...)] Begin playing the effect with priority: 1
2022-06-30T13:33:10.691Z [HYPERHDR] Sound stream:  succesfully captured audio data and the sound is detected.
2022-06-30T13:33:10.693Z [MUXER0] Priority 1 is now active
2022-06-30T13:33:10.693Z [MUXER0] Set visible priority to 1
2022-06-30T13:33:10.693Z [IMAGETOLED0] Total index number is: 288 (memory: 288). User sparse processing is: enabled, image size: 80 x 45, area number: 106
2022-06-30T13:33:10.693Z [HYPERHDR0] New priority[1], previous [240]
2022-06-30T13:33:10.694Z [SMOOTHING0] Clearing queued colors before: enabling. Smoothing configuration changed: restarting timer.
2022-06-30T13:33:10.694Z [SMOOTHING0] Smoothing queue is cleared
2022-06-30T13:33:10.694Z [SMOOTHING0] Selecting config (30) => type: Linear, dirMode: true, pause: false, settlingTime: 200ms, interval: 40ms (25Hz), antiFlickTres: 0, antiFlickStep: 0, antiFlickTime: 0
2022-06-30T13:33:10.694Z [SMOOTHING0] Clearing queued colors before: disabling
2022-06-30T13:33:10.694Z [SMOOTHING0] Smoothing queue is cleared
2022-06-30T13:33:20.573Z [EFFECT0(Music:...)] The effect quits with priority: 1
2022-06-30T13:33:20.573Z [EFFECT0(Music:...)] Releasing sound handle 7 for effect named: 'Music: equalizer test (turn on video preview)'
2022-06-30T13:33:20.573Z [HYPERHDR] Sound device is stopping
2022-06-30T13:33:20.573Z [HYPERHDR] Disconnecting from sound driver: 'default:CARD=Capture | FHD Capture, USB Audio
Default Audio Device | Input'
2022-06-30T13:33:25.678Z [HYPERHDR] Sound device has stopped
2022-06-30T13:33:20.573Z [MUXER0] Removed source priority 1
2022-06-30T13:33:20.574Z [MUXER0] Set visible priority to 240
2022-06-30T13:33:20.574Z [IMAGETOLED0] Total index number is: 16068 (memory: 16068). User sparse processing is: enabled, image size: 640 x 360, area number: 106
2022-06-30T13:33:20.574Z [HYPERHDR0] New priority[240], previous [1]
2022-06-30T13:33:20.574Z [EFFECTENGINE0] Effect 'Music: equalizer test (turn on video preview)' has finished.
2022-06-30T13:33:20.574Z [EFFECT0(Music:...)] Deleting effect named: 'Music: equalizer test (turn on video preview)'
2022-06-30T13:33:20.574Z [EFFECT0(Music:...)] Effect named: 'Music: equalizer test (turn on video preview)' is deleted
2022-06-30T13:33:20.574Z [SMOOTHING0] Clearing queued colors before: enabling. Smoothing configuration changed: restarting timer.
2022-06-30T13:33:20.574Z [SMOOTHING0] Smoothing queue is cleared
2022-06-30T13:33:20.575Z [SMOOTHING0] Selecting config (0) => type: Alternative, dirMode: false, pause: false, settlingTime: 150ms, interval: 16ms (62Hz), antiFlickTres: 0, antiFlickStep: 0, antiFlickTime: 0
2022-06-30T13:33:20.575Z [SMOOTHING0] Using alternative smoothing input (0)
2022-06-30T13:33:20.591Z [SMOOTHING0] Using alternative smoothing procedure (0)

awawa-dev commented 2 years ago

Just tested on Linux again: works just fine.

https://user-images.githubusercontent.com/69086569/176713229-67313476-29c8-4727-8538-684ca706f8aa.mp4

Evaluating this incomplete log fragment, the HyperHDR gets the same looped sound and it is not freezed. Otherwise, the effect priority effect would be removed much sooner in the absence of input data. Whether it is a question of the system or the driver is irrelevant to me, because the problem lies in your sound source and cannot reproduce it using my configuration.

gxgani commented 2 years ago

Thanks for the quick response @awawa-dev

https://user-images.githubusercontent.com/3357587/176726983-f8db7710-42a2-4c05-bb72-bf14f3e5e2ac.mp4

I am adding more information that could help debug this issue. When I set to plughw, default, sysdefault I am not able to even record using arecord.

ARECORD HW Params Dump

root@hostname:~/captures# arecord -D hw:CARD=Capture,DEV=0 --dump-hw-params
Warning: Some sources (like microphones) may produce inaudiable results
         with 8-bit sampling. Use '-f' argument to increase resolution
         e.g. '-f S16_LE'.
Recording WAVE 'stdin' : Unsigned 8 bit, Rate 8000 Hz, Mono
HW Params of device "hw:CARD=Capture,DEV=0":
--------------------
ACCESS:  MMAP_INTERLEAVED RW_INTERLEAVED
FORMAT:  S16_LE
SUBFORMAT:  STD
SAMPLE_BITS: 16
FRAME_BITS: 32
CHANNELS: 2
RATE: [8000 48000]
PERIOD_TIME: [1000 1000000]
PERIOD_SIZE: [16 48000]
PERIOD_BYTES: [64 192000]
PERIODS: [2 1024]
BUFFER_TIME: (666 2000000]
BUFFER_SIZE: [32 96000]
BUFFER_BYTES: [128 384000]
TICK_TIME: ALL
--------------------
arecord: set_params:1352: Sample format non available
Available formats:
- S16_LE

ARECORD Audio Clip Recording works

root@hostname:~/captures# arecord -f S16_LE -r 48000 -d 5 -c 2 -D hw:CARD=Capture,DEV=0 test2.wav 
Recording WAVE 'test2.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Stereo
arecord: set_params:1416: Unable to install hw params:
ACCESS:  RW_INTERLEAVED
FORMAT:  S16_LE
SUBFORMAT:  STD
SAMPLE_BITS: 16
FRAME_BITS: 32
CHANNELS: 2
RATE: 48000
PERIOD_TIME: 125000
PERIOD_SIZE: 6000
PERIOD_BYTES: 24000
PERIODS: 4
BUFFER_TIME: 500000
BUFFER_SIZE: 24000
BUFFER_BYTES: 96000
TICK_TIME: 0

ARECORD detected devices

null
    Discard all samples (playback) or generate zero samples (capture)
lavrate
    Rate Converter Plugin Using Libav/FFmpeg Library
samplerate
    Rate Converter Plugin Using Samplerate Library
speexrate
    Rate Converter Plugin Using Speex Resampler
jack
    JACK Audio Connection Kit
oss
    Open Sound System
pulse
    PulseAudio Sound Server
upmix
    Plugin for channel upmix (4,6,8)
vdownmix
    Plugin for channel downmix (stereo) with a simple spacialization
usbstream:CARD=Generic
    HD-Audio Generic
    USB Stream Output
hw:CARD=Generic_1,DEV=0
    HD-Audio Generic, ALC233 Analog
    Direct hardware device without any conversions
hw:CARD=Generic_1,DEV=2
    HD-Audio Generic, ALC233 Alt Analog
    Direct hardware device without any conversions
plughw:CARD=Generic_1,DEV=0
    HD-Audio Generic, ALC233 Analog
    Hardware device with all software conversions
plughw:CARD=Generic_1,DEV=2
    HD-Audio Generic, ALC233 Alt Analog
    Hardware device with all software conversions
default:CARD=Generic_1
    HD-Audio Generic, ALC233 Analog
    Default Audio Device
sysdefault:CARD=Generic_1
    HD-Audio Generic, ALC233 Analog
    Default Audio Device
front:CARD=Generic_1,DEV=0
    HD-Audio Generic, ALC233 Analog
    Front output / input
dsnoop:CARD=Generic_1,DEV=0
    HD-Audio Generic, ALC233 Analog
    Direct sample snooping device
dsnoop:CARD=Generic_1,DEV=2
    HD-Audio Generic, ALC233 Alt Analog
    Direct sample snooping device
usbstream:CARD=Generic_1
    HD-Audio Generic
    USB Stream Output
hw:CARD=Capture,DEV=0
    FHD Capture, USB Audio
    Direct hardware device without any conversions
plughw:CARD=Capture,DEV=0
    FHD Capture, USB Audio
    Hardware device with all software conversions
default:CARD=Capture
    FHD Capture, USB Audio
    Default Audio Device
sysdefault:CARD=Capture
    FHD Capture, USB Audio
    Default Audio Device
front:CARD=Capture,DEV=0
    FHD Capture, USB Audio
    Front output / input
dsnoop:CARD=Capture,DEV=0
    FHD Capture, USB Audio
    Direct sample snooping device
usbstream:CARD=Capture
    FHD Capture
    USB Stream Output

gxgani commented 2 years ago

Adding plughw debug session

https://user-images.githubusercontent.com/3357587/176729098-b51c3b9c-49b1-468d-a685-44c808d4669a.mp4

awawa-dev commented 2 years ago

Could you test 22050 one channel (mono) capturing using arecord?

gxgani commented 2 years ago

Capturing with both the source fails (mono and stereo) with arecord: set_params:1416: Unable to install hw params:


root@kotl:~/captures# arecord -f S16_LE -r 22050 -d 5 -c 1 -D hw:CARD=Capture,DEV=0 test-22050.wav 
Recording WAVE 'test-22050.wav' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
arecord: set_params:1358: Channels count non available

root@hostname:~/captures# arecord -f S16_LE -r 22050 -d 5 -c 1 -D plughw:CARD=Capture,DEV=0 test-22050.wav 
Recording WAVE 'test-22050.wav' : Signed 16 bit Little Endian, Rate 22050 Hz, Mono
arecord: set_params:1416: Unable to install hw params:
ACCESS:  RW_INTERLEAVED
FORMAT:  S16_LE
SUBFORMAT:  STD
SAMPLE_BITS: 16
FRAME_BITS: 16
CHANNELS: 1
RATE: 22050
PERIOD_TIME: (124988 124989)
PERIOD_SIZE: 2756
PERIOD_BYTES: 5512
PERIODS: (4 5)
BUFFER_TIME: 500000
BUFFER_SIZE: 11025
BUFFER_BYTES: 22050
TICK_TIME: 0

root@hostname:~/captures# arecord -f S16_LE -r 22050 -d 5 -c 2 -D plughw:CARD=Capture,DEV=0 test-22050.wav 
Recording WAVE 'test-22050.wav' : Signed 16 bit Little Endian, Rate 22050 Hz, Stereo
arecord: set_params:1416: Unable to install hw params:
ACCESS:  RW_INTERLEAVED
FORMAT:  S16_LE
SUBFORMAT:  STD
SAMPLE_BITS: 16
FRAME_BITS: 32
CHANNELS: 2
RATE: 22050
PERIOD_TIME: (124988 124989)
PERIOD_SIZE: 2756
PERIOD_BYTES: 11024
PERIODS: (4 5)
BUFFER_TIME: 500000
BUFFER_SIZE: 11025
BUFFER_BYTES: 44100
TICK_TIME: 0

root@hostname:~/captures# arecord -f S16_LE -r 22050 -d 5 -c 2 -D hw:CARD=Capture,DEV=0 test-22050.wav 
Recording WAVE 'test-22050.wav' : Signed 16 bit Little Endian, Rate 22050 Hz, Stereo
arecord: set_params:1416: Unable to install hw params:
ACCESS:  RW_INTERLEAVED
FORMAT:  S16_LE
SUBFORMAT:  STD
SAMPLE_BITS: 16
FRAME_BITS: 32
CHANNELS: 2
RATE: 22050
PERIOD_TIME: (124988 124989)
PERIOD_SIZE: 2756
PERIOD_BYTES: 11024
PERIODS: (4 5)
BUFFER_TIME: 500000
BUFFER_SIZE: 11025
BUFFER_BYTES: 44100
TICK_TIME: 0

awawa-dev commented 2 years ago

Does mono (-c 1) change anything for 22050 and 48000Hz? HyperHDR is using little endian 22050Hz mono format for capturing and it's supported on most platforms and for most grabbers unless some manufacture decides to provide limited UVC capabilities (it's rather not a hardware limitation since down-sampling should be easy to achieved). Unfortunately migration for 44000/48000Hz would require to rewrite some code related to FFT and that affects other OS as well (Windows/macOS/Linux).

gxgani commented 2 years ago

Does mono (-c 1) change anything for 22050 and 48000Hz? HyperHDR is using little endian 22050Hz mono format for capturing and it's supported on most platforms and for most grabbers unless some manufacture decides to provide limited UVC capabilities (it's rather not a hardware limitation since down-sampling should be easy to achieved). Unfortunately migration for 44000/48000Hz would require to rewrite some code related to FFT and that affects other OS as well (Windows/macOS/Linux).

It does captures for 48000 and 8000. I haven't peaked at the source but is it because the fft algorithm is bound to 22050? I was thinking to either have a go at a personal build or find a way to loop this source to lower bitrate.

awawa-dev commented 2 years ago

FFT handling is SoundCapture.cpp/h and with OS dependent part SoundCapLinux.cpp it's adjusted to 22050. Migration to 44100 requires changing some parameters to maintain proper bucket computing (frequencies range bars visible in the equalizer effect). Easier way it to change SoundCapLinux.cpp to capture 44100 mono, then create on fly 22050 buffer from every second word and pass it to SoundCapture.cpp.

awawa-dev commented 2 years ago

If 44100 is not supported (it's a standard ... frankly I don't understand it why they released such device without it), then for 48000 it definitely requires FFT adjustment: the buffer sound size and buckets' parameters for FFT.

gxgani commented 2 years ago

If 44100 is not supported (it's a standard ... frankly I don't understand it why they released such device without it), then for 48000 it definitely requires FFT adjustment: the buffer sound size and buckets' parameters for FFT.

Understandable, but I'm willing to try! I'm fairly surprised as well! I opted this purely because of it's functionality to bypass/loopout HDMI as-is, they also seem to have good-enough support from where I got it.

awawa-dev commented 2 years ago

Yes, ezcoo support is nice and they provide neccesery updates for their devices.

And for migration to 48000. SoundCapLinux.h: SOUNDCAPLINUX_BUF_LENP increase by one to 11, SoundCapLinux.cpp unsigned int exactRate = 22050; to 48000, change else if (exactRate != 22050) to 48000

in SoundCapture.cpp change int16_t resolutionP = 10; to 11 but buckets wont be accurate for 48khz, since we are not limited to 11025 but to 24000 for FFT. Anyway it's pure for the visual effect not strict pharmacy rules, you can try to multiply each element of sec table by 2 I think. Test it with online freqency generators like this. Good luck.

int16_t sec[SOUNDCAP_RESULT_RES] = {
                2 * 2,   // 0-86
                4 * 2,   // 86-258
                6 * 2,   // 258-516
                12 * 2,  // 516-1032
                24 * 2,  // 1032-2064
                48 * 2,  // 2064-4128
                48 * 2,  // 4128-6192
                512    // 6192-11025
    };

EDIT: Oh, SOUNDCAP_N_WAVE to 2048, SOUNDCAP_LOG2_N_WAVE to 11 in SoundCapture.h But probably new sin LUT table (_lutSin) must be computed...

gxgani commented 2 years ago

I got some time to build and deploy today but it seems to have hard time setting the rates

Log

2022-07-02T14:04:10.661Z [HYPERHDR] Sound device is starting
2022-07-02T14:04:10.661Z [HYPERHDR] Opening device: hw:CARD=Capture,DEV=0
2022-07-02T14:04:10.661Z [HYPERHDR] Cannot set snd_pcm_hw_params_set_channels: 'Invalid argument'

Diff

diff --git a/include/base/SoundCapture.h b/include/base/SoundCapture.h
index 5bb2549..92bc9fb 100644
--- a/include/base/SoundCapture.h
+++ b/include/base/SoundCapture.h
@@ -10,8 +10,8 @@
 #include <QColor>
 #include <effectengine/AnimationBaseMusic.h>

-#define SOUNDCAP_N_WAVE      1024
-#define SOUNDCAP_LOG2_N_WAVE 10
+#define SOUNDCAP_N_WAVE      2048
+#define SOUNDCAP_LOG2_N_WAVE 11

 #define SOUNDCAP_RESULT_RES 8

diff --git a/include/grabber/SoundCapLinux.h b/include/grabber/SoundCapLinux.h
index d03e59a..a7aa3e5 100644
--- a/include/grabber/SoundCapLinux.h
+++ b/include/grabber/SoundCapLinux.h
@@ -11,7 +11,7 @@

 #include <base/SoundCapture.h>

-#define SOUNDCAPLINUX_BUF_LENP 10
+#define SOUNDCAPLINUX_BUF_LENP 11
 class SoundCapLinux : public SoundCapture
 {
     friend class HyperHdrDaemon;
diff --git a/sources/base/SoundCapture.cpp b/sources/base/SoundCapture.cpp
index dd5cafe..3264c20 100644
--- a/sources/base/SoundCapture.cpp
+++ b/sources/base/SoundCapture.cpp
@@ -296,7 +296,7 @@ SoundCaptureResult* SoundCapture::hasResult(AnimationBaseMusic* effect, uint32_t

 bool SoundCapture::AnaliseSpectrum(int16_t soundBuffer[], int sizeP)
 {
-   int16_t resolutionP = 10;
+   int16_t resolutionP = 11;
    int16_t sec[SOUNDCAP_RESULT_RES] = {
                2 * 2,   // 0-86
                4 * 2,   // 86-258
diff --git a/sources/grabber/SoundCapLinux/SoundCapLinux.cpp b/sources/grabber/SoundCapLinux/SoundCapLinux.cpp
index 03458cf..7fe47b1 100644
--- a/sources/grabber/SoundCapLinux/SoundCapLinux.cpp
+++ b/sources/grabber/SoundCapLinux/SoundCapLinux.cpp
@@ -112,7 +112,7 @@ void SoundCapLinux::Start()
    {
        int             status;     
        bool            error = false;      
-       unsigned int    exactRate = 22050;      
+       unsigned int    exactRate = 48000;      

        snd_pcm_uframes_t periodSize = (1 << SOUNDCAPLINUX_BUF_LENP) * 2;
        snd_pcm_uframes_t bufferSize = periodSize * 2;
@@ -161,9 +161,9 @@ void SoundCapLinux::Start()
                Error(Logger::getInstance("HYPERHDR"), "Cannot set snd_pcm_hw_params_set_rate_near: '%s'", snd_strerror (status));
                throw 4;
            }
-           else if (exactRate != 22050)
+           else if (exactRate != 48000)
            {
-               Error(Logger::getInstance("HYPERHDR"), "Cannot set rate to 22050");
+               Error(Logger::getInstance("HYPERHDR"), "Cannot set rate to 48000");
                throw 5;
            }

awawa-dev commented 2 years ago

The error is thrown from here:

            if ((status = snd_pcm_hw_params_set_channels (_handle, hw_params, 1)) < 0) {
                Error(Logger::getInstance("HYPERHDR"), "Cannot set snd_pcm_hw_params_set_channels: '%s'",snd_strerror (status));
                throw 6;
            }

it tries to set mono channel (1) for 48000 now and it fails. You can try to set it to stereo (2) to see if it helps, but it will require more code to rewrite later. It's very picky device.

gxgani commented 2 years ago

Quick update, I'm now able to use default:CARD=Capture

https://user-images.githubusercontent.com/3357587/177004854-3830bb6d-e5d8-4c54-bb51-0d443711c9fa.mp4

But if you notice if I switch the viz, it intermittently crashes and reloads, not sure why that's happening but I'm seeing

2022-07-02T14:25:17.583Z [V4L2:FHD CAPTURE: F] Frame too small: 0 != 1843200
2022-07-02T14:25:17.583Z [V4L2:FHD CAPTURE: F] Frame too small: 0 != 1843200
2022-07-02T14:25:17.583Z [V4L2:FHD CAPTURE: F] Frame too small: 0 != 1843200
2022-07-02T14:25:17.583Z [V4L2:FHD CAPTURE: F] Frame too small: 0 != 1843200

full log

022-07-02T14:23:59.673Z [IMAGETOLED0] Total index number is: 60572 (memory: 60572). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:00.001Z [V4L2:FHD CAPTURE: F] Video cache: enabled, size: 1, hits: 18, needed: 1, cleanup: 0, limit: 8
2022-07-02T14:24:00.670Z [IMAGETOLED0] Total index number is: 46344 (memory: 46344). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:04.153Z [WEBSOCKET] (JsonAPI.cpp:1258) log streaming activated for client ::ffff:192.168.1.104
2022-07-02T14:24:11.552Z [IMAGETOLED0] Total index number is: 46088 (memory: 46088). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:13.435Z [IMAGETOLED0] Total index number is: 46344 (memory: 46344). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:18.051Z [IMAGETOLED0] Total index number is: 46216 (memory: 46216). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:20.851Z [IMAGETOLED0] Total index number is: 46344 (memory: 46344). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:24.417Z [IMAGETOLED0] Total index number is: 46216 (memory: 46216). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:26.217Z [IMAGETOLED0] Total index number is: 46344 (memory: 46344). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:34.149Z [IMAGETOLED0] Total index number is: 45960 (memory: 45960). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:35.183Z [IMAGETOLED0] Total index number is: 46344 (memory: 46344). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:36.849Z [IMAGETOLED0] Total index number is: 45960 (memory: 45960). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:38.682Z [IMAGETOLED0] Total index number is: 46344 (memory: 46344). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:43.932Z [IMAGETOLED0] Total index number is: 45960 (memory: 45960). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:47.681Z [IMAGETOLED0] Total index number is: 45832 (memory: 45832). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:24:49.864Z [IMAGETOLED0] Total index number is: 46344 (memory: 46344). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:25:00.013Z [PERFORMANCE] [USB capturing: FPS = 60.01, decoding = 2ms, frames = 3601, invalid = 0], [INSTANCE0: FPS = 60.01, processed = 3601], [LED0: FPS = 62.50, send = 3750, processed = 3750]
2022-07-02T14:25:02.312Z [IMAGETOLED0] Total index number is: 45960 (memory: 45960). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:25:03.979Z [IMAGETOLED0] Total index number is: 46088 (memory: 46088). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:25:04.873Z [EFFECTENGINE0] Run effect "Music: quatro for LED strip (MULTI COLOR FAST)" on channel 1
2022-07-02T14:25:04.873Z [EFFECTENGINE0] (EffectEngine.cpp:174) Start the effect: name [Music: quatro for LED strip (MULTI COLOR FAST)], smoothCfg [23]
2022-07-02T14:25:04.873Z [MUXER0] Register new input 'Web Configuration@::ffff:192.168.1.104/EFFECT' with priority 1 as inactive
2022-07-02T14:25:04.873Z [HYPERHDR] Sound device is starting
2022-07-02T14:25:04.873Z [HYPERHDR] Opening device: default:CARD=Capture
2022-07-02T14:25:04.878Z [HYPERHDR] Sound period size = 1024
2022-07-02T14:25:04.878Z [HYPERHDR] Sound buffer size = 8192
2022-07-02T14:25:04.879Z [HYPERHDR] Sound device has started
2022-07-02T14:25:04.879Z [EFFECT0(Music:...)] Begin playing the effect with priority: 1
2022-07-02T14:25:04.935Z [HYPERHDR] Sound stream:  succesfully captured audio data and the sound is detected.
2022-07-02T14:25:04.985Z [MUXER0] Priority 1 is now active
2022-07-02T14:25:04.985Z [MUXER0] Set visible priority to 1
2022-07-02T14:25:04.985Z [IMAGETOLED0] Total index number is: 288 (memory: 288). User sparse processing is: enabled, image size: 80 x 45, area number: 106
2022-07-02T14:25:04.985Z [HYPERHDR0] New priority[1], previous [240]
2022-07-02T14:25:04.985Z [SMOOTHING0] Clearing queued colors before: enabling. Smoothing configuration changed: restarting timer.
2022-07-02T14:25:04.986Z [SMOOTHING0] Smoothing queue is cleared
2022-07-02T14:25:04.986Z [SMOOTHING0] Selecting config (23) => type: Linear, dirMode: true, pause: false, settlingTime: 200ms, interval: 40ms (25Hz), antiFlickTres: 0, antiFlickStep: 0, antiFlickTime: 0
2022-07-02T14:25:04.986Z [SMOOTHING0] Clearing queued colors before: disabling
2022-07-02T14:25:04.986Z [SMOOTHING0] Smoothing queue is cleared
2022-07-02T14:25:12.172Z [MUXER0] Removed source priority 1
2022-07-02T14:25:12.173Z [MUXER0] Set visible priority to 240
2022-07-02T14:25:12.173Z [EFFECT0(Music:...)] The effect quits with priority: 1
2022-07-02T14:25:12.173Z [IMAGETOLED0] Total index number is: 60572 (memory: 60572). User sparse processing is: enabled, image size: 1280 x 720, area number: 106
2022-07-02T14:25:12.173Z [EFFECT0(Music:...)] Releasing sound handle 3 for effect named: 'Music: quatro for LED strip (MULTI COLOR FAST)'
2022-07-02T14:25:12.173Z [HYPERHDR] Sound device is stopping
2022-07-02T14:25:12.173Z [HYPERHDR] Disconnecting from sound driver: 'default:CARD=Capture | FHD Capture, USB Audio
Default Audio Device | Input'
2022-07-02T14:25:12.189Z [HYPERHDR] Sound device has stopped
2022-07-02T14:25:12.173Z [HYPERHDR0] New priority[240], previous [1]
2022-07-02T14:25:12.174Z [EFFECTENGINE0] Effect 'Music: quatro for LED strip (MULTI COLOR FAST)' has finished.
2022-07-02T14:25:12.174Z [SMOOTHING0] Clearing queued colors before: enabling. Smoothing configuration changed: restarting timer.
2022-07-02T14:25:12.174Z [SMOOTHING0] Smoothing queue is cleared
2022-07-02T14:25:12.174Z [SMOOTHING0] Selecting config (0) => type: Alternative, dirMode: false, pause: false, settlingTime: 150ms, interval: 16ms (62Hz), antiFlickTres: 0, antiFlickStep: 0, antiFlickTime: 0
2022-07-02T14:25:12.174Z [SMOOTHING0] Using alternative smoothing input (0)
2022-07-02T14:25:12.174Z [EFFECT0(Music:...)] Deleting effect named: 'Music: quatro for LED strip (MULTI COLOR FAST)'
2022-07-02T14:25:12.174Z [EFFECT0(Music:...)] Effect named: 'Music: quatro for LED strip (MULTI COLOR FAST)' is deleted
2022-07-02T14:25:12.191Z [SMOOTHING0] Using alternative smoothing procedure (0)
2022-07-02T14:25:12.287Z [EFFECTENGINE0] Run effect "Music: equalizer test (turn on video preview)" on channel 1
2022-07-02T14:25:12.287Z [EFFECTENGINE0] (EffectEngine.cpp:174) Start the effect: name [Music: equalizer test (turn on video preview)], smoothCfg [30]
2022-07-02T14:25:12.287Z [MUXER0] Register new input 'Web Configuration@::ffff:192.168.1.104/EFFECT' with priority 1 as inactive
2022-07-02T14:25:12.287Z [HYPERHDR] Sound device is starting
2022-07-02T14:25:12.287Z [HYPERHDR] Opening device: default:CARD=Capture
2022-07-02T14:25:17.501Z [HYPERHDR] Cannot open input sound device 'default:CARD=Capture'. Error: 'Connection timed out'
2022-07-02T14:25:17.501Z [HYPERHDR] Sound device has started
2022-07-02T14:25:13.876Z [MUXER0] Priority 240 is now inactive
2022-07-02T14:25:13.877Z [MUXER0] Set visible priority to 255
2022-07-02T14:25:13.877Z [HYPERHDR0] New priority[255], previous [240]
2022-07-02T14:25:13.877Z [HYPERHDR0] No source left -> switch LED-Device off
2022-07-02T14:25:13.877Z [SMOOTHING0] Clearing queued colors before: disabling
2022-07-02T14:25:13.877Z [SMOOTHING0] Smoothing queue is cleared
2022-07-02T14:25:13.877Z [LEDDEVICE_WLED] (LedDevice.cpp:136) Disable device
2022-07-02T14:25:13.877Z [LEDDEVICE_WLED] (LedDevice.cpp:336) Switch off
2022-07-02T14:25:13.877Z [LEDDEVICE_WLED] (LedDeviceWled.cpp:175) 
2022-07-02T14:25:13.877Z [LEDDEVICE_WLED] (LedDevice.cpp:294) Set LED strip to black/power off
2022-07-02T14:25:13.877Z [LEDDEVICE] (ProviderRestApi.cpp:165) PUT: [http://192.168.1.135:80/json/state] [{"on":false,"live":false}]
2022-07-02T14:25:14.009Z [LEDDEVICE] (ProviderRestApi.cpp:194) PUT exit: [http://192.168.1.135:80/json/state] [{"on":false,"live":false}]
2022-07-02T14:25:14.009Z [LEDDEVICE] (ProviderRestApi.cpp:383) Reply.httpStatusCode [200]
2022-07-02T14:25:17.502Z [PERFORMANCE] [USB capturing: FPS = 60.01, decoding = 2ms, frames = 3601, invalid = 0], [LED0: FPS = 62.50, send = 3750, processed = 3750]
2022-07-02T14:25:14.009Z [COMPONENTREG0] LED device: disabled
2022-07-02T14:25:17.502Z [PERFORMANCE] [USB capturing: FPS = 60.01, decoding = 2ms, frames = 3601, invalid = 0]
2022-07-02T14:25:17.502Z [V4L2:FHD CAPTURE: F] The video control requested for the grabber restart due to signal lost
2022-07-02T14:25:17.503Z [V4L2:FHD CAPTURE: F] Stopped
2022-07-02T14:25:17.503Z [V4L2:FHD CAPTURE: F] Multithreading for V4L2 is enabled. Available thread's count 8
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] Found capture device: /dev/video0
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] (V4L2Grabber.cpp:402) Device has 'brightness' control => min: 0, max: 255, default: 128
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] (V4L2Grabber.cpp:408) Device has 'contrast' control => min: 0, max: 255, default: 128
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] (V4L2Grabber.cpp:414) Device has 'saturation' control => min: 0, max: 255, default: 128
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] (V4L2Grabber.cpp:420) Device has 'hue' control => min: -32, max: 31, default: 0
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] (V4L2Grabber.cpp:212) Searching for FHD Capture: FHD Capture (video0) 1280 x 720 @ 60 fps, input: -1 (yuyv)
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] *************************************************************************************************
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] Starting V4L2 grabber. Selected: FHD Capture: FHD Capture (video0) [/dev/video0] 1280 x 720 @ 60 fps yuyv
2022-07-02T14:25:17.504Z [V4L2:FHD CAPTURE: F] *************************************************************************************************
2022-07-02T14:25:17.505Z [V4L2:FHD CAPTURE: F] (V4L2Grabber.cpp:807) Hardware cropping is not supported: ignoring
2022-07-02T14:25:17.505Z [V4L2:FHD CAPTURE: F] Set device input to: Camera 1
2022-07-02T14:25:17.515Z [V4L2:FHD CAPTURE: F] Set resolution to: 1280 x 720
2022-07-02T14:25:17.525Z [V4L2:FHD CAPTURE: F] Set framerate to 60 FPS
2022-07-02T14:25:17.526Z [V4L2:FHD CAPTURE: F] Brightness set to: 128 (default)
2022-07-02T14:25:17.527Z [V4L2:FHD CAPTURE: F] Contrast set to: 128 (default)
2022-07-02T14:25:17.528Z [V4L2:FHD CAPTURE: F] Saturation set to: 128 (default)
2022-07-02T14:25:17.528Z [V4L2:FHD CAPTURE: F] Hue set to: 0 (default)
2022-07-02T14:25:17.528Z [V4L2:FHD CAPTURE: F] (V4L2Grabber.cpp:105) LUT folder location: '/usr/share/hyperhdr/lut'
2022-07-02T14:25:17.528Z [V4L2:FHD CAPTURE: F] LUT file is not found here: /root/.hyperhdr/lut_lin_tables.3d
2022-07-02T14:25:17.528Z [V4L2:FHD CAPTURE: F] (Grabber.cpp:445) LUT file found: /usr/share/hyperhdr/lut/lut_lin_tables.3d
2022-07-02T14:25:17.528Z [V4L2:FHD CAPTURE: F] (Grabber.cpp:460) Index 2 for YUV
2022-07-02T14:25:17.538Z [V4L2:FHD CAPTURE: F] Found and loaded LUT: '/usr/share/hyperhdr/lut/lut_lin_tables.3d'
2022-07-02T14:25:17.538Z [V4L2:FHD CAPTURE: F] Video pixel format is set to: YUYV
2022-07-02T14:25:17.542Z [V4L2:FHD CAPTURE: F] Started
2022-07-02T14:25:17.501Z [EFFECT0(Music:...)] Begin playing the effect with priority: 1
2022-07-02T14:25:17.583Z [V4L2:FHD CAPTURE: F] Frame too small: 0 != 1843200
2022-07-02T14:25:17.583Z [V4L2:FHD CAPTURE: F] Frame too small: 0 != 1843200
2022-07-02T14:25:17.583Z [V4L2:FHD CAPTURE: F] Frame too small: 0 != 1843200
2022-07-02T14:25:17.583Z [V4L2:FHD CAPTURE: F] Frame too small: 0 != 1843200
2022-07-02T14:25:17.677Z [MUXER0] Priority 240 is now active
2022-07-02T14:25:17.677Z [MUXER0] Set visible priority to 240
2022-07-02T14:25:17.677Z [HYPERHDR0] New priority[240], previous [255]
2022-07-02T14:25:17.677Z [HYPERHDR0] New source available -> switch LED-Device on
2022-07-02T14:25:17.677Z [SMOOTHING0] Clearing queued colors before: enabling
2022-07-02T14:25:17.677Z [SMOOTHING0] Smoothing queue is cleared
2022-07-02T14:25:17.677Z [LEDDEVICE_WLED] (LedDevice.cpp:102) Enable device
2022-07-02T14:25:17.677Z [LEDDEVICE_WLED] (LedDevice.cpp:310) Switch on
2022-07-02T14:25:17.677Z [LEDDEVICE_WLED] (LedDeviceWled.cpp:128)

awawa-dev commented 2 years ago

From logs: when you enabled the music effect for second time probably it locked HyperHDR for 5 seconds till timeout occured. Probably it stopped video processing and caused the priority handler to drop the video device device as it didn't send new frames. Other explanation is the grabber disconnected from the system then but I don't see errors related to that (maybe there is something in dmesg)

2022-07-02T14:25:12.287Z [HYPERHDR] Opening device: default:CARD=Capture
2022-07-02T14:25:17.501Z [HYPERHDR] Cannot open input sound device 'default:CARD=Capture'. Error: 'Connection timed out'

gxgani commented 2 years ago

Got it! Works flawlessly when triggered as an effect from the API, no delays no hitches! Thanks a lot :)

Do you think it would be feasible to have an option in the UI to set the rate/buffer for audio devices?
Current viz colors seem to be limited, do you think it has to to do with tuning the algorithm for 48000?

https://user-images.githubusercontent.com/3357587/177005635-e8325fb8-72a1-4db8-a20a-af3def77a113.mp4

awawa-dev commented 2 years ago

The main obstacle is FFT: if you didn't change int16_t sec[SOUNDCAP_RESULT_RES] = {...} it incorrectly (well, at least differently that in the original version) assigns frequencies to the bucket. Also constant LUT table for sinus must be changed to dynamic as it's still fixed for processing 22050 https://github.com/awawa-dev/HyperHDR/blob/459aa9b81fe04bfd7a259d761a0fdff94178e10d/sources/base/SoundCapture.cpp#L777

awawa-dev commented 2 years ago

That FFT algorithm is extremely fast but terrible to adjust. Maybe it needs to be replaced even at the expense of more CPU load in v19.

gxgani commented 2 years ago

SOUNDCAP_RESULT_RES

Tried to adjust those tables, the multi color is still a little biased to blue-cyan

awawa-dev commented 2 years ago

I doubt FFT properly functioning without correct sinus table. I gave you link where you can test the frequencies: they should match comments from code pasted here https://github.com/awawa-dev/HyperHDR/issues/291#issuecomment-1171618165 for example first bucket should match aprox. 0-86Hz (first bar visible in the equalizer effect) and so on. Without it colors are unpredictable for multi-colors music effects.

gxgani commented 2 years ago

Going to extensively test and report back! thanks!

gxgani commented 2 years ago

Offtopic but I am trying to add "reload" from https://github.com/hyperion-project/hyperion.ng/blob/master/libsrc/api/JsonAPI.cpp#L974-L988

I added Process header and source but while building I am not able to get past linker not able to refer the method

[100%] Linking CXX executable ../../bin/hyperhdr-remote
../../lib/libhyperhdr-api.a(JsonAPI.cpp.o):JsonAPI.cpp:function JsonAPI::handleConfigCommand(QJsonObject const&, QString const&, int): error: undefined reference to 'Process::restartHyperhdr(bool)'
collect2: error: ld returned 1 exit status
make[2]: *** [sources/hyperhdr-remote/CMakeFiles/hyperhdr-remote.dir/build.make:164: bin/hyperhdr-remote] Error 1
make[1]: *** [CMakeFiles/Makefile2:2464: sources/hyperhdr-remote/CMakeFiles/hyperhdr-remote.dir/all] Error 2

awawa-dev commented 2 years ago

That's the problem when you try to call restartHyperhdr under Linux obraz

gxgani commented 2 years ago

That's the problem when you try to call restartHyperhdr under Linux

I've used this source, the definition for linux is right beneath it (I've renamed the methods) https://github.com/hyperion-project/hyperion.ng/blob/master/libsrc/utils/Process.cpp

awawa-dev commented 2 years ago

So I don't know what else also has changed. Notice that you received an error for hyperhdr-remote app, not hyperhdr app. Maybe it's nothing or maybe you have just modified only one project.

awawa-dev commented 2 years ago

If nothing helps just migrate namespace to a class ans use static methods instead of its naked function. It should help with linker when you don't want to modify cmake project structure.

BTW when you add Process.h/cpp you should re-run cmake for the sources/utils to refresh&add new files. Probably it's best to remove cmakecache file from build folder and run configuration again.

awawa-dev / HyperHDR