search

JorenSix / Panako

The Panako acoustic fingerprinting system.
GNU Affero General Public License v3.0
185 stars 38 forks source link

Monitor is hanging on big files #16

Closed MZehren closed 2 years ago

MZehren commented 5 years ago

Hello all,

I'm using "monitor" from Panako to analyse big files of music (1-2 hours). But after 10 minutes the process is hanging and never ends, the processor doesn't seem to be used anymore at this stage.

No exception is raised nor logged in the log files (I set the log configuration to "ALL"), and the last window analysed seemed to be random, even though it is always around the 1600th window analysed and about 10 minutes of computation. Cutting the file in small segments of 15 minutes and running "monitor" on each one of them solves the issue, but this is not convenient.

I'm using a custom configuration with Panako from @werthen's master thesis: https://lib.ugent.be/fulltxt/RUG01/002/494/859/RUG01-002494859_2018_0001_AC.pdf I tried to find and increase the settings linked to stop conditions with no success so far. Here's the configuration I'm currently using:

STRATEGY=NCTEQ MONITOR_STEP_SIZE=15 MONITOR_OVERLAP=14 NFFT_SIZE=512 NFFT_MIN_FILTER_WINDOW_SIZE=7 NFFT_USE_PHASE_REFINED_HASH=FALSE NFFT_MAPDB_DATABASE=/home/mickael/Documents/programming/Panako/dbs/nfft_panako_db NFFT_EVENT_POINTS_MAX_PER_FFT_FRAME=100 NFFT_STEP_SIZE=256 NFFT_MAX_FINGERPRINTS_PER_EVENT_POINT=100 NFFT_EVENT_POINT_MIN_DISTANCE=300 NFFT_SAMPLE_RATE=8000 NFFT_MAX_FILTER_WINDOW_SIZE=15 HTTP_SERVER_PORT=8080 META_DATA_COMMAND=/opt/panako/extract_metadata.sh DECODER_PIPE_LOG_FILE=decoder_log.txt SYNC_MIN_ALIGNED_MATCHES=7 CHECK_DUPLICATE_FILE_NAMES=TRUE DECODER_PIPE_ENVIRONMENT=/bin/bash AVAILABLE_PROCESSORS=0 SYNC_MARKER_TIME_ERROR_ALLOWED=10 PCH_SAMPLE_RATE=22050 DECODER_PIPE_BUFFER_SIZE=44100 PCH_SIZE=2048 DECODER_PIPE_ENVIRONMENT_ARG=-c DECODER=PIPE MAX_FILE_SIZE=2097152000 PCH_OVERLAP=1024 DECODER_PIPE_COMMAND=ffmpeg -ss %input_seeking% %number_of_seconds% -i "%resource%" -vn -ar %sample_rate% -ac %channels% -sample_fmt s16 -f s16le pipe\:1 PCH_FILES=/home/mickael/Documents/programming/Panako/dbs/pch SYNC_MARKER_LOUDNESS_DELTA=17 META_DATA_DIRECTORY=/opt/panako/metadata SYNC_MARKER_MAX_TIMESTAMP=3 NCTEQ_MAPDB_DATABASE=/home/mickael/Documents/programming/Panako/dbs/cteq_panako_db_2 NCTEQ_SAMPLE_RATE=44100 NCTEQ_STEP_SIZE=1536 NCTEQ_MIN_FREQ=3700 NCTEQ_MAX_FREQ=12200 NCTEQ_BINS_PER_OCTAVE=36 NCTEQ_EVENT_POINTS_PER_SECOND_FOR_QUERY=400 NCTEQ_EVENT_POINTS_PER_SECOND_FOR_STORAGE=200 NCTEQ_EVENT_POINT_FREQUENCY_DELTA_MAX=1066 NCTEQ_EVENT_POINT_BRANCHING_FOR_STORAGE=200 NCTEQ_EVENT_POINT_BRANCHING_FOR_QUERY=600 NCTEQ_MAX_HASH_COLLISIONS=10000 NCTEQ_MINIMUM_MATCHES_THRESHOLD=0 NCTEQ_MINIMUM_ALIGNED_MATCHES_THRESHOLD=0

Am I missing a setting stopping the computation somehow? I increased NCTEQ_MAX_HASH_COLLISIONS, SYNC_MARKER_MAX_TIMESTAMP, and SYNC_MARKER_TIME_ERROR_ALLOWED with no impact on the issue. But it's hard to find the correct configuration as I didn't find any description of those settings.

VictorTrapenok commented 5 years ago

Hi, I use NFFT for recognition. And It work fine, and files more than 10 minutes too. Do you ever try use other STRATEGY with your files?

MZehren commented 5 years ago

Hi, thank you for your help.

I was able to reproduce the same issue on a different machine (still ubuntu 18.04 though). This occurred both with the strategy NFFT and NCTEQ and the exact same set of parameters as detailed in my previous message.

But you were right to point to a different strategy: I had to use a longer file to make the process stop with NFFT. It stopped after ~6000 segments analysed (~100 minutes in the file).

JorenSix commented 2 years ago

The Panako code has been completely rewritten for version 2. I have tested the new code with 100+ minute files and did not experience the problem. Please let me know if you still run in to this problem.