Queue is full!! with alexbelgium HA Addon

[thor0215]

My setup has a rstp audio source and after an hour or so now, the addon logs will show "2024/06/24 10:30:49 Queue is full!" I searched the Birdnet-go source code and couldn't find where that message would even be coming from. I'm not sure how to easily reproduce this, but if I could figure out where the queue is, I could clear it.

Log snippet: 2024/06/24 10:30:18 Queue is full! 2024/06/24 10:30:19 Queue is full! 2024/06/24 10:30:21 Queue is full! 2024/06/24 10:30:22 Queue is full! 2024/06/24 10:30:24 Queue is full! 2024/06/24 10:30:25 Queue is full! 2024/06/24 10:30:27 Queue is full! 2024/06/24 10:30:28 Queue is full! 2024/06/24 10:30:30 Queue is full! 2024/06/24 10:30:31 Queue is full! 2024/06/24 10:30:33 Queue is full! 2024/06/24 10:30:34 Queue is full! 2024/06/24 10:30:36 Queue is full!

[tphakala]

This is queue between BirdNET Analyzer go routine and Processor go routine, this indicates that something is blocking actions (database writes etc.) which causes results queue to fill. I have never seen this before, only way to clear this queue is to restart birdnet-go process.

Can you share your config.yaml (please remove sensitive data like BirdWeather token if you have configured one). Also share birdnet-go startup messages which contain version and hardware information.

[thor0215]

# BirdNET-Go configuration

debug: true # print debug messages, can help with problem solving
# Node specific settings
main:
    name: BirdNET-Go # name of node, can be used to identify source of notes
    timeas24h: true # true for 24-hour time format, false for 12-hour time format
    log:
        enabled: true # true to enable log file
        path: /root/.config/birdnet-go/birdnet.log # path to log file
        rotation: daily # daily, weekly or size
        maxsize: 1048576 # max size in bytes for size rotation
        rotationday: 0 # day of the week for weekly rotation, 0 = Sunday
# BirdNET model specific settings
birdnet:
    sensitivity: !!float 1 # sigmoid sensitivity, 0.1 to 1.5
    threshold: 0.8 # threshold for prediction confidence to report, 0.0 to 1.0
    overlap: 1.5 # overlap between chunks, 0.0 to 2.9
    threads: 0 # 0 to use all available CPU threads
    locale: en # language to use for labels
    latitude: 44.9478 # latitude of recording location for prediction filtering
    longitude: -93.0902 # longitude of recording location for prediction filtering
    rangefilter:
        model: latest # model to use for range filter: "latest" or "legacy" for previous model
        threshold: 0.01 # rangefilter species occurrence threshold
# Realtime processing settings
realtime:
    interval: 15 # duplicate prediction interval in seconds
    processingtime: false # true to report processing time for each prediction
    audio:
        source: "sysdefault" # audio source to use for analysis
        export:
            enabled: true # true to export audio clips containing indentified bird calls
            debug: false # true to enable audio export debug messages
            path: clips/ # path to audio clip export directory
            type: wav # only wav supported for now
            retention:
                policy: usage # retention policy: none, age or usage
                maxage: 30d # age policy: maximum age of clips to keep before starting evictions
                maxusage: 80% # usage policy: percentage of disk usage to trigger eviction        
                minclips: 2 # minumum number of clips per species to keep before starting evictions
    dynamicthreshold:
        enabled: true # true to enable dynamic confidence threshold
        trigger: 0.90 # dynamic threshold is activated on detections at this confidence level
        min: 0.20 # dynamic threshold will not go lower than this
        validhours: 24 # number of hours to consider for dynamic confidence
    rtsp:
        transport: udp # RTSP Transport Protocol
        urls: # RTSP stream URLs
            - rtsp://192.168.1.21:8080/stream.sdp
            # - rtsp://user:password@example.com/stream1
            # - rtsp://user:password@example.com/stream2
            # - rtsp://user:password@example.com/stream3
    log:
        enabled: false # true to enable OBS chat log
        path: birdnet.txt # path to OBS chat log
    birdweather:
        enabled: false # true to enable birdweather uploads
        locationaccuracy: 500 # accuracy of location in meters
        debug: false # true to enable birdweather api debug mode
        id: 0 # birdweather ID
    openweather:
        enabled: false
        apikey: "" # OpenWeather API key
        endpoint: "https://api.openweathermap.org/data/2.5/weather" # OpenWeather API endpoint
        units: metric # metric or imperial
        language: en # language code
    mqtt:
        enabled: true # true to enable MQTT
        broker: tcp://localhost:1883 # MQTT (tcp://host:port)
        topic: birdnet # MQTT topic
        username: username # MQTT username
        password: password # MQTT password
    privacyfilter: # Privacy filter prevents audio clip saving if human voice 
        enabled: true # is detected durin audio capture
        confidence: 0.05 # threshold for human voice detection
    dogbarkfilter:
        enabled: true
        confidence: 0.1 # confidence threshold for dog bark detection
        remember: 5 # number of minutes to remember dog barks
    telemetry:
        enabled: false # true to enable Prometheus compatible telemetry endpoint
        listen: "0.0.0.0:8090" # IP address and port to listen on
webserver:
    enabled: true # true to enable web server
    port: 8080 # port for web server
    autotls: false # true to enable auto TLS
    log:
        enabled: false # true to enable log file
        path: webui.log # path to log file
        rotation: daily # daily, weekly or size
        maxsize: 1048576 # max size in bytes for size rotation
        rotationday: 0 # day of the week for weekly rotation, 0 = Sunday
# Ouput settings
output:
    file:
        enabled: true # true to enable file output for file and directory analysis
        path: output/ # path to output directory
        type: table # ouput format, Raven table or csv
    # Only one database is supported at a time
    # if both are enabled, SQLite will be used.
    sqlite:
        enabled: true # true to enable sqlite output
        path: birdnet.db # path to sqlite database
    mysql:
        enabled: false # true to enable mysql output
        username: birdnet # mysql database username
        password: secret # mysql database user password
        database: birdnet # mysql database name
        host: localhost # mysql database host
        port: 3306 # mysql database port

[thor0215]

Starting...
/etc/cont-init.d/00-banner.sh: executing

-----------------------------------------------------------
 Add-on: Birdnet-go
 Realtime BirdNET soundscape analyzer
-----------------------------------------------------------
 Add-on version: 0.5.5
 You are running the latest version of this add-on.
 System: Home Assistant OS 12.4  (amd64 / qemux86-64)
 Home Assistant Core: 2024.6.3
 Home Assistant Supervisor: 2024.06.0
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums
-----------------------------------------------------------
 Provided by: https://github.com/alexbelgium/hassio-addons 
-----------------------------------------------------------
/etc/cont-init.d/00-global_var.sh: executing
COMMAND='realtime --rtsp rtsp://192.168.1.21:8080/stream.sdp --rtsptransport udp -d'
Timezone set from Etc/UTC to America/Chicago
/etc/cont-init.d/00-local_mounts.sh: executing
/etc/cont-init.d/00-smb_mounts.sh: executing
/etc/cont-init.d/01-config_yaml.sh: executing
Setting permissions for the config_env.yaml directory

Load environment variables from /config/config_env.yaml if existing
If accessing the file with filebrowser it should be mapped to /addon_configs/db21ed7f-birdnet-go/config_env.yaml
---------------------------------------------------------
Wiki here on how to use : github.com/alexbelgium/hassio-addons/wiki/Add‐ons-feature-:-add-env-variables

... no env variables found, exiting
/etc/cont-init.d/01-custom_script.sh: executing
Execute /addon_configs/db21ed7f-birdnet-go/birdnet-go.sh if existing
Wiki here : github.com/alexbelgium/hassio-addons/wiki/Add-ons-feature-:-customisation
/etc/cont-init.d/32-nginx_ingress.sh: executing
Adapting for ingress
/etc/cont-init.d/33-mariadb.sh: executing
---
MariaDB addon is active on your system! If you want to use it instead of sqlite, here are the informations to encode :
Database user : service
Database password : xxxxxxx
Database name : xxxxxxx
Host-name : core-mariadb:3306
---
/etc/cont-init.d/99-run.sh: executing

[12:23:34] INFO: ALSA_CARD option is set to null. If the microphone doesn't work, please adapt it
[12:23:34] INFO: Starting app...
BirdNET-Go build date: 2024-06-19T17:08:03Z, using config file: /root/.config/birdnet-go/config.yaml
BirdNET GLOBAL 6K V2.4 FP32 model initialized, using 8 threads of available 8 CPUs
2024/06/24 12:23:34 Using latest range filter model
System details: linux debian 12.5 on unknown hardware
Starting analyzer in realtime mode. Threshold: 0.8, overlap: 1.5, sensitivity: 1, interval: 15
2024/06/24 12:23:34 SQLite database connection initialized: birdnet.db
2024/06/24 12:23:34 Loading species config from file: species_config.csv
2024/06/24 12:23:34 Config loaded species: dog confidence: 2.000
2024/06/24 12:23:34 Config loaded species: engine confidence: 2.000
2024/06/24 12:23:34 Config loaded species: environmental confidence: 2.000
2024/06/24 12:23:34 Config loaded species: fireworks confidence: 2.000
2024/06/24 12:23:34 Config loaded species: gun confidence: 2.000
2024/06/24 12:23:34 Config loaded species: human non-vocal confidence: 2.000
2024/06/24 12:23:34 Config loaded species: human vocal confidence: 2.000
2024/06/24 12:23:34 Config loaded species: human whistle confidence: 2.000
2024/06/24 12:23:34 Config loaded species: noise confidence: 2.000
2024/06/24 12:23:34 Config loaded species: power tools confidence: 2.000
2024/06/24 12:23:34 Config loaded species: siren confidence: 2.000
2024/06/24 12:23:34 Config loaded species: common kingfisher confidence: 1.000
2024/06/24 12:23:34 Config loaded species: eurasian moorhen confidence: 1.000
2024/06/24 12:23:34 Config loaded species: common snipe confidence: 1.000
2024/06/24 12:23:34 Config loaded species: tawny owl confidence: 0.400
2024/06/24 12:23:34 Config loaded species: long-eared owl confidence: 0.400
2024/06/24 12:23:34 Config loaded species: boreal owl confidence: 0.400
2024/06/24 12:23:34 Config loaded species: great grey owl confidence: 0.400
2024/06/24 12:23:34 Config loaded species: northern hawk-owl confidence: 0.400
2024/06/24 12:23:34 Config loaded species: snowy owl confidence: 0.400
2024/06/24 12:23:34 Config loaded species: northern pygmy-owl confidence: 0.400
2024/06/24 12:23:34 Connecting to MQTT broker
Logging disabled
2024/06/24 12:23:34 Clip retention policy: usage
2024/06/24 12:23:34 Starting ffmpeg with command:  /usr/bin/ffmpeg -rtsp_transport udp -i rtsp://192.168.1.21:8080/stream.sdp -loglevel error -vn -f s16le -ar 48000 -ac 1 pipe:1
⇨ http server started on [::]:8080
[12:23:34] INFO: Starting NGinx...
2024/06/24 12:23:42 
Using confidence threshold of 1.00 for eurasian moorhen
2024/06/24 12:23:44 
Using confidence threshold of 1.00 for eurasian moorhen
2024/06/24 12:23:45 
Using confidence threshold of 1.00 for eurasian moorhen
2024/06/24 12:23:48 
Using confidence threshold of 2.00 for engine
2024/06/24 12:23:48 
Using confidence threshold of 0.40 for tawny owl
2024/06/24 12:23:50 Species not on included list: Juniper Titmouse
2024/06/24 12:23:51 New detection: House Wren with confidence: 0.94, source: rtsp://192.168.1.21:8080/stream.sdp
2024/06/24 12:23:53 Updating detection: House Wren with confidence: 0.94, source: rtsp://192.168.1.21:8080/stream.sdp
2024/06/24 12:23:54 
Using confidence threshold of 1.00 for common snipe
2024/06/24 12:23:56 
Using confidence threshold of 2.00 for engine
2024/06/24 12:23:56 
Using confidence threshold of 1.00 for eurasian moorhen

[tphakala]

Thank you. I see that you have MQTT enabled. I suspect there may be a blocking bug within the MQTT code causing the results queue to become full.

[thor0215]

I'll turn that off and see if the issue continues. I'm also running it on an Opensuse VM with MQTT enabled to troubleshoot the issue and will see if it occurs over there.

[tphakala]

I just committed a PR (https://github.com/tphakala/birdnet-go/pull/232) which adds timeout for MQTT publish action, can you upgrade to dev container image which contains this change to see it if fixes the issue?

[thor0215]

Now I'm seeing these MQTT log entries:

[16:03:17] INFO: Starting NGinx...
2024/06/24 16:03:23 New detection: American Robin with confidence: 0.86, source: rtsp://192.168.1.21:8080/stream.sdp
2024/06/24 16:03:31 No species config found, using default actions for american robin
2024/06/24 16:03:41 Error executing action: publish timeout
2024/06/24 16:03:46 Failed to connect to MQTT broker (attempt 1/5): connection timeout
2024/06/24 16:03:46 Retrying in 1s
2024/06/24 16:03:47 Connecting to MQTT broker
2024/06/24 16:03:47 Failed to connect to MQTT broker (attempt 2/5): already connected
2024/06/24 16:03:47 Retrying in 2s
2024/06/24 16:03:49 Connecting to MQTT broker
2024/06/24 16:03:49 Failed to connect to MQTT broker (attempt 3/5): already connected
2024/06/24 16:03:49 Retrying in 4s
2024/06/24 16:03:53 Connecting to MQTT broker
2024/06/24 16:03:53 Failed to connect to MQTT broker (attempt 4/5): already connected
2024/06/24 16:03:53 Retrying in 8s
2024/06/24 16:04:01 Connecting to MQTT broker
2024/06/24 16:04:01 Failed to connect to MQTT broker (attempt 5/5): already connected
2024/06/24 16:04:01 Failed to connect to MQTT broker after maximum retries
2024/06/24 16:04:07 Human detected, confidence 0.228777

[thor0215]

After more investigation, the main issue was using the wrong broker hostname for the configuration. Because I'm using the MQTT Broker addon, I needed the mqtt broker in config.yaml set to tcp://core-mosquitto:1883.

[08:39:49] INFO: Starting app...
BirdNET-Go build date: 2024-06-24T22:21:00Z, using config file: /root/.config/birdnet-go/config.yaml
2024/06/25 08:39:49 Using latest range filter model
BirdNET GLOBAL 6K V2.4 FP32 model initialized, using 8 threads of available 8 CPUs
System details: linux debian 12.5 on unknown hardware
Starting analyzer in realtime mode. Threshold: 0.8, overlap: 1.5, sensitivity: 1, interval: 15
2024/06/25 08:39:49 Loading species config from file: species_config.csv
2024/06/25 08:39:49 Connecting to MQTT broker
Logging disabled
2024/06/25 08:39:49 Clip retention policy: usage
2024/06/25 08:39:49 Starting ffmpeg with command:  /usr/bin/ffmpeg -rtsp_transport udp -i rtsp://192.168.1.21:8080/stream.sdp -loglevel error -vn -f s16le -ar 48000 -ac 1 pipe:1
⇨ http server started on [::]:8080
2024/06/25 08:39:49 Connected to MQTT broker: tcp://core-mosquitto:1883
2024/06/25 08:39:49 Successfully connected to MQTT broker

[Maarc]

Hi! I have had the same issue. My problem was that my MQTT broker was not correctly configured. I replaced « core-mosquitto » by the IP adress of HA, as I used the HA mosquitto add-on. Then it worked like a charm.

[thor0215]

I added some enhancements to the addon that Alex released yesterday. Now the addon will detect if the mqtt add-on service is available and if so the birdnet-go add-on log will output the mqtt login info like this:

/etc/cont-init.d/33-mqtt.sh: executing
---
MQTT addon is active on your system! Add the MQTT details below to the Birdnet-go config.yaml :
MQTT user : addons
MQTT password : Ri5ahV1aipeiw0aelerooteixai5ohtoeNg6oo3mo0thi5te0phiezuge4Phoore
MQTT broker : tcp://core-mosquitto:1883
---

[tphakala]

@thor0215 thank you for reporting this issue, MQTT client code was lacking several checks for network connectivity issues etc., I rewrote it (well Claude.AI did) to handle broker connectivity issues more gracefully.