tphakala commented 2 months ago

Background

Currently, BirdNET-Go supports only a single RTSP stream. There is a need to develop a solution that allows for the support of multiple streams.

Things to Consider

1. Architecture for Supporting Multiple Streams

Single vs. Multiple Processes: Determine whether a single BirdNET-Go process should handle multiple streams or if this functionality should be distributed across multiple BirdNET-Go instances, each updating a single database.

2. Audio Processing Strategy

Downmixing vs. Individual Analysis: If opting for a single process approach, should audio from multiple RTSP streams be downmixed into a single stream for processing, or should each stream be analyzed individually? This decision involves trade-offs with simplicity, performance, and detection accuracy.

3. Performance and Limitations

Inference Performance: With the potential increase in audio streams, BirdNET's inference performance must be carefully considered. The number of streams that can be effectively supported is limited by the hardware used by the user.

4. User Interface Changes

User Interface Changes: What changes are needed for the Web UI in the case of multiple audio stream sources?

5. Implementation of RTSP Stream Analysis

Native Go vs. FFmpeg Approach: Assess whether RTSP stream analysis should be implemented natively in Go, moving away from the current reliance on FFmpeg. FFmpeg has proven reliable and supports a wide range of audio codecs, Go has a limited set of audio libraries available for audio decoding.

This issue aims to gather feedback, suggestions, and potential contributions from the community to address these considerations comprehensively.

becausebirds commented 2 months ago

My thoughts on multiple RTSP streams:

I would be nice to filter detections by stream (e.g. frontyard, backyard)
I'm curious if any "deduping" can be done if two streams make the same unique detection
Something in the UI that indicates the URL was valid and audio is present

matthew73210 commented 2 months ago

Regarding multiple streams, should be able to cut them into segments and feed them in. No need to start another analyser. Just have the source appended to the file name : i.e. backyard.YYYY.MM.DD.HH.MM.SS.wav etc

Swiftnesses commented 1 month ago

@tphakala - https://github.com/mcguirepr89/BirdNET-Pi/pull/818

They've just merged the capability for multiple rtsp streams.

AKHwyJunkie commented 1 month ago

I'm currently using Birdcage due to its multiple RTSP stream support, but that project looks dead and has lots of annoying bugs. I'd like to switch over, but I want/need multiple streams to consider switching.

1, I think a single instance is best. Multiple streams does place more demand on the compute side, but it's just much easier to support. People like me are going to want to bring in like 10-12 audio streams, so multiple instances would be a bear to support.

2, I think analyzing each stream is best. Yes, things like de-duplication will be difficult/impossible with this method, but I do think a lot of unforeseen issues can develop if you mux the streams. (Echoes, interference, slight timing differences due to network latency, louder noises overwhelming quieter ones, etc.)

3, I think this is a given. On Birdcage, I currently analyze 12 streams with 3 vCPU's on an AMD 5950X. Average usage is around 20% with peaks to 30%. Dual CPU would be fine, even in extreme cases. Modern hardware is quite good and efficient, this will only be the trend going forward.

4, After using things for awhile, I found I don't really care about a lot of things like which RTSP stream the audio came from. Users really just want to capture the audio, know what's around and have ways to look at the data.

5, I think sticking with ffMpeg will provide a lot more flexibility. While a lot of RTSP sources use similar codecs and settings, in general I think more options (i.e. compatibility) is better than less options.

Thanks for your interest in getting multiple RTSP streams going. I'd love to see it and would be happy to do any early testing if I can.

tphakala commented 1 month ago

Yeah, I think we can start with simple implementation and improve it over time. I'll try to work on this.

bbrodriguez commented 1 month ago

My RTSP streams are passed first via go2rtc (frigate), and there are only two of them.

In the interim I have put this go2rtc config in place in frigate which creates a combined audio stream (and one copied video stream) by re-streaming the existing go2rtc streams which birdnet-go then picks up.

combined_audio:
  - "ffmpeg:rtsp://127.0.0.1:8554/stream1#rtsp://127.0.0.1:8554/stream2#filter_complex=[0:a][1:a]amerge=inputs=2[aout]#map=0:v#map=[aout]#c:v=copy#c:a=aac"

Seems to work so far. I imagine this would have a scale limit at which point the audio would be too cluttered to be useful.

tphakala commented 4 weeks ago

Support for multiple RTSP streams is now implemented, for more information please see https://github.com/tphakala/birdnet-go/discussions/187

matthew73210 commented 3 weeks ago

Hey,

Since the modification to audio sources, I can't seem to get it working again with my rtsp source. sysdefault / null just hangs the system as there isn't a microphone attached. How do you config it to use rtsp streams now via config.yaml? default lets the system boot but no recordings

tphakala commented 3 weeks ago

Format changed due to supporting multiple sources, URLs are configured like this now

rtsp:
  transport: tcp         # RTSP Transport Protocol
  urls:                  # List of RTSP stream URLs
    - rtsp://user:password@example.com/stream1
    - rtsp://user:password@example.com/stream2