search

orcasound / aifororcas-livesystem

Real-time AI-assisted killer whale notification system (model and moderator portal) :star:
http://orcahello.ai4orcas.net/
MIT License
37 stars 25 forks source link
artificial-intelligence audio-analysis audio-processing bioacoustics deep-learning inference machine-learning marine marine-biology neural-network orcas real-time realtime whales

readme

OrcaHello: A real-time AI-assisted killer whale notification system 🎱 🐋

OpenSSF Scorecard

Orcasound maintains a hydrophone network in Puget Sound (near Seattle, WA, USA, Northeast Pacific). Killer whales (aka orcas) often swim by these hydrophones (underwater microphones) and vocalize with a wide range of calls.

Through annual Microsoft hackathons since 2019 and with the volunteer efforts of many heroic Orcasound open source contributors, we have trained and continue to refine a deep learning model to find these calls in live hydrophone audio. The model is at the core of the real time inference system we call OrcaHello which aims to help recover the endangered population of Southern Resident Killer Whale (SRKW) - the iconic orcas that frequently seek salmon in Puget Sound, and also range annually from California to Alaska. OrcaHello is a part of ai4orcas.net, an International effort to apply cutting-edge artificial intelligence to orca conservation.

Learn more about OrcaHello via:

This repository contains the implementations for the following components that make up the OrcaHello real time inference system:

System overview

The diagram below describes the flow of data through OrcaHello and the technologies used.

System Overview

As of September, 2022, the data flow steps include:

  1. Live streaming of audio data via AWS (from Raspberry Pis running orcanode code to Orcaound's S3 open data registry buckets)
  2. Azure-based analysis (via AKS in 2021-2, ICI 2019-2020; ingestion of 10-second segments from S3, inference on 2-second samples using the current OrcaHello binary call classifier, concatenation of raw audio into 60-second WAV files and spectrogram generation)
  3. Moderation of model detections by orca call experts (moderator notification, authentication in moderator portal, annotation and validation)
  4. Notification of confirmed calls from endangered orcas for a wide range of subscribers (researchers, community scientists, managers, outreach/education network nodes, marine mammal observers, dynamic mitigation systems, oil spill response agencies, enforcement agencies, Naval POCs for sonar testing/training situational awareness, etc.)

Each overlapping 2-second data segment is classified as a whale call or / not. Shown below is a 1-minute segment of hydrophone audio visualized as a spectrogram with whale calls detected by the model (delineated by white boundaries).

Detections

When whale activity is detected by the model, it sends an email to our Moderators who are killer whale experts (bioacousticians).

Moderator Email

Once they receive this notification, they can visit the public Moderator Portal shown below to confirm or reject model detections, and to annotate each candidate.

Moderator Portal

Most importantly, they confirm whether or not the whale call was emitted by an endangered Southern Resident Killer Whale (SRKW). If a SRKW is confirmed, notifications are sent to subscribers, like this email message (2022 example):

Subscriber Email

Contributing

You can contribute by

  1. Creating an issue to capture problems with the Moderator Portal and documentation here.
  2. Forking the repo and generating a pull request to fix an issue with the code or documentation.
  3. Joining the Orcasound open source organization on Github to edit the wiki and/or help review pull requests.

To contribute a pull request for a specific subsystem, please read the corresponding contributing guidelines and READMEs.

Current subteams and leads are as follows:

New volunteers are welcome in all subteams.

General Resources

Project Page - contains information about the system and a brief history of the project.

Related Projects