Project 2: Real-time Orca Call Detection in No Time

[valentina-s]

Real-time Orca Call Detection in No Time

This project will focus on binary classification of orca calls in real time.

Tonal whale sound detectors exist in open-source software (e.g. PAMGuard) and can run in real-time. However, these programs are not fully platform-independent, and little is known about how they perform in real-time compared to cloud-based real-time ML pipelines.

The real-time orca prediction toolkit will provide an easy way to deploy trained models, and test them on a real-time audio streams. Users will be able to upload their models in standardized formats, and then deploy them on different data streams. The objective is to make the toolkit flexible enough to be adopted widely by passive acoustic researchers with little technical experience. The processing will occur on a public cloud provider, so the design is not limited by lab-specific servers. Optimizing the processing and minimizing the cost will be of prime importance. The project can be expanded to detect other sounds (by uploading other detectors), and allow comprehensive study of the ocean soundscape.

Required skills: Python, deep learning, signal processing

Bonus skills: cloud computing, parallel processing, Docker

Possible mentors: Jesse Lopez, Scott, Val, Valentina

References and open-source building blocks:

• Models: Tensorflow, Pytorch, Keras, Scikit-learn, ONNX format, TensorflowJS
• Parallel processing: Dask, Kubernetes, Flink, Cloud functions (e.g. lambda), Github Actions
• Virtualization: docker
• Ketos: Python package for acoustic data analysis with neural networks
• Data streams:
  • Orcasound raw data: aws --no-sign-request s3 ls streaming-orcasound-net/rpi_bush_point/hls/ (files are in .ts format (~10 sec each) , you can use ffmpeg to convert it to .avi)
  • Ocean Observatories Initiative data: https://rawdata.oceanobservatories.org/files/RS01SLBS/LJ01A/09-HYDBBA102/2020/03/ (files are in the .mseed format which can be opened with the python obspy library)
• Existing classifiers:
  • OrcaCNN: https://github.com/axiom-data-science/OrcaCNN/tree/master/Detection/Labelling (Alaska Orcas - prelim)
  • Spectral Peaks: https://github.com/orcasound/orcadata/tree/master/WAV_FLACprocessor_Val/ProcessLocalFilesForCalls

Points to consider in the proposal:

Design of the system:

• What are the inputs, what are the outputs?
• What are the package requirements?

Performance:

• How long does it take? What are the time limitations?
• What resources can be used to speed it up? Is there associated cost?

Getting started (these are suggestions, there is no one way to go):

• Set up a docker file for running a simple classifier: background noise vs something else (or one of the existing classifiers) on the Orcasound raw data.
• Use github actions to set up an automatic classification on an Orcasound raw file.
• We will possibly have some cloud credits (form Microsoft Azure) for the summer! Explore what tools can be useful for your project (you do not have to set up an account, just read on the options and identify how they can be integrated with your work).

[rishiraj]

Really interesting project to work on. I am sharing my Draft Proposal to promote more open development of innovative ideas. Feedback from Mentors and Orcasound experts would be highly appreciated.

https://docs.google.com/document/d/1w7FtKRzoXLnWc350gh4cWn2Q_draUSNzT5DEnQacm-o/edit?usp=drivesdk

[MoAmrYehia]

This is my draft. Thank you for your time and I really excited for your feedback.

https://docs.google.com/document/d/1p3Xf86C6ciit4K3tIVujX4fbfSHa4Cb0JUB-J7xVY_4/edit?usp=sharing

[nikhilrana015]

This is my GSoC draft. Excited for your feedback. https://docs.google.com/document/d/1o7WRiHJW01YWgxkNnH56J9cKfpiJQJWNoNP8JDb8C48/edit?usp=sharing

[dharmsen]

Linked here is my preliminary draft. I look forward to hearing your feedback. Thanks in advance for your time. https://docs.google.com/document/d/1rNihm7jI5eBdJRP2d1YxVSK3JpxUByLXBaPtVSkU4AE/edit?usp=sharing

[abhinav99ash29]

Here is my draft for idea 2. Please share your feedback Thanks

https://docs.google.com/document/d/1B_RC4_j57e1ceTXjvaa1w3llBWk48LWbxLubOY945LE/edit?usp=sharing

[sainimohit23]

Here is my draft for the given project:

https://docs.google.com/document/d/1FnwVOES2W4ibBjGuVFIwSM59_tXxJFQZlkajF5s_z-M/edit?usp=sharing

[yusufkhanmohammad]

Hello Community Members! Here is the link to my draft for Idea 2, Looking forward for your valuable feedback.

https://docs.google.com/document/d/1NY8oZbMx7j1IOUiJH1V8KYUXLpGDYZz4RjjtGq5dUbU/edit?usp=sharing

[scottveirs]

A participant in a hackathon at Microsoft related to Orcasound mentioned this Python toolbox. Anyone have any experience with it (@valentina-s ?), or a sense of whether it could be a useful tool for this project?

https://tut-arg.github.io/sed_eval/index.html

[sainimohit23]

Hi @scottveirs , I read the documentation of SED EVAL toolkit. This toolkit provides various metrics to evaluate the performance of models on the task of sound event detection. And yes it can be used to evaluate the performance of the models for this project.

[atreyamaj]

Hello Mentors! Here is the link to my draft for Project 2. https://docs.google.com/document/d/1OKgHNGlshNkoRhs6argPi2rYPseX7vc_owarpCK2WQI/edit?usp=sharing It would be great if you could provide me with some feedback in these final few hours until submission. Thank you for your time. Atreya Majumdar.