Listen to archived audio data

[scottveirs]

When new users visit Orcasound, often they are excited to hear some orca sounds, but when none are heard, they may not stay engaged. This project aims to develop a feature that displays archived audio data -- interesting live-streamed events that have already occurred. Such functionality will also serve the regular Orcasound visitors who have received a notification, but have just missed the live “Orca Concert”!

The workflow will require:

• automatic identification of periods of time when human annotations are dense, or when an automatic detection algorithm has found many potential whale sounds
• extraction of the corresponding stream in an easy to play/share media format
• sharing the library of events on the website.

Extra features could involve options for users to summarize and visualize the data (building on the embedding GSOC 2021 project), or an option to label the streams in a more granular level. This can be an excellent education tool and also can steer some discussions following up the events.

Quick intro to a list of steps or objectives involved:

1. The simplest version of this is to allow passing in a URL parameter to play back audio older the current live stream. Then we could add a button to generate sharable links anchored at a specific time in the stream (like youtube share feature, with timestamp in URL).
2. This could be expanded into some sort of “event” page for a specific listening event (e.g. concert from last month).
3. Finally, adding these events to a list on the live app for users to discover past events.
4. There are some technical challenges in getting the corresponding audio for a specific time & date, because we don’t have a structured inventory of all the audio streams (though we could probably generate a lookup table by running through all the past HLS files).

Expected outcomes: A suite of methods that make it easy for Orcasound community members to access, share, and play back archived audio.

Required skills: Python, basic web development

Bonus skills: Visualization, Cloud computing, Audio processing, UX Design

Possible mentors: Valentina, Scott

Difficulty level: Medium

Project Size: 175 or 350 h

Getting Started

• Get familiar with the Orcasound data
• Access this spreadsheet with human annotations, and create a function to detect start and end time of a dense subset of annotations of a certain type.
• The spreadsheet is a dump from graphql database which stores the realtime annotations. Some example queries from the database can be found here
• Try using orca-hls-utils to extract audio files for specific times.

Points to consider in the proposal:

• Can you come up with a diagram of the pipeline workflow?
• What computing environment do you plan to use?
• How would you design the interface of the archived events webpage?
• How would you evaluate if the design engages the user?

[hardco2020]

@scottveirs @valentina-s Hi, I am interested in this projects and I would like to ask a few questions regarding the project.

automatic identification of periods of time when human annotations are dense, or when an automatic detection algorithm has found many potential whale sounds

Access this spreadsheet with human annotations, and create a function to detect start and end time of a dense subset of annotations of a certain type.

1. I am not sure what the human annotations are , is it regarding the ML related term ?
2. Is the project gonna be involving a lot of ML ? Because I am proficient at Web design and AWS related field, but I am not quite familiar with ML/AI field.
3. I would like to learn if question 2 is true, is there any docs I can read first to get to learn the basic ML requirement for drafting the project proposal ?

[valentina-s]

@hardco2020

1. Human annotations are the ones coming from when Orcasound website users click on the button "I hear something interesting".
2. The use of ML is to represent the sounds more compactly to the user, so it is going to come after doing the first steps of getting the sounds and serving them to the website. I think clustering/dimensionality reduction will help. Also some audio file manipulation might be needed to extract the appropriate sounds.
3. You can look at some of Jose Giraldo's (@wetdog ) blog posts: https://www.orcasound.net/author/josegiraldo/ for visualization examples. It is sort of open how to present this information so feel free to think of your own ideas.

[ArthrowAbstract]

Hello Moderators, I am Devanshu SInha. I am an undergrad student in Computer Science at SRM Institute of Science and Technology India. I would love to work over "Listen to archived audio data" as a part of GSOC 2022 with @scottveirs and @valentina-s. I have experience working with large streams of data as well as development. In a recent project, I worked in with Coral Reef's detection and discovered the various aspects of working on an underwater project. As for the educational part, I am always fascinated when I would really appreciate your help in formulating a proposal for the same. As a first step would you please give a brief as to how I should approach the human annotation file and work on it. Thank You Devanshu Sinha LinkedIn - https://www.linkedin.com/in/arthrow Github: https://www.github.com/ArthrowAbstract

[alphanoobie]

Hi,

I am interested in working 'Listen to archived audio data' project. I love to code in python and I am familiar with Machine Learning concepts. @valentina-s @scottveirs The link on the GSOC page to contributor guidance shows 404 error. Can you guide me towards applying and writing a proposal formally for working on this project.

[devesh7]

Hello, I am highly interested in being a contributor for 'Listen to archived audio data' project. I would love to be a part of it as it is coding for a great cause, so it will be great to recieve guidance on how to write a proposal for this project. @valentina-s @scottveirs Thankyou.

[920DEV]

Hlo Mentors @scottveirs , @valentina-s I am Dev Choudhary, a BCA pursuing underdrag, I want to contribute for orcasound "listen to archived audio project'. And would you tell me where do i write the official poropsal for the same, as the link you have given in the gsoc website shows "404 error". It would be great help thank you

[valentina-s]

@920DEV @devesh7 @ArthrowAbstract @alphanoobie Welcome! If you have not yet, join the slack team and join #gsoc channel after that. The link to the proposal document has been fixed. The annotations are a simple excel file so you all can explore it to get familiar with the format of the data.

[KaoutharBousbaa1]

Hi all, I am very interested in being a contributor for 'Listen to archived audio data' project. I am currently a Data Science student, and I have the enough skills to start the project. I am so so interested about the subject too. I would love so much to work with you @valentina-s @scottveirs LinkedIn - https://www.linkedin.com/in/kaouthar-bousbaa-38b569146/ Kaouthar BOUSBAA.

[KaoutharBousbaa1]

Can you please send me a Slack Invitation to join the slack workplace.

Thank you, @valentina-s @scottveirs LinkedIn - https://www.linkedin.com/in/kaouthar-bousbaa-38b569146/ Kaouthar.

[valentina-s]

@kaoutharBousbaa1 and others, if you have trouble signing in send me your email at vms16 @ uw.edu, and I will send invitation.

[KaoutharBousbaa1]

@valentina-s Hi ! I sent you my email at vms16@uw.edu

Here is my email in case you missed it: Kaouthar.bousbaa@um6p.ma

Thank you ! Best, Kaouthar.

[alphanoobie]

@920DEV @devesh7 @ArthrowAbstract @alphanoobie Welcome! If you have not yet, join the slack team and join #gsoc channel after that. The link to the proposal document has been fixed. The annotations are a simple excel file so you all can explore it to get familiar with the format of the data.

I have sent you an email, mentioning my email ID below in case you missed it. hussaingadi.hg@gmail.com Hoping to get invited to the slack workspace soon. Thank you

[valentina-s]

@920DEV @devesh7 @alphanoobie @ArthrowAbstract It turns out the slack workspace now has reached the limit of its allowed invitations! We will have to wait for @scottveirs to look into that since I do not have all the admin permissions. In the mean time try to first make a slack account and then using this account to log in. And you can use this Github thread to post questions regarding the project.

[valentina-s]

Or see if this link works out.

[Bhavye2003Developer]

I want to Contribute in Listen to archived audio data, I have required skills i.e I know Python, django(Web Development). I have also worked with various python libraries like matplotlib, pandas, seaborn for visualisation. My linkedin profile :- https://www.linkedin.com/in/bhavye-jain-8980ab255 . Can you please tell me where can I start contributing.

[airpods69]

Has there been any progress to look into?