Sound ID

[alexshepard]

What would it take to get sound ID into iNat?

• Grant & the Merlin folks seem to do this using CNN pans or strides over audio spectrograms,
• I think if we want to follow in these footsteps, we'd need to learn how to process audio clips into spectrograms, pan across them, select segments for analysis, and then make suggestions.

To me there are a lot of different approaches to incorporate Sound ID as a general concept into iNat, and some of those approaches have some implications for what we'd need to learn and build, so this would need to get sorted out before we can start investigating solutions, building prototypes, or anything like that.

• Should iNat have some UI to help a user understand that there are multiple identified organisms in the clip, and show them that at time signature a:aa one taxon was observed but at time signature b:bb another was?
• Or should iNat simply merge the identifications made across the whole sound clip and make a single suggestion?
• Is there anything interesting we should do if the user provides both a photo and a sound clip?
• Do we envision this as a live feature, akin to the AR camera live suggestions? Or is this more about making suggestions for an observation. If it's both, like for iNat mobile and the camera, how do we design a user flow that encourages a user to record a single organism instead of following the example of other apps where you just record everything you hear during a hike?
• Or do we imagine this as a tool that you could turn on during a hike, walk the route, get home, and then somehow use an iNat sound ID tool to break that long recording up into segments of audio of individual organisms and turn each of those segments into a distinct observation?
• Basically, how do we imagine incorporating time-based media into the iNat observation model?
• Basically, we need a user story.

[alexshepard]

@abhasinat

[loarie]

very curious on alex's and abhas's product thoughts on how to learn what kind of differences from whats in place for photos would be minimally necessary to make sound ID successful - thanks for fleshing this out alex

[alexshepard]

I agree that we don't want to overthink an MVP concept or bite off a pile of redesign work. My main goal is to make sure than an MVP that we build is something we can learn from.

We can definitely start by getting organized, do some investigation into the technologies, and make a plan to simply integrate something into the observation ID flow with no new UX or UI in order to see how that works. Depending on how well that works, that could become an MVP or it could become the prototype/starting point for the simplest possible UX changes to make something shippable.

[abhasinat]

Sounds good to me! Let me know when I can be helpful in the process for feedback, UX/UI, or otherwise.