[Friend V2] Connect Friend v2's onboard memory to the Omi app ($2,000)

[kodjima33]

Is your feature request related to a problem? Please describe. Recently we've added onboard memory support for Friend dev kit 2 but it is not integrated with the Omi app

Describe the solution you'd like You need to integrate onboard memory support with Omi app. You need to do that by creating a separate listener in the mobile app that will listen for when the device reconnects and checks and if there is any audio in onboard storage. If yes, the listener needs to:

• receive the audio file
• process it with a separate backend endpoint (you need to create the endpoint)
• cut the audio file in multiple memories, if there are long waits
• save those memories in the backend (processing each memory independently)
• Ideally this endpoint streams to the backend, instead of waiting for the opus encoded data to arrive, as it can take a few hours, so this functionality will be mostly made by users while sleeping, letting their Friend sync.

Additional context This is a paid task. Reward is $2000 in cash. Simply link your PR with this task and we will check it asap. Check how Contributions and bounties work here

• If no one is assigned to this task, then no one is working on it.
• We don't lock bounty tasks on you unless you've done PRs before. But you can solve that by doing this PR first
• if you need the v2 device, build it yourself or come to our San Francisco office and we will give you one. DM @kodjima33 in telegram for details. If you can't do neither, buy device here and dm on the same telegram

[kevvz]

I will take this. I built the memory so I think I know how to integrate this best

[kodjima33]

@kevvz pls try but since you told me multiple times that you don't feel comfortable with flutter, if anyone else wants to try, they can still try to integrate it as well (We need to start shipping v2 asap)

[kevvz]

Yes, anyone else who wants to take on this task, feel free to do so!

[eltoob]

Question about the SD Card @kodjima33, don't you think that BLE bandwidth is going to be a problem. BLE has a max speed of 1Mbps.

[kodjima33]

@eltoob no will not be a problem because we don't have another choice

[josancamon19]

• [x] Once I open the app I want to get an alert in the top of the app, that is dismissable, but says, you have "X hours" of recording available." I also want this static alert below speech language in settings.

• [x] That should open a new page

  • [x] This page should contain a single buton that says start importing memories
  • [x] It should have a progress bar and an estimate left,
  • [x] Somewhere here it should explain, that the rest of the app "will not work" realtime while they import this is happening.
  • [x] Should explain they should have good internet connectivity while it happens.
  • [x] The device storage bytes -> should be streamed to the backend, and the backend should stream the storage into Google cloud temporarilly (there's some existing code for this, reuse), there should be a timer on the backend, or a background tasks that if the "session" or "file id" stopped being streamed for longer than 10 seconds, it should be processed then
  • [x] So if the connection cuts in the middle, it should still process whatever was transmitted, and the device shouldn't have those bytes anymore.
  • [x] When processing on the backend
    • [x] Use VAD endpoint to find cut's of audio in the conversation, separate as single files, every time the splits without voice are longer than 2 minutes
    • [x] Process each file independetly as a single memory, and use fal_whisperx for the stt
    • [x] Save a value in the memory object, that tell us that that memory was created from storage caching.

• [x] Once the file finishes streaming, it should let the user know that the file was received, and the processing will take a few minutes, check their notifications are on.

• [ ] Once the processing of all memories are received, it should send a notification to the user letting them know that the memories where processed ~ The app should refresh this case the memories list, as if the app is not reopened the memories will not be seen.

[josancamon19]

I might be missing a couple UX possible issues, or edge cases.

[beastoin]

should we...?