Project Timeline - First Sprint (2/22 - 3/8)

[GindaChen]

1. (Device) Acquire new material - @aditya2397
   • [x] Runnable old Android phones : Get 4 (abandoned)
   • [x] Audio Framework
   • [x] Packagetized audio
   • [x] Phone play audio feedback — timing issue
2. (Cloud) - @dereklh4
   • [x] Service Provider: AWS — with speech to test
   • [x] Starting Cloud API: storage, data, interaction ...
   • [ ] Otter
3. (Edge) - @GindaChen @joelhaynie
   • [ ] Hardward - WIFI access + mic
   • [ ] Software - OpenEdge, Databox, Green Grass, EdgeX
   • [x] Machine Learning Tool kit - Audio stream
   • [ ] Starting API - Protocol, language

[GindaChen]

Vote: What kind of platform do we want to use

1. Device: Raspberry Pi 3 B+
2. Cloud: AWS
3. Edge
   • [ ] DataBox
   • [ ] OpenEdge
   • [ ] Green Grass
   • [ ] EdgeX

[GindaChen]

Meet 2/27

Adi (Device)

Progress

• [x] Record media on Android
  • [x] Find API (MediaRecorder)
  • [x] Build and deploy to device
• [ ] Media transfer to server (currently, laptop)
  • [ ] Connection
  • [ ] Transfer static audio file
  • [ ] Streaming audio format
  • [ ] Record and stream simultaneously

Notes

• Android API 20 onwards, the recording app must be in the foreground

• Android recording scrapped in favor of browser recording for:

  • better portability
  • no extra hardware required
  • easier to stream data to edge

Concerns

• How do we resync when someone enters / leaves the meeting?
  • Perhaps beeping

Derek (Cloud)

• [ ] Speech to text
  • [x] Find speech to text - DeepSpeech
  • [x] Test on local python package
  • [x] OpenLambda
  • [ ] open-lambda storage
  • [x] AWS
  • [x] working speech (wav) to text
  • [x] run latency experiments
    • seems that compute, not downloading the model, is bottleneck. Can request more (to 3 GB RAM) to get runtime to 4-5 seconds for a 3 second file, same as local. With 1 GB, runs 9-10 seconds.
  • [x ] include model as dependency?
    • aws imposes limit on size, doesn't fit

TODO

• [ ] Big Model - 180MB
  • [ ] Find Optimal Chunk?
  • [ ] Partition of Model?
  • [ ] Include model as dependency?
  • [ ] Include model as Code Base?

Issue

• Lambda download model for 5-10s
• Include the model as dependency

Junda

• [x] Landscaped existing tools for Diarization.
  • [ ] Google AI: RNN, probably not promising to break down.
  • [ ] Learn from the
  • [ ] LIUM:
  • [ ] Speaker2Vec: No open source
  • [ ]

Joel

• [x] Pi configuration: Hit the wall setting up the network
• [x] OpenLambda Arch: Why things fail. Getting deployed on Docker.
• [ ]

[joelhaynie]

Can I post here? Yes I can!

[GindaChen]

Can I post here? Yes I can!

Fantasitic!

[GindaChen]

3/6: Problems and possible solutions

• [ ] Split the record at the place where a word is chopped. Solution: @GindaChen Tell @dereklh4 : which segment is the complete segment you can transcript.

[GindaChen]

3/6 Achievement:

• [x] @aditya2397 : Switch completely to web browser. Works totally fine!
• [x] @GindaChen : Switch to pyAudioAnalysis and works perfect.
• [x] @dereklh4 : Server is up. Can test now!

[GindaChen]

3/8 Demo preperation:

• [ ] @dereklh4 :
• [ ] @GindaChen :
  • [ ] Show speaker diarization example.
  • [ ] Show speaker diarization example in package style.
  • [ ] Show segmentation of record.
• [ ] @aditya2397 :
  • [ ] Record audio in web browser
• [ ] @joelhaynie :