esp32 + esp32cam

[eerison]

Hi @nikivanov

thank you for the project, you did a nice project <3

I was wondering if you have any example using esp32 I guess it could reduce cost of the project.

and to make mic+speaker works, did you use i2s protocol or something else?

[nikivanov]

@eerison The software stack runs on python / linux and relies on raspberry pi's hardware video encoding. ESP32 can do none of those things. It could be rewritten to use ESP32 / esp32cam, but the video quality and framerate would be drastically worse. Besides that, ESP32 most likely isnt powerful enough to encode and decode audio either. But to answer your other question - yes! Both microphone and the audio amp are using I2S - one uses the left channel, and the other uses the right one.

[eerison]

Hey @nikivanov

Do you mind to say where exactly you handle

I2s and camera to appear in the browser?

[nikivanov]

I2S functionality resides exclusively on the raspberry pi device, the server. The browser simply gets an audio/video feed from the server. The technology behind it is WebRTC - same technology that's used for audio / video calls through the browser. The browser does not know where AV signals come from.

Here are some code links that set up the audio:

• Here we disable the analog onboard audio of the raspberry pi
• Here we add the googlevoicehat-soundcard overlay which is basically the driver for the microphone and the amp
• Here is some i2s magic that does... something. If you google it, it might not be needed anymore :)
• Here and here we set up the "soft volume" which allows us to control the volume of the amplifier in software via Alsa
• Here is a snippet from the GStreamer pipeline that records audio from the mic - note the "channels=1" parameter. It might be easier to look at the testrecord.sh and testplayback.sh pipelines that are much simpler. The first one will record a file from the mic, the second will play it back.
• Lastly, note that in the audio wiring diagram, the SD pin of the amplifier is connected to GPIO16, which is controlled by the googlevoicehat driver. When it's low, the amp is shut down. When it's high, per this spec, If the voltage on SD is higher than 1.4V then the output is the Left channel. . Also observe that the microphone's LR pin is connected to the 3.3V pin of the raspberry pi, and it says on the microphone itself LR=H (meaning High) - Right. So the amp uses the I2S left channel when doing playback, and the microphone is using the right channel for recording

Hope that helps!

[nikivanov]

As for the camera, things are much simpler there - the server starts this file when the WebRTC session is started. It acquires the hardware-encoded h264 video stream from the Pi, then adapts it to the RTSP (real time streaming protocol) and then pipes it to a local UDP port, where it gets picked up by the Janus WebRTC server.

In general, the pattern is to have Janus handle all WebRTC related stuff. Janus either pushes the audio bytes to a UDP socket, or reads bytes from a UDP socket to play back or record audio / video. The bytes are either picked up by GStreamer in case of playback, or pushed by the GStreamer to those UDP sockets in the case of recording

[nikivanov]

@eerison I'm closing this issue since your questions have been answered. Feel free to reach out if you have any followups.