balena-labs-projects / balena-cam

Network Camera with Raspberry Pi and WebRTC. Tutorial:
https://balena.io/blog/build-a-raspberry-pi-based-network-camera/
MIT License
175 stars 77 forks source link

Do sound as well #45

Open imrehg opened 4 years ago

imrehg commented 4 years ago

When a microphone is present.

See: https://forums.balena.io/t/balenaos-startup/37794/39

ghost commented 4 years ago

I figured I'd take a look at this as I'll be doing a fair amount of camera work over the next couple of months. Right off the bat there'll need to be some reconfig as the current config only pulls back a single track from the MediaStream, the video one.

I'll be using a Logitech C920 for testing.

I'll be happy to create a GitHub public repo and commit regularly for whomever is interested. :)

mbalamat commented 4 years ago

@webtop we are open for PRs, we would love your contribution to this project.

hinzendo27 commented 4 years ago

April 23, 2020

Hello:

Is anyone still looking at this thread? I'm attempting to make Balena-Cam operate under a better framerate. The existing one is very very slow. ??

Brian Wayne

ghost commented 4 years ago

I'm still poking at it. I have the refresh rate much better, but ran into an issue that was a bug in Python 3.5, which is what the project Dockerfile pulls from. The issue has been resolve now, but the Python version needs increasing.

First of all, for audio, you need to get rid of using OpenCV, as that is for video only. I'm using PyAV (already used partially), but that is how the bug was found.

@mbalamat Is there a version of fincm3 image that uses a newer version of Python?

I'll post my changes so far to my GitHub repo, for anyone interested. Audio is almost there now.

ghost commented 4 years ago

@hinzendo27 If you'd like to collaborate and list/fix changes together, let me know. That way we aren't both doing the same work. :)

hinzendo27 commented 4 years ago

Apr 23, 2020

Hello:

The issue with regard to audio, is not what I'm currently working. I'm not that up so speed on dockers and all of that. When I "run" the camera RasPi, the framerate is very very slow. There's no big monster rush right now. If one is drawing on a graph board, or something, one would like to use gestures to help better communicate a point. Currently, the framerate is too slow for that, but as I was saying it's not a big monster rush.

If anyone has time, whenever they have the time, any new info would help.

Brian Wayne

On Thu, Apr 23, 2020 at 8:17 AM Paul Allsopp notifications@github.com wrote:

I'm still poking at it. I have the refresh rate much better, but ran into an issue that was a bug in Python 3.5, which is what the project Dockerfile pulls from. The issue has been resolve now, but the Python version needs increasing.

First of all, for audio, you need to get rid of using OpenCV, as that is for video only. I'm using PyAV (already used partially), but that is how the bug was found.

@mbalamat https://github.com/mbalamat Is there a version of fincm3 image that uses a newer version of Python?

I'll post my changes so far to my GitHub repo, for anyone interested. Audio is almost there now.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-618457079, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTAIMHITPQBZ6FFCIA3ROBLXJANCNFSM4JH6GPJA .

hinzendo27 commented 4 years ago

Apr 23, 2020

Paul:

If you need help, let me know. With regard to pulling a "docker" file out of a RasPi, and then parsing through the python files and all that, I'm very much in the dark on all of that. I'm base level, in regard to python, and base level with OpenCV. But, I have no idea how to reconstruct a docker file or any of that.

Any help you would need, I can try to assist somewhat, I'm mostly in the dark here.

Brian Wayne (hinzendo)

On Thu, Apr 23, 2020 at 8:21 AM Paul Allsopp notifications@github.com wrote:

@hinzendo27 https://github.com/hinzendo27 If you'd like to collaborate and list/fix changes together, let me know. That way we aren't both doing the same work. :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-618459800, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTCXKPVTQEILEZPUMJ3ROBMHFANCNFSM4JH6GPJA .

mbalamat commented 4 years ago

Hi guys great to hear that you like the project and want to contribute! @webtop what python version do you need? I can try bumping the python version for you though the only blocker that I had and didn't bump the version in past was the library dependencies. @hinzendo27 awesome idea about the frame rate increase can you explain to me how do you plan on increasing the frame rate?

hinzendo27 commented 4 years ago

Apr 24, 2020

Marios:

Thank you for the email. My recent message was addressing that issue. I'd like to spell out some big long series of steps about improving framerate, but I don't know how to unassemble a Docker Container, or any of the files so associated. The other guy is Paul, I mentioned to him how in the dark I am here. Any info would help.

Brian

On Fri, Apr 24, 2020, 7:51 AM Marios Balamatsias notifications@github.com wrote:

Hi guys great to hear that you like the project and want to contribute! @webtop https://github.com/webtop what python version do you need? I can try bumping the python version for you though the only blocker that I had and didn't bump the version in past was the library dependencies. @hinzendo27 https://github.com/hinzendo27 awesome idea about the frame rate increase can you explain to me how do you plan on increasing the frame rate?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-619058830, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTE6E2PMBZVVTN5WUELROGRQJANCNFSM4JH6GPJA .

pallsopp commented 4 years ago

@mbalamat Anything >= 3.5.8 should be good as that's when the buf got fixed.

mbalamat commented 4 years ago

@hinzendo27 to start tinkering with the code, you can try run balena-cam locally on your computer. Just install python 3. In our dockerfile there are the dependencies you need. See the pip install part. @pallsopp whenever I get some time I'll try myself to use another version. For now can you try changing the base image from what we currently have to something from here https://hub.docker.com/r/balenalib/raspberrypi3-python and regarding the tags something from here https://hub.docker.com/r/balenalib/raspberrypi3-python/tags?page=1&name=3.7?

hinzendo27 commented 4 years ago

Apr 24, 2020

Paul:

Thank you for the help. I'll try to contribute, you guys would be forced to bring me out of the stone age on that one. Not sure people would want to take the time.

Brian

On Thu, Apr 23, 2020, 8:17 AM Paul Allsopp notifications@github.com wrote:

I'm still poking at it. I have the refresh rate much better, but ran into an issue that was a bug in Python 3.5, which is what the project Dockerfile pulls from. The issue has been resolve now, but the Python version needs increasing.

First of all, for audio, you need to get rid of using OpenCV, as that is for video only. I'm using PyAV (already used partially), but that is how the bug was found.

@mbalamat https://github.com/mbalamat Is there a version of fincm3 image that uses a newer version of Python?

I'll post my changes so far to my GitHub repo, for anyone interested. Audio is almost there now.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-618457079, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTAIMHITPQBZ6FFCIA3ROBLXJANCNFSM4JH6GPJA .

mbalamat commented 4 years ago

@hinzendo27 sure no problem! Just post your question here or better in our forums here > https://forums.balena.io/ . I think in our forums the communication would be better. Because here there were a couple of times I lost the notifications. cc @pallsopp

pallsopp commented 4 years ago

More than happy to help out with any explanations and questions. Don't hold back, and definately use the forums because the community there is great. And yes, cc me and I'll be sure to get a notification.

hinzendo27 commented 4 years ago

Apr 24, 2020

Marios:

There is like a 90 second delay for the video (one has to wait that long). I don't know if there's an IDE for these Docker Files or whatever ( I know Python has associated IDEs -- Thonny on RasPi being one of them). I hope these videos maintain something of a private nature.

Brian

On Fri, Apr 24, 2020, 9:13 AM Marios Balamatsias notifications@github.com wrote:

@hinzendo27 https://github.com/hinzendo27 to start tinkering with the code, you can try run balena-cam locally on your computer. Just install python 3. In our dockerfile there are the dependencies you need. See the pip install part. @pallsopp https://github.com/pallsopp whenever I get some time I'll try myself to use another version. For now can you try changing the base image from what we currently have to something from here https://hub.docker.com/r/balenalib/raspberrypi3-python and regarding the tags something from here https://hub.docker.com/r/balenalib/raspberrypi3-python/tags?page=1&name=3.7 ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-619106177, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTENR3VXNLCC3ND6AMLROG3A5ANCNFSM4JH6GPJA .

pallsopp commented 4 years ago

The technology behind all of this is called WebRTC. You can check it out on webrtc.org The WebRTC APIs are JavaScript based, with bindings for Python. That delay at the start is partly due to the technology handshaking between the front-end and back-end. That's where the security comes into it. There are some great videos on YouTube also concerning WebRTC, that are definitely worth a look.

hinzendo27 commented 4 years ago

Apr 24, 2020

Paul:

The directory structure of a provisioning system on one's hard drive is a bit of a head scratcher. It's confusing to make out what's being provisioned to the RasPi, and how. I'm updating a resume too, so I have to steer away from this for a time. I'll get back in a few days.

Brian

On Fri, Apr 24, 2020, 9:29 AM pallsopp notifications@github.com wrote:

The technology behind all of this is called WebRTC. You can check it out on webrtc.org The WebRTC APIs are JavaScript based, with bindings for Python. That delay at the start is partly due to the technology handshaking between the front-end and back-end. That's where the security comes into it. There are some great videos on YouTube also concerning WebRTC, that are definitely worth a look.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-619115579, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTGHTEBFHO54UD7XHRDROG47VANCNFSM4JH6GPJA .

pallsopp commented 4 years ago

No problem.Just to be clear, in case it wasn't obvious, webtop and pallsopp are the same person....me. This is my work account, and webtop is my personal account. During the day I'm usually logged in under my work account, and in the evening, my personal account. Sometimes I forget to switch :)

mbalamat commented 4 years ago

@pallsopp yep I noticed, thanks for clarifying though!

pallsopp commented 4 years ago

@hinzendo27 The basics of a Dockerfile contain everything that's needed to build a system-on-the-fly. The FROM parts pulls a system image that someone has built and contains all the software you need to run a project. That system image contains your system files, folders, and libraries: almost a complete linux system, but just enough to make everything work. The RUN parts are your command as if you were logged into the system: for example apt-get install commands to install the libraries you need for the project to work. Once everything is in place, you COPY your code folder to a path on the system, and set that as your working directory.

This really isn't the place for a detailed explanation, but I thought I'd fill you in on the most important parts: FROM an image you get your system RUN your command to tailor the system to your needs COPY your code to a place you can run it on the system

Hope that helps.

hinzendo27 commented 4 years ago

Apr 25, 2020

pallsop:

Thanks for taking the time to plow through some of the detail. We're all busy.

If one "populates" a windows pc with 1) a DOS "lobe" needed to force the balena cli into local operation, and 2) a "lobe" which contains the balena-cam provisioning tree, that all makes sense.

Perhaps I need to approach the general balena forum to "disect" what's going on in that entire balena-cam "lobe". Only then can I start to upgrade potential framerate issues.

Thanks for taking the time.

henzendo

On Fri, Apr 24, 2020, 11:07 PM pallsopp notifications@github.com wrote:

@hinzendo27 https://github.com/hinzendo27 The basics of a Dockerfile contain everything that's needed to build a system-on-the-fly. The FROM parts pulls a system image that someone has built and contains all the software you need to run a project. That system image contains your system files, folders, and libraries: almost a complete linux system, but just enough to make everything work. The RUN parts are your command as if you were logged into the system: for example apt-get install commands to install the libraries you need for the project to work. Once everything is in place, you COPY your code folder to a path on the system, and set that as your working directory.

This really isn't the place for a detailed explanation, but I thought I'd fill you in on the most important parts: FROM an image you get your system RUN your command to tailor the system to your needs COPY your code to a place you can run it on the system

Hope that helps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-619328009, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTAJBEBVUMKKBN7ILHDROJ427ANCNFSM4JH6GPJA .

hinzendo27 commented 4 years ago

April 27, 2020

pallsop:

Do you know of a way to ssh into a balena Raspi from a raspbian Raspi? I'm using a wifi hotspot. I'm not sure how to ssh over wireless, especially if it's with someone's hotspot. When I'm using ethernet, ssh is very easy, but the wireless seems to throw a wrench into the works. I need to be able to quickly post photos to a subordinate type directory. I'm having trouble performing ssh over wifi. Maybe you're too busy, I just thought I'd ask. Posting photos by a provision, is a non-issue -- very easy. SSH over wireless, not so much.

hinzendo

On Sat, Apr 25, 2020, 7:42 AM Brian Wayne wayne1br@gmail.com wrote:

Apr 25, 2020

pallsop:

Thanks for taking the time to plow through some of the detail. We're all busy.

If one "populates" a windows pc with 1) a DOS "lobe" needed to force the balena cli into local operation, and 2) a "lobe" which contains the balena-cam provisioning tree, that all makes sense.

Perhaps I need to approach the general balena forum to "disect" what's going on in that entire balena-cam "lobe". Only then can I start to upgrade potential framerate issues.

Thanks for taking the time.

henzendo

On Fri, Apr 24, 2020, 11:07 PM pallsopp notifications@github.com wrote:

@hinzendo27 https://github.com/hinzendo27 The basics of a Dockerfile contain everything that's needed to build a system-on-the-fly. The FROM parts pulls a system image that someone has built and contains all the software you need to run a project. That system image contains your system files, folders, and libraries: almost a complete linux system, but just enough to make everything work. The RUN parts are your command as if you were logged into the system: for example apt-get install commands to install the libraries you need for the project to work. Once everything is in place, you COPY your code folder to a path on the system, and set that as your working directory.

This really isn't the place for a detailed explanation, but I thought I'd fill you in on the most important parts: FROM an image you get your system RUN your command to tailor the system to your needs COPY your code to a place you can run it on the system

Hope that helps.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-619328009, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTAJBEBVUMKKBN7ILHDROJ427ANCNFSM4JH6GPJA .

pallsopp commented 4 years ago

If your raspis are running containers that are on the cloud, you should be able to use balena ssh. If you are using ssh from the commandline, you might need to change the port to 22222, which is the ssh port to a local instance. Give that a try, or let me know more info about how the containers are.

Thx

pallsopp commented 4 years ago

FYI: I work from home and work online, so I'm always available :)

hinzendo27 commented 4 years ago

Apr 27, 2020

pallsop:

Right now, funds are a little tight. It may be asking quite a bit, but I'm trying to use the Raspbian system on one of the pis, and ssh into the balena OS on another. There are like 4 raspberry pis. Using the hotspot, I'm trying to make it work. I don't have a huge background with firewalls or any such thing.

Using ssh across multiple platforms would help, but I'm not sure "sshing" across them would fly (especially with a WiFi hotspot in the middle). If I put a photo onto the Raspbian Pi, then ssh into the balena pi, I could transfer photos. Just trying to save money. I don't know how to transfer files from like a usb drive or whatever into the balena pi. ?? I don't have all of my networking stuff right now, I stored it in a friend's for a while.

The Picture Server I'm trying to set up, is a Pi Zero W, so, an ethernet connection will be a problem.

This is kind of a head scratcher. Maybe I'm making it more complicated than it has to be.

hinzendo

On Mon, Apr 27, 2020 at 12:07 PM pallsopp notifications@github.com wrote:

FYI: I work from home and work online, so I'm always available :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-620175455, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTD7QN4GHWEUE3CWL53ROXJWHANCNFSM4JH6GPJA .

pallsopp commented 4 years ago

Why don't we take this to the forums, and I'll be able to help you some more as this is getting off-topic for its current location.

Thx

hinzendo27 commented 4 years ago

Apr 28, 2020

pallsop;

OK, the forums can probably help. Thank you.

hinzendo

On Tue, Apr 28, 2020, 3:10 PM pallsopp notifications@github.com wrote:

Why don't we take this to the forums, and I'll be able to help you some more as this is getting off-topic for its current location.

Thx

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-620881563, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTFBOEDKE7AYPOB6JETRO5H3TANCNFSM4JH6GPJA .

hinzendo27 commented 4 years ago

April 30, 2020

Hello:

Has anyone seen this note? I'm back working on the video system. The push isn't working like it had been. The OS says: "not deploying release". Anyone have any idea why this is the case?

hinzendo

On Tue, Apr 28, 2020 at 3:13 PM Brian Wayne wayne1br@gmail.com wrote:

Apr 28, 2020

pallsop;

OK, the forums can probably help. Thank you.

hinzendo

On Tue, Apr 28, 2020, 3:10 PM pallsopp notifications@github.com wrote:

Why don't we take this to the forums, and I'll be able to help you some more as this is getting off-topic for its current location.

Thx

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-620881563, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTFBOEDKE7AYPOB6JETRO5H3TANCNFSM4JH6GPJA .

pallsopp commented 4 years ago

What does your push command look like?

hinzendo27 commented 4 years ago

April 30, 2020

Hello:

Immediately following the push command all of the provisioning commands appear on the windows screen (seems to take like 4 to 9 minutes, or so). Then at the very end of the provision sequence there is a statement about "HTTP code 404", and the whole process dumps -- "not deploying release". In all of the "sea of red ink" that shows up at that point, it says "remote build failed. It says something about using "export DEBUG" -- don't know what they're driving at.

hinzendo

On Thu, Apr 30, 2020, 2:40 PM pallsopp notifications@github.com wrote:

What does your push command look like?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-622129476, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTGJ5J6DNNRYDPRQ7F3RPHV4RANCNFSM4JH6GPJA .

hinzendo27 commented 4 years ago

April 30, 2020

Hello:

The video sever (Balena-Cam) was not working earlier, but it is working now.

hinzendo

On Thu, Apr 30, 2020 at 3:32 PM Brian Wayne wayne1br@gmail.com wrote:

April 30, 2020

Hello:

Immediately following the push command all of the provisioning commands appear on the windows screen (seems to take like 4 to 9 minutes, or so). Then at the very end of the provision sequence there is a statement about "HTTP code 404", and the whole process dumps -- "not deploying release". In all of the "sea of red ink" that shows up at that point, it says "remote build failed. It says something about using "export DEBUG" -- don't know what they're driving at.

hinzendo

On Thu, Apr 30, 2020, 2:40 PM pallsopp notifications@github.com wrote:

What does your push command look like?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-622129476, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTGJ5J6DNNRYDPRQ7F3RPHV4RANCNFSM4JH6GPJA .

pallsopp commented 4 years ago

Using export debug allows for a verbose output, so you get a lot more detail.

Getting a 404 means that a web request failed, specifically that an endpoint for one of the requests could not be found.

From the command line, type DEBUG=1 then hit enter and type your push command again. You should then get a verbose output, which you can post and I’ll take a look at it

It ain't easy being green! Please recycle.

On Apr 30, 2020, at 3:33 PM, hinzendo27 notifications@github.com wrote:

 April 30, 2020

Hello:

Immediately following the push command all of the provisioning commands appear on the windows screen (seems to take like 4 to 9 minutes, or so). Then at the very end of the provision sequence there is a statement about "HTTP code 404", and the whole process dumps -- "not deploying release". In all of the "sea of red ink" that shows up at that point, it says "remote build failed. It says something about using "export DEBUG" -- don't know what they're driving at.

hinzendo

On Thu, Apr 30, 2020, 2:40 PM pallsopp notifications@github.com wrote:

What does your push command look like?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/balenalabs/balena-cam/issues/45#issuecomment-622129476, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANRDVTGJ5J6DNNRYDPRQ7F3RPHV4RANCNFSM4JH6GPJA .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.