Audio lags a bit behind of the video

I tried using videomail.co concept is pretty kewl, however audio is lagging behind video, I tried the audio beta function. I used chrome browser with my windows 10 desktop pc. Somehow video and audio needs to be stitched together. Also i tried while recording video playing music on a background of my computer using system sound and not microphone, but that didn't work and resumed it. Often i would like to use music on background while talking on the microphone and record the video.

Also it would be nice to add a screen sharing capability in the mix and being able to record it. Also probably Captcha would not hurt to be added as well for extra security.

That's my take on it. I sent couple of videos to my friend concept for video mail is great, however i just don't like relying on chrome in regards to getting auido and video, is there another way using native app to get audio/video on windows 10 pc? I want to only use Surface Pro and nothing else. Not sure what exactly would be better using WebRTC or simply web sockets if you want to get recording.

Note #1 complain from users is not being able to share files... etc... recorded files.. currently as it stands right now i don't see a slick way of combining both audio and video into a file without seeing some lag. If you know of any methods of combining mic audio + video do let me know.

Websockets are faster but... i feel like there is just has to be away to get rid of audio + video lagging behind. We need more contributors to this. Web RTC is too slow with development and using their getmedia method that is constantly changing is not viable for businesses. I either need API that i can tap into to record voice + video and be able to share it with uers or something that would make it easy for me to write business apps.

Every time chrome comes out with new version... it's like someone pulling a rug under me.... :) while i am standing on the solid ground, i do not want to depend on Browser ever and just want to use WebSockets or something and native Windows Apps. If anyone knows way of handling this app better let me know and i will chime in regards to what i think needs to be improved on.

nice, good feedback man! here some comments inbetween

however audio is lagging behind video yup, that's not a surprise and i am working on this. hence it's in beta. and if you haven't noted yet, i am a deaf programmer, so quite challenging to adjust correct ffmpeg parameters on server side to achieve best sync between audio + video when you can't hear yourself. that's why i'd welcome new contributors who could help me with this part. but experience in websockets + ffmpeg both would be a requirement.

recording video playing music on a background of my computer using system sound and not microphone interesting case. haven't tried this yet but i think when you load page, your browser is asking for your audio input source. you normally choose the microphone, right? but in your case can you try to reload browser and pick the system sound card or so as the audio input. can you try that? let me know if that worked.

Also it would be nice to add a screen sharing capability in the mix and being able to record it. Also probably Captcha would not hurt to be added as well for extra security. absolutely, that would be cool - many other colleagues have lots of ideas too, pretty crazy ones i d say. been working hard on videomail.io for almost 4 years on a voluntary basis. if one day i get more help or financial support, i would add all those cool features you all suggest without hesitance.

yeah, the videomail solution with websockets does not rely on webrtc which makes it pretty cool. all plain javascript. your constructive criticism is very welcome. feel free to drop more questions/ideas in new tickets.

in regards to the main issue here, allow me to reveal some technical stuff. for example you can see the central audio recording code here: https://github.com/binarykitchen/videomail-client/blob/master/src/util/audioRecorder.js

and there are some audio-related parameters, especially the buffer size https://github.com/binarykitchen/videomail-client/blob/master/src/options.js#L56

on server side, each pcm audio sample will be assembled to a single wav file:

const wavWriter = new WavFileWriter(options.file, {
            channels:   1,
            sampleRate: [depends on your audio input],
            bitDepth:   16
})

... loop ...

wavWriter.write(pcm)

...

wavWriter.end()

then this goes to ffmpeg and gets encoded together with the images which is quite another complex code (can paste here later).

just got an idea right now: maybe i should record the wav on client side instead of pcm samples? maybe this could help preventing these audio lags... hmmm, will have to think about this.

maybe useful: https://www.npmjs.com/package/wavencoder https://github.com/doomjs/wav-arraybuffer ... and there is more ...

will experiment later when i have time avail.

Sorry haven't realized about your condition. I see what you mean about audio and video syncing and i quickly checked out the wav encoder and wav array buffer. All of that is interesting to compile together.

I know I mentioned about GetMedia method and how i want to stay away from having too many dependencies from the realm of the shifting ground.

However here is underlining thing that i want to do, basically i want to record (mic audio)+video+(audio from the system sound) + (screen of desktop) + (whiteboard canvas)

Very good example of it is OBS that is designed for broadcasting sort of like when you are streaming games. Except OBS is more for broadcasting, it doesn't really do this in real time.

https://www.webrtc-experiment.com/

Take a look at web rtc... i am trying to see fundamental difference between your existing socket code and web rtc.

Note i realize web rtc is... all about.. peer to peer...

So to make the story short, i realize you are in the midst of deciphering complex boundaries of trying to inter combine voice and video can be tough, this is pain i realize it...

However my goal is to be able to record all of this and replay it back... I haven't actually seen example at WEBRTC that can do that, sure it taps into microphone and API etc... and that is where WEBRTC i imagine is used in your case, or at least partially used to tap into microphone and video.

Is my understanding correct? Once you tap into mic+ video then you hand that off to websocket right? Do you know if there is any native C++ libraries to tap into screen recording by any chance?

In regards to initial question... during recording.... when i chose audio it doesn't really allow me to set soundcard sound using chrome.. it just has a popup of using mic... also when recording what would be neat is to ad actual synthesizer bar... that shows when u speak bar... values of voice....

I think major challange here is to record system sound... and mic... when i play both mic and system sound recording doesn't capture both.

In regards to "websockets + ffmpeg both would be a requirement"

This is my first response on github to anything, i can provide advise, but not entirely sure if i can provide exact code, i am trying to learn Python, i have tasked my 17 years old nephew to take that is everything in my head and lay it out in code....

Are you sure there is no better way for mixing both voice and video together without having to use ffmpeg? I think problem with ffmpeg is that... it depends on other factors such as hardware....

Recording at client level can be possibility but passing that data to ... the server... maybe there will be lag... not entirely sure... whats really a best practice here.. i think it's extremely complicated...

In regards to financial support, i would love to help but tight on money. Maybe we can get other contributors to donate, i can come up with business ideas and you can code. We do have one thing in common you have BinaryKitchen in your name and my company's name is binary fusion.

I been trying to take a stab at this issue before of building something compatible...... i have business where i basically help people create whiteboard as one of my business models.... but i also need whiteboard for myself for my software as a service model... Note i am not sure how flexible my service as a software model can be, but i think since i have very narrow type of use case... it maybe feasible........ note model is a bit crazy. I can share some of these ideas if u like, but ideas are only ideas without execution of these ideas. Taking 4 years to do something like is a very long time...

i am afraid if we don't get contribution another 4 years pass by without going further..... so guys who is watching this post donate please if you can. Also let me know how it goes with mixing sound and voice... note i would like to have videos that can go for 1 hour or 1.5 hour.. one on one... type of interaction real time.... if it goes beyond 2 mins and starts lagging this is not good...

i have particular case with web sockets, Web sockets are tough, but my major question is can we use somehow functions of WEB RTC to tap into voice video, microphone, system sound.... and whiteboard canvas and be able to record it all at particular key frame, and sequence?

I realize there are many apps out there that do many things but i just feel like they are completely disconnected... or not sophisticated enough for a solid business use case...

in case if u want to fire up videomail on my server and maybe we can try to make it better let me know, we can place it on my server in the basement, and try to inter combine many of the functions.. at least initially....

wow, long post. okay, let me comment briefly between your lines and go from there:

On 11/08/16 06:20, romanwell wrote:

Sorry haven't realized about your condition.

ha, don't be. pretty proud here and it was the motivation to build videomail. because current technologies do not embrace sign language very well.

I see what you mean about audio and video syncing and i quickly checked out the wav encoder and wav array buffer. All of that is interesting to compile together.

I know I mentioned about GetMedia method and how i want to stay away from having too many dependencies from the realm of the shifting ground.

However here is underlining thing that i want to do, basically i want to record (mic audio)+video+(audio from the system sound) + (screen of desktop) + (whiteboard canvas)

thats definitely possible but lots of work

Very good example of it is OBS that is designed for broadcasting sort of like when you are streaming games. Except OBS is more for broadcasting, it doesn't really do this in real time.

https://www.webrtc-experiment.com/

Take a look at web rtc... i am trying to see fundamental difference between your existing socket code and web rtc.

Note i realize web rtc is... all about.. peer to peer...

exactly. you have to ask yourself, are you after peer to peer or client to server? both are very different use cases! for the latter one, forget web rtc. that's why videomail.io isn't using web rtc at all.

So to make the story short, i realize you are in the midst of deciphering complex boundaries of trying to inter combine voice and video can be tough, this is pain i realize it...

i am not really in the middle. the video part works very well and already > 760 videomails are sent in average every week by 90% deaf people for their conversations based on sign language. that said, the video part is very stable. i am sure, with an hearing programmer the audio part can be stabilized as well.

However my goal is to be able to record all of this and replay it back... I haven't actually seen example at WEBRTC that can do that, sure it taps into microphone and API etc... and that is where WEBRTC i imagine is used in your case, or at least partially used to tap into microphone and video.

Is my understanding correct? Once you tap into mic+ video then you hand that off to websocket right? Do you know if there is any native C++ libraries to tap into screen recording by any chance?

explored c++ solutions a while ago but dropped. prefer to stick to javascript for best compatibility.

In regards to initial question... during recording.... when i chose audio it doesn't really allow me to set soundcard sound using chrome.. it just has a popup of using mic... also when recording what would be neat is to ad actual synthesizer bar... that shows when u speak bar... values of voice....

I think major challange here is to record system sound... and mic... when i play both mic and system sound recording doesn't capture both.

must be possible somehow. never tried it before. maybe also depends on your machine settings, to make your system sound recording a public interface for browsers to access to. no idea.

In regards to "websockets + ffmpeg both would be a requirement"

This is my first response on github to anything, i can provide advise, but not entirely sure if i can provide exact code, i am trying to learn Python, i have tasked my 17 years old nephew to take that is everything in my head and lay it out in code....

Are you sure there is no better way for mixing both voice and video together without having to use ffmpeg? I think problem with ffmpeg is that... it depends on other factors such as hardware....

can't say much. for the videomail scenario ffmpeg seems to be the best solution.

Recording at client level can be possibility but passing that data to ... the server... maybe there will be lag... not entirely sure... whats really a best practice here.. i think it's extremely complicated...

In regards to financial support, i would love to help but tight on money. Maybe we can get other contributors to donate, i can come up with business ideas and you can code. We do have one thing in common you have BinaryKitchen in your name and my company's name is binary fusion.

nice suggestion but after 4 years of mad, voluntary development, i dont want to do another big sacrifice and not getting anything in return. with someone else i am currently developing a wordpress plugin for videomail and hope to get something in return.

if you want me to build something with you, i am afraid and will charge for my hours unless it is videomail-client related per se. videomail-client will remain open source and i ll always fix bugs and add features for free whenever my time allows it. but new projects, no thanks.

I been trying to take a stab at this issue before of building something compatible...... i have business where i basically help people create whiteboard as one of my business models.... but i also need whiteboard for myself for my software as a service model... Note i am not sure how flexible my service as a software model can be, but i think since i have very narrow type of use case... it maybe feasible........ note model is a bit crazy. I can share some of these ideas if u like, but ideas are only ideas without execution of these ideas. Taking 4 years to do something like is a very long time...

i am afraid if we don't get contribution another 4 years pass by without going further..... so guys who is watching this post donate please if you can. Also let me know how it goes with mixing sound and voice... note i would like to have videos that can go for 1 hour or 1.5 hour.. one on one... type of interaction real time.... if it goes beyond 2 mins and starts lagging this is not good...

i have particular case with web sockets, Web sockets are tough, but my major question is can we use somehow functions of WEB RTC to tap into voice video, microphone, system sound.... and whiteboard canvas and be able to record it all at particular key frame, and sequence?

like i said, possible. but lots of work. and no, web sockets aren't tough. the npm package websocket-stream is really elegant and makes it easy.

I realize there are many apps out there that do many things but i just feel like they are completely disconnected... or not sophisticated enough for a solid business use case...

yup, its new land and what makes videomail unique :)

in case if u want to fire up videomail on my server and maybe we can try to make it better let me know, we can place it on my server in the basement, and try to inter combine many of the functions.. at least initially....

well, the videomail server stays where it is at the moment. but with the api the videomail-client offers, anyone is welcome to use it. and i am very open to suggestions for enhancements.

Question have u looked at kurento? It uses a neat trick of tapping into web rtc using hybrid approach of having sessions of voice and video initiated using peer to peer communication using getmedia making the one on one video and voice still high quality... Between 2 people.... While on background sending copy of the session to the media server that records.... Stream using media elements through sink and source interconnects to pipes....

I think this is somewhat more relevant to my case, since I want to be able to maneuver with the recorded data.... I saw video on YouTube about kurento I think it's pretty slick.

Meaning it requires turning on elasticrtc on aws and making possible to augment on top of video and making decision..... Of video movements.....

In another words... U can record.... And based on recorded video output generate alert..... In another words as it stands now with web rtc u can record things on peer to peer level, but it's useless from the perspective of being able to do anything .... Innovative....

For example some companies use kurento to record... Video.... Then process it using media server and then for example determine how fast cars drive and what size these cars are for whatever reasons .... For my case it's not that type of logic... It's a bit different Iogic.

I realize for video mail u will support it and all and open to new features... Well I want to be able to doodle on whiteboard and include that in addition to video recording.... So while I am doodling on whiteboard.... And talking in mic and recording video explaining in my video email my message.... I think that is useful....

Perhaps adding that as a feature can make more people use it... The fact that 700 people already use video email is quiet awesome.... I am just trying to see how our interests can converge in what I am doing and in what u r doing....

It's interesting to see how u said u explored c++ method and came to conclusion that JavaScript is most compatible... I did some research because I wanted to do native app in Windows and using websockets... While being able to tap into recording functionalities and better encoding and processing functionaries.... And not worrying to much about compatibility issues if I force my customers to use only Windows surface pro for example.. I spent about this whole week researching.... Note I also in past week wrote entire book about software defined networking, if interested in reading it let me know I am going to release it on amazon soon in next 2 months....

Anyway back to c++ I did find some company that provide API and SDKs allowing to tap into Windows native world and make interaction with web rtc.... While being able to run things on powerful hardware boxes that can handle connectivity... And being able to mix things together.... However their library is expensive like 3k for something which I am not 100 percent sure if it will fit my needs or not....

Then I started reading and watching about kurento.... And basically it is built on c plus plus and has variety of flexible APIs into application server..... JavaScript or Java etc... Still providing flexibility for you to use JavaScript while providing scalability using elasticrtc... Bunch of start up companies using it... And some even creating their own c plus plus modules for their use cases on kurento.... The lead guy of kurento made a good video describing the concept..... However it requires stun/turn server... Which is basically media server... It also uses web sockets and providing hybrid approach.... While still leveraging high quality of web rtc peer to peer stream....

I am trying to compare this to videomail app here.... Is it I guess sort of similar like kurento in a way? Except u r not really providing 1 on 1... Life video stream but rather tapping into camera using getmedia then simply forwarding that getmedia into videomail server? I have to loom at your code to see exactly how u do it... But it's somewhat very similar to kurento concept... Except kurento is really for real time....

I am trying to cook up something ... Also I am planning to do some marketing on solutions that I provide for customers... Maybe this is where i can use your help once I get a client.... Who needs particular use case in which case i would be able to pay you....I am an it solution provider... I specialize in Ebook publishing and writing books people read these books and contact me or find me through web.... However they all have variety of cases... Not the same... But of the concepts are the same... Majority of cases are around Internet of things....

FYI the book that I am writing literally talks about deciphering entire concept of SDN... Where I am literally creating from scratch networks at software level and then translating difficult concepts into.... Simpler things... In the form of APIs etc...

The video mail app can be handy... If it supports processing imagine being able to add augmented graphic objects in video email.... Especially for holidays.... Or birthdays..... So instead of receiving just a dull video mail... People can for example add a hat on top of a pirate or something like that... Making it more interesting.....

So video mail is fun to watch....

Also I am not sure if u experience hallow lens or some sort of virtual reality type of projects.... I am trying to do this whole whiteboard and voice and video etc... But I am wondering if there is any technology that allows u to interact with people in virtual world... Face to face as they were sitting there.... I am not sure if oculus does something like that... I have not explored that concept in detail but my goal is to simply process of learning... For 2 people... One on one...

Existing methods are yes whiteboard video etc... However if I can have 2 people sit side by side as if it was in real life... And learn this way from each other where one person literally draws things on whiteboard and as u r wearing glasses it looks like person is sitting next to u and.... U also see whiteboard... Using some augmented reality....sort of like hollow lens from Microsoft....

Anyway these r just some thoughts... Some thoughts r a bit crazy but...then again the ones that r close to reality is probably taking that video mail and adding function of audio recording from system sound and adding possibility to add augmented objects on top especially for holiday emails... People will use it to send email to each other and it can get popular too quicker... :)

Sent from my iPad

On Aug 10, 2016, at 11:23 PM, Michael Heuberger notifications@github.com wrote:

wow, long post. okay, let me comment briefly between your lines and go from there:

On 11/08/16 06:20, romanwell wrote:

Sorry haven't realized about your condition.

ha, don't be. pretty proud here and it was the motivation to build videomail. because current technologies do not embrace sign language very well.

I see what you mean about audio and video syncing and i quickly checked out the wav encoder and wav array buffer. All of that is interesting to compile together.

I know I mentioned about GetMedia method and how i want to stay away from having too many dependencies from the realm of the shifting ground.

However here is underlining thing that i want to do, basically i want to record (mic audio)+video+(audio from the system sound) + (screen of desktop) + (whiteboard canvas)

thats definitely possible but lots of work

Very good example of it is OBS that is designed for broadcasting sort of like when you are streaming games. Except OBS is more for broadcasting, it doesn't really do this in real time.

https://www.webrtc-experiment.com/

Take a look at web rtc... i am trying to see fundamental difference between your existing socket code and web rtc.

Note i realize web rtc is... all about.. peer to peer...

exactly. you have to ask yourself, are you after peer to peer or client to server? both are very different use cases! for the latter one, forget web rtc. that's why videomail.io isn't using web rtc at all.

So to make the story short, i realize you are in the midst of deciphering complex boundaries of trying to inter combine voice and video can be tough, this is pain i realize it...

i am not really in the middle. the video part works very well and already > 760 videomails are sent in average every week by 90% deaf people for their conversations based on sign language. that said, the video part is very stable. i am sure, with an hearing programmer the audio part can be stabilized as well.

However my goal is to be able to record all of this and replay it back... I haven't actually seen example at WEBRTC that can do that, sure it taps into microphone and API etc... and that is where WEBRTC i imagine is used in your case, or at least partially used to tap into microphone and video.

Is my understanding correct? Once you tap into mic+ video then you hand that off to websocket right? Do you know if there is any native C++ libraries to tap into screen recording by any chance?

explored c++ solutions a while ago but dropped. prefer to stick to javascript for best compatibility.

In regards to initial question... during recording.... when i chose audio it doesn't really allow me to set soundcard sound using chrome.. it just has a popup of using mic... also when recording what would be neat is to ad actual synthesizer bar... that shows when u speak bar... values of voice....

I think major challange here is to record system sound... and mic... when i play both mic and system sound recording doesn't capture both.

must be possible somehow. never tried it before. maybe also depends on your machine settings, to make your system sound recording a public interface for browsers to access to. no idea.

In regards to "websockets + ffmpeg both would be a requirement"

This is my first response on github to anything, i can provide advise, but not entirely sure if i can provide exact code, i am trying to learn Python, i have tasked my 17 years old nephew to take that is everything in my head and lay it out in code....

Are you sure there is no better way for mixing both voice and video together without having to use ffmpeg? I think problem with ffmpeg is that... it depends on other factors such as hardware....

can't say much. for the videomail scenario ffmpeg seems to be the best solution.

Recording at client level can be possibility but passing that data to ... the server... maybe there will be lag... not entirely sure... whats really a best practice here.. i think it's extremely complicated...

In regards to financial support, i would love to help but tight on money. Maybe we can get other contributors to donate, i can come up with business ideas and you can code. We do have one thing in common you have BinaryKitchen in your name and my company's name is binary fusion.

nice suggestion but after 4 years of mad, voluntary development, i dont want to do another big sacrifice and not getting anything in return. with someone else i am currently developing a wordpress plugin for videomail and hope to get something in return.

if you want me to build something with you, i am afraid and will charge for my hours unless it is videomail-client related per se. videomail-client will remain open source and i ll always fix bugs and add features for free whenever my time allows it. but new projects, no thanks.

I been trying to take a stab at this issue before of building something compatible...... i have business where i basically help people create whiteboard as one of my business models.... but i also need whiteboard for myself for my software as a service model... Note i am not sure how flexible my service as a software model can be, but i think since i have very narrow type of use case... it maybe feasible........ note model is a bit crazy. I can share some of these ideas if u like, but ideas are only ideas without execution of these ideas. Taking 4 years to do something like is a very long time...

i am afraid if we don't get contribution another 4 years pass by without going further..... so guys who is watching this post donate please if you can. Also let me know how it goes with mixing sound and voice... note i would like to have videos that can go for 1 hour or 1.5 hour.. one on one... type of interaction real time.... if it goes beyond 2 mins and starts lagging this is not good...

i have particular case with web sockets, Web sockets are tough, but my major question is can we use somehow functions of WEB RTC to tap into voice video, microphone, system sound.... and whiteboard canvas and be able to record it all at particular key frame, and sequence?

like i said, possible. but lots of work. and no, web sockets aren't tough. the npm package websocket-stream is really elegant and makes it easy.

I realize there are many apps out there that do many things but i just feel like they are completely disconnected... or not sophisticated enough for a solid business use case...

yup, its new land and what makes videomail unique :)

in case if u want to fire up videomail on my server and maybe we can try to make it better let me know, we can place it on my server in the basement, and try to inter combine many of the functions.. at least initially....

well, the videomail server stays where it is at the moment. but with the api the videomail-client offers, anyone is welcome to use it. and i am very open to suggestions for enhancements.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/binarykitchen/videomail-client/issues/89#issuecomment-238956532, or mute the thread https://github.com/notifications/unsubscribe-auth/AAK16U_d5KlC-Ccc1A6T75Ok-5KRNC3sks5qehZvgaJpZM4Jgq2I.

Binary Kitchen Michael Heuberger 1/33 Parrish Road Sandringham Auckland 1025 (New Zealand)

Mobile (text only) ... +64 21 261 89 81 Email ................ michael@binarykitchen.com Website .............. http://www.binarykitchen.com

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

thanks for that - been busy. so yeah, check my comments below, inline

On 11/08/16 17:02, romanwell wrote:

Question have u looked at kurento? It uses a neat trick of tapping into web rtc using hybrid approach of having sessions of voice and video initiated using peer to peer communication using getmedia making the one on one video and voice still high quality... Between 2 people.... While on background sending copy of the session to the media server that records.... Stream using media elements through sink and source interconnects to pipes.... hmmm, korento media server is written in C++ and the client uses a very different approach. respect to them but cant say much. up to you to decide which one to pick.

I realize for video mail u will support it and all and open to new features... Well I want to be able to doodle on whiteboard and include that in addition to video recording.... So while I am doodling on whiteboard.... And talking in mic and recording video explaining in my video email my message.... I think that is useful....

Perhaps adding that as a feature can make more people use it... The fact that 700 people already use video email is quiet awesome.... I am just trying to see how our interests can converge in what I am doing and in what u r doing.... sure can add more features and have the skills/motivation for that. but that's a big one you're asking. in that case can't sacrifice tons of hours of my free time while doing another full time job.

Then I started reading and watching about kurento.... And basically it is built on c plus plus and has variety of flexible APIs into application server..... JavaScript or Java etc... Still providing flexibility for you to use JavaScript while providing scalability using elasticrtc... Bunch of start up companies using it... And some even creating their own c plus plus modules for their use cases on kurento.... The lead guy of kurento made a good video describing the concept..... However it requires stun/turn server... Which is basically media server... It also uses web sockets and providing hybrid approach.... While still leveraging high quality of web rtc peer to peer stream.... good on them. can't say much. but sounds like the set up including server and hardware is a little more complicated. whereas for videomail client/server all you need is git pull ..., npm init and npm run and then it works right away without much hassle.

I am trying to compare this to videomail app here.... Is it I guess sort of similar like kurento in a way? Except u r not really providing 1 on 1... Life video stream but rather tapping into camera using getmedia then simply forwarding that getmedia into videomail server? I have to loom at your code to see exactly how u do it... But it's somewhat very similar to kurento concept... Except kurento is really for real time.... right, i am sending image frames over the super-fast webstream socket to the server who is going to ffmpeg-encode them to a video on the press of the stop button. this has some advantages and few other disadvantages compared to the kurento concept.

would be nice to see a performance comparison between videomail and kurento. maybe that will help you with your decision?

I am trying to cook up something ... Also I am planning to do some marketing on solutions that I provide for customers... Maybe this is where i can use your help once I get a client.... Who needs particular use case in which case i would be able to pay you....I am an it solution provider... I specialize in Ebook publishing and writing books people read these books and contact me or find me through web.... However they all have variety of cases... Not the same... But of the concepts are the same... Majority of cases are around Internet of things....

FYI the book that I am writing literally talks about deciphering entire concept of SDN... Where I am literally creating from scratch networks at software level and then translating difficult concepts into.... Simpler things... In the form of APIs etc...

The video mail app can be handy... If it supports processing imagine being able to add augmented graphic objects in video email.... Especially for holidays.... Or birthdays..... So instead of receiving just a dull video mail... People can for example add a hat on top of a pirate or something like that... Making it more interesting..... yup, i had something like this in mind. possibilities of videomail are unlimited! just lack of time do explore and make sure of all of them.

Anyway these r just some thoughts... Some thoughts r a bit crazy but...then again the ones that r close to reality is probably taking that video mail and adding function of audio recording from system sound and adding possibility to add augmented objects on top especially for holiday emails... People will use it to send email to each other and it can get popular too quicker... :) you are sharing quite lots of thoughts here. sure its good to be crazy heh but i think, since you aren't raising any bugs/issues here, we better continue this convo by email. and since my time is limited, i prefer if you come with concrete project ideas to get started.

Sent from my iPad

On Aug 10, 2016, at 11:23 PM, Michael Heuberger notifications@github.com wrote:

wow, long post. okay, let me comment briefly between your lines and go from there:

On 11/08/16 06:20, romanwell wrote:

Sorry haven't realized about your condition.

ha, don't be. pretty proud here and it was the motivation to build videomail. because current technologies do not embrace sign language very well.

I see what you mean about audio and video syncing and i quickly checked out the wav encoder and wav array buffer. All of that is interesting to compile together.

I know I mentioned about GetMedia method and how i want to stay away from having too many dependencies from the realm of the shifting ground.

However here is underlining thing that i want to do, basically i want to record (mic audio)+video+(audio from the system sound) + (screen of desktop) + (whiteboard canvas)

thats definitely possible but lots of work

Very good example of it is OBS that is designed for broadcasting sort of like when you are streaming games. Except OBS is more for broadcasting, it doesn't really do this in real time.

https://www.webrtc-experiment.com/

Take a look at web rtc... i am trying to see fundamental difference between your existing socket code and web rtc.

Note i realize web rtc is... all about.. peer to peer...

exactly. you have to ask yourself, are you after peer to peer or client to server? both are very different use cases! for the latter one, forget web rtc. that's why videomail.io isn't using web rtc at all.

So to make the story short, i realize you are in the midst of deciphering complex boundaries of trying to inter combine voice and video can be tough, this is pain i realize it...

i am not really in the middle. the video part works very well and already > 760 videomails are sent in average every week by 90% deaf people for their conversations based on sign language. that said, the video part is very stable. i am sure, with an hearing programmer the audio part can be stabilized as well.

However my goal is to be able to record all of this and replay it back... I haven't actually seen example at WEBRTC that can do that, sure it taps into microphone and API etc... and that is where WEBRTC i imagine is used in your case, or at least partially used to tap into microphone and video.

Is my understanding correct? Once you tap into mic+ video then you hand that off to websocket right? Do you know if there is any native C++ libraries to tap into screen recording by any chance?

explored c++ solutions a while ago but dropped. prefer to stick to javascript for best compatibility.

In regards to initial question... during recording.... when i chose audio it doesn't really allow me to set soundcard sound using chrome.. it just has a popup of using mic... also when recording what would be neat is to ad actual synthesizer bar... that shows when u speak bar... values of voice....

I think major challange here is to record system sound... and mic... when i play both mic and system sound recording doesn't capture both.

must be possible somehow. never tried it before. maybe also depends on your machine settings, to make your system sound recording a public interface for browsers to access to. no idea.

In regards to "websockets + ffmpeg both would be a requirement"

This is my first response on github to anything, i can provide advise, but not entirely sure if i can provide exact code, i am trying to learn Python, i have tasked my 17 years old nephew to take that is everything in my head and lay it out in code....

Are you sure there is no better way for mixing both voice and video together without having to use ffmpeg? I think problem with ffmpeg is that... it depends on other factors such as hardware....

can't say much. for the videomail scenario ffmpeg seems to be the best solution.

Recording at client level can be possibility but passing that data to ... the server... maybe there will be lag... not entirely sure... whats really a best practice here.. i think it's extremely complicated...

In regards to financial support, i would love to help but tight on money. Maybe we can get other contributors to donate, i can come up with business ideas and you can code. We do have one thing in common you have BinaryKitchen in your name and my company's name is binary fusion.

nice suggestion but after 4 years of mad, voluntary development, i dont want to do another big sacrifice and not getting anything in return. with someone else i am currently developing a wordpress plugin for videomail and hope to get something in return.

if you want me to build something with you, i am afraid and will charge for my hours unless it is videomail-client related per se. videomail-client will remain open source and i ll always fix bugs and add features for free whenever my time allows it. but new projects, no thanks.

I been trying to take a stab at this issue before of building something compatible...... i have business where i basically help people create whiteboard as one of my business models.... but i also need whiteboard for myself for my software as a service model... Note i am not sure how flexible my service as a software model can be, but i think since i have very narrow type of use case... it maybe feasible........ note model is a bit crazy. I can share some of these ideas if u like, but ideas are only ideas without execution of these ideas. Taking 4 years to do something like is a very long time...

i am afraid if we don't get contribution another 4 years pass by without going further..... so guys who is watching this post donate please if you can. Also let me know how it goes with mixing sound and voice... note i would like to have videos that can go for 1 hour or 1.5 hour.. one on one... type of interaction real time.... if it goes beyond 2 mins and starts lagging this is not good...

i have particular case with web sockets, Web sockets are tough, but my major question is can we use somehow functions of WEB RTC to tap into voice video, microphone, system sound.... and whiteboard canvas and be able to record it all at particular key frame, and sequence?

like i said, possible. but lots of work. and no, web sockets aren't tough. the npm package websocket-stream is really elegant and makes it easy.

I realize there are many apps out there that do many things but i just feel like they are completely disconnected... or not sophisticated enough for a solid business use case...

yup, its new land and what makes videomail unique :)

in case if u want to fire up videomail on my server and maybe we can try to make it better let me know, we can place it on my server in the basement, and try to inter combine many of the functions.. at least initially....

well, the videomail server stays where it is at the moment. but with the api the videomail-client offers, anyone is welcome to use it. and i am very open to suggestions for enhancements.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub

https://github.com/binarykitchen/videomail-client/issues/89#issuecomment-238956532,

or mute the thread

https://github.com/notifications/unsubscribe-auth/AAK16U_d5KlC-Ccc1A6T75Ok-5KRNC3sks5qehZvgaJpZM4Jgq2I.

Binary Kitchen Michael Heuberger 1/33 Parrish Road Sandringham Auckland 1025 (New Zealand)

Mobile (text only) ... +64 21 261 89 81 Email ................ michael@binarykitchen.com Website .............. http://www.binarykitchen.com

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/binarykitchen/videomail-client/issues/89#issuecomment-239074554, or mute the thread https://github.com/notifications/unsubscribe-auth/AAK16Tpx_nsIg2vmyyKnYI9aHEkWjXVPks5qeqz1gaJpZM4Jgq2I.

Binary Kitchen Michael Heuberger 1/33 Parrish Road Sandringham Auckland 1025 (New Zealand)

Mobile (text only) ... +64 21 261 89 81 Email ................ michael@binarykitchen.com Website .............. http://www.binarykitchen.com

the lagging of audio vs video has now been fixed on server side :)

binarykitchen / videomail-client

Audio lags a bit behind of the video #89