search

augmentmy-world / arStudio

A innovative Web editor for Augmented Reality
https://webarstudio.tripod-digital.co.nz
MIT License
53 stars 34 forks source link

[feature]:use video plane rendering #28

Closed itamayo closed 4 years ago

kalwalt commented 4 years ago

I tested your code, and think that is much better than before! :smile: I will see how to improve it. It seems like is added an alpha channel onto the viewport probably for this reason is offuscated, but i think it is not difficult to solve. i need also to test what happens if we resize the viewport and other with different scenes.

ThorstenBux commented 4 years ago

Hi @itamayo thanks for the PR. Great work. I've followed your conversation with @kalwalt. Great idea to use a video plane as background rendering.

One question:

ThorstenBux commented 4 years ago

Ah and @kalwalt was faster :)

kalwalt commented 4 years ago

@ThorstenBux Have you tried?

ThorstenBux commented 4 years ago

No haven't tried. Early morning here and bad light. I'm about to have breakfast and be on the way out. Totally excited about the idea though. Want to run that tomorrow. But you say it works right?

kalwalt commented 4 years ago

Much better, but cube seems shifted: maybe i'm not using the right scene setting. Opening dev console the cube is moving but is displayed. Much better than before that cube disappear. But as i said i need more testing, for sure is the right path.

ThorstenBux commented 4 years ago

If the cube is shifted that could also be screen-resolution vs. camera-resolution offset of the view port.

kalwalt commented 4 years ago

See this: arStudio-with-dev-console

ThorstenBux commented 4 years ago

The video isn't full screen is the first thing that is obvious. Cube seems to be too big for the marker which might be related. I would start with making the video background fullscreen and then looking at the viewport

ThorstenBux commented 4 years ago

Woot. 😎 the offset now only looks like a project setup thing.

Get Outlook for iOShttps://aka.ms/o0ukef

From: itamayo notifications@github.com Sent: Wednesday, March 18, 2020 9:48:11 AM To: augmentmy-world/arStudio arStudio@noreply.github.com Cc: Thorsten Bux thorsten.bux@outlook.com; Mention mention@noreply.github.com Subject: Re: [augmentmy-world/arStudio] [feature]:use video plane rendering (#28)

@itamayo commented on this pull request.

In editor/js/components/arcontroller.component.jshttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faugmentmy-world%2FarStudio%2Fpull%2F28%23discussion_r393960161&data=02%7C01%7C%7C27c86be993eb438e17a508d7cab4825f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637200748934027316&sdata=sCe3xX3ktM39sJVSQTTzv4H4vAejptR05%2FXioiYySWI%3D&reserved=0:

  • uniform mat4 u_model;\
  • void main() {\
  • v_coord = a_coord;\
  • v_normal = (u_model * vec4(a_normal,1.0)).xyz;\
  • gl_Position = u_mvp * vec4(a_vertex,1.0);\
  • }\
  • ', '\
  • precision highp float;\
  • varying vec3 v_normal;\
  • varying vec2 v_coord;\
  • uniform vec3 u_lightvector;\
  • uniform vec4 u_color;\
  • uniform sampler2D u_texture;\
  • void main() {\
  • vec3 N = normalize(v_normal);\
  • vec4 color = u_color * texture2D( u_texture, v_coord);\

I fixed the problem of the shader ... Tomorrow If I had time I would like to implemented resizing and other bugs ;) [Screenshot from 2020-03-17 21-47-23]https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F3738104%2F76900051-eedd0000-6898-11ea-9237-372783b81dae.png&data=02%7C01%7C%7C27c86be993eb438e17a508d7cab4825f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637200748934037324&sdata=D18Q2fBP09Zb21Xz9Er%2BUbmiPvcGtylISQmq3Lv439w%3D&reserved=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faugmentmy-world%2FarStudio%2Fpull%2F28%23discussion_r393960161&data=02%7C01%7C%7C27c86be993eb438e17a508d7cab4825f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637200748934037324&sdata=ZN%2FETiaH7h9UDzGu0dmE4sj6X3qzNIAmc0RnCmzSmRE%3D&reserved=0, or unsubscribehttps://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PEZ6AGFFTJUUD2D5ADRH7OYXANCNFSM4LNOAMRQ&data=02%7C01%7C%7C27c86be993eb438e17a508d7cab4825f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637200748934047336&sdata=IGAevTxSH0G4wQcf9Er50DWtp6IjUyO9pJpLFGBaUmw%3D&reserved=0.

itamayo commented 4 years ago

Hi again =)

I fix some problems with projection matrix, It was using different projection of the scene =/ Seems that It works fine when windows is resized... Hope it will help ! Screenshot from 2020-03-20 20-50-34

kalwalt commented 4 years ago

@itamayo i will test tomorrow! Thank you :smile:

itamayo commented 4 years ago

Playing with different mesh ... Screenshot from 2020-03-20 23-20-12

ThorstenBux commented 4 years ago

Amazing

Get Outlook for iOShttps://aka.ms/o0ukef

From: itamayo notifications@github.com Sent: Saturday, March 21, 2020 11:23:06 AM To: augmentmy-world/arStudio arStudio@noreply.github.com Cc: Thorsten Bux thorsten.bux@outlook.com; Mention mention@noreply.github.com Subject: Re: [augmentmy-world/arStudio] [feature]:use video plane rendering (#28)

Playing with different mesh ... [Screenshot from 2020-03-20 23-20-12]https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F3738104%2F77210588-aa07d200-6b01-11ea-877b-419a61734188.png&data=02%7C01%7C%7C69308f50316a497d9ec808d7cd1d43ab%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637203397873133676&sdata=Dj48mD%2F%2Bf1NxlvYni%2F01SJt6k4ywwCWBbVpStvr8Bp4%3D&reserved=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faugmentmy-world%2FarStudio%2Fpull%2F28%23issuecomment-601935533&data=02%7C01%7C%7C69308f50316a497d9ec808d7cd1d43ab%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637203397873143670&sdata=MKJ610vh29gBnitDv%2F6lRVl%2BBc30Gxk83y%2FSXE1wLQM%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PBC5MOFQYUFLYBHB6DRIPUEVANCNFSM4LNOAMRQ&data=02%7C01%7C%7C69308f50316a497d9ec808d7cd1d43ab%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637203397873143670&sdata=w%2B7l59RTPL3p729ka93UN1%2BbOlknz9MqNeGiAEyc9kY%3D&reserved=0.

kalwalt commented 4 years ago

Works much much better now! arStudio-test01 i need to set the z position of the cube to -0.5 that sounds a bit strange to me, Anyway great improvement @itamayo !

ThorstenBux commented 4 years ago

Awesome, thanks so much @itamayo I'm looking at this right now. When in player mode it looks awesome as the video keeps the aspect ratio and gets cropped when resizing. I noticed in Scene mode that the video is stretched and squeezed when resizing. Do you think it is possible that it keeps the aspect ratio as well? Not very important but would be nice :).

I'm merging it now 👍

kalwalt commented 4 years ago

I noticed in Scene mode that the video is stretched and squeezed when resizing. Do you think it is possible that it keeps the aspect ratio as well? Not very important but would be nice :).

i also noticed this, i agree with you.

I'm merging it now +1

Super!

itamayo commented 4 years ago

great ! I'll check about aspect ratio ... I'm planning to check if we can implement something markeless, like using image plane detection or something... let's see want be can do, thanks

ThorstenBux commented 4 years ago

Well plane detection would be awesome. Don’t worry about the background in the editor mode. We didn’t have any before. I’m actually thinking if I’m editor mode it could be removed. I think to would be irritating anyway.

Regarding plane detection. Basically I think computing a point cloud from the video frame then getting them through solvePNP.js and maybe even optimising solvePNP with GPU.js that should be a start. I somehow think a few points that are on ground level would be a start because tbh I don’t think 8thwall does do much more. Once we anchored the object using the phone sensors to rotate or something like that.

Get Outlook for iOShttps://aka.ms/o0ukef

From: itamayo notifications@github.com Sent: Monday, March 23, 2020 9:31:54 PM To: augmentmy-world/arStudio arStudio@noreply.github.com Cc: Thorsten Bux thorsten.bux@outlook.com; State change state_change@noreply.github.com Subject: Re: [augmentmy-world/arStudio] [feature]:use video plane rendering (#28)

great ! I'll check about aspect ratio ... I'm planning to check if we can implement something markeless, like using image plane detection or something... let's see want be can do, thanks

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faugmentmy-world%2FarStudio%2Fpull%2F28%23issuecomment-602453966&data=02%7C01%7C%7Ca22b1e0b2ac84927d01308d7cf04a5c1%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205491170486836&sdata=dmkQxLDlSb%2B3mQOnAnMsijowgx6CG2OVM7iYyrT3sA0%3D&reserved=0, or unsubscribehttps://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PDZSPVFVLKE6JJYV4TRI4M7VANCNFSM4LNOAMRQ&data=02%7C01%7C%7Ca22b1e0b2ac84927d01308d7cf04a5c1%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205491170496841&sdata=RK01dGUe1viYQRi%2BpwxVb7cf7NUlGEaGAvtKD2Dco68%3D&reserved=0.

itamayo commented 4 years ago

@ThorstenBux I didn't know even about solvePNP, I'll check it, I was thinking about this implementation https://github.com/itamayo/planercnn

thanks !

ThorstenBux commented 4 years ago

That is ML based? You need so search GitHub for solvePNP.js it is part of another project. I haven’t worked with it yet but it sounded like a good idea. I also have solvePNP from openCV compiled to JS but with rather slow FPS and combining that with motion sensors might be tricky. I fear I have the theory in my head but not the background knowledge nor the time right now to go down that isle. But if you like to exchange ideas let me know we can setup a Slack.

Get Outlook for iOShttps://aka.ms/o0ukef

From: itamayo notifications@github.com Sent: Monday, March 23, 2020 9:48:59 PM To: augmentmy-world/arStudio arStudio@noreply.github.com Cc: Thorsten Bux thorsten.bux@outlook.com; Mention mention@noreply.github.com Subject: Re: [augmentmy-world/arStudio] [feature]:use video plane rendering (#28)

@ThorstenBuxhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux&data=02%7C01%7C%7Ca02728d93724446a3e2108d7cf0707f0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205501407487596&sdata=8yPm9gyyr1OW%2FCmlHL1GvdjjQP4Pb2M4nwd5skfxh%2BQ%3D&reserved=0 I didn't know even about solvePNP, I'll check it, I was thinking about this implementation https://github.com/itamayo/planercnnhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fitamayo%2Fplanercnn&data=02%7C01%7C%7Ca02728d93724446a3e2108d7cf0707f0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205501407497601&sdata=hiyWQw7H31Yks1je%2BfWEoFIUV7fwLBO9v%2FikDHaW2AM%3D&reserved=0

thanks !

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faugmentmy-world%2FarStudio%2Fpull%2F28%23issuecomment-602461337&data=02%7C01%7C%7Ca02728d93724446a3e2108d7cf0707f0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205501407507612&sdata=xnR91zW7jrYA6DCaTCrxhvovmqer%2B8WaIj5Gl5crsaI%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PFH6MKDCGL3YVGKDLLRI4O7XANCNFSM4LNOAMRQ&data=02%7C01%7C%7Ca02728d93724446a3e2108d7cf0707f0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205501407517617&sdata=FbDcBzXFUPqf304Uk5k%2F%2BjS0SvRdWVeE58ekOwWzwOM%3D&reserved=0.

ThorstenBux commented 4 years ago

Actually if we get planercnn into the web that would be better than 8wall 🤩

Get Outlook for iOShttps://aka.ms/o0ukef

From: Thorsten Bux thorsten.bux@outlook.com Sent: Monday, March 23, 2020 9:55:02 PM To: augmentmy-world/arStudio reply@reply.github.com; augmentmy-world/arStudio arStudio@noreply.github.com Cc: Mention mention@noreply.github.com Subject: Re: [augmentmy-world/arStudio] [feature]:use video plane rendering (#28)

That is ML based? You need so search GitHub for solvePNP.js it is part of another project. I haven’t worked with it yet but it sounded like a good idea. I also have solvePNP from openCV compiled to JS but with rather slow FPS and combining that with motion sensors might be tricky. I fear I have the theory in my head but not the background knowledge nor the time right now to go down that isle. But if you like to exchange ideas let me know we can setup a Slack.

Get Outlook for iOShttps://aka.ms/o0ukef

From: itamayo notifications@github.com Sent: Monday, March 23, 2020 9:48:59 PM To: augmentmy-world/arStudio arStudio@noreply.github.com Cc: Thorsten Bux thorsten.bux@outlook.com; Mention mention@noreply.github.com Subject: Re: [augmentmy-world/arStudio] [feature]:use video plane rendering (#28)

@ThorstenBuxhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FThorstenBux&data=02%7C01%7C%7Ca02728d93724446a3e2108d7cf0707f0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205501407487596&sdata=8yPm9gyyr1OW%2FCmlHL1GvdjjQP4Pb2M4nwd5skfxh%2BQ%3D&reserved=0 I didn't know even about solvePNP, I'll check it, I was thinking about this implementation https://github.com/itamayo/planercnnhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fitamayo%2Fplanercnn&data=02%7C01%7C%7Ca02728d93724446a3e2108d7cf0707f0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205501407497601&sdata=hiyWQw7H31Yks1je%2BfWEoFIUV7fwLBO9v%2FikDHaW2AM%3D&reserved=0

thanks !

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Faugmentmy-world%2FarStudio%2Fpull%2F28%23issuecomment-602461337&data=02%7C01%7C%7Ca02728d93724446a3e2108d7cf0707f0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205501407507612&sdata=xnR91zW7jrYA6DCaTCrxhvovmqer%2B8WaIj5Gl5crsaI%3D&reserved=0, or unsubscribehttps://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAD765PFH6MKDCGL3YVGKDLLRI4O7XANCNFSM4LNOAMRQ&data=02%7C01%7C%7Ca02728d93724446a3e2108d7cf0707f0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637205501407517617&sdata=FbDcBzXFUPqf304Uk5k%2F%2BjS0SvRdWVeE58ekOwWzwOM%3D&reserved=0.