search

viromedia / viro

ViroReact: AR and VR using React Native
MIT License
2.31k stars 483 forks source link

Image processing #401

Closed bendeve closed 5 years ago

bendeve commented 6 years ago

HI,

Is it possible to do some image processing while using the ViroARSceneNavigator ?

I mean using some libraries (openGL, ...) to do image processing or simply access/modify each pixels RGB color.

If yes is there any example ?

Thank you.

manbod commented 6 years ago

Hi @bendeve , Can you elaborate on what use cases you are trying to achieve? We currently don't have a way to expose realtime camera frames (due to performance constraints involved in passing frame buffer data from native all the way to react-native), but we do so in ViroCore. We do have a few limited post-processing effects you can try by passing postProcessEffects in ViroARScene. Documentation here -> https://docs.viromedia.com/v2.10.0/docs/viroarscene#post-process-effects. We don't have the ability for developers to write custom post processing shaders, but it is something we have in our backlog. Here is an open issue tracking that work -> https://github.com/viromedia/viro/issues/259

Thanks,

kennedyryan commented 5 years ago

Hi @manbod,

I am also interested in being able to grab the raw camera frame from within ViroReact. Essentially what I am trying to do is grab a frame or couple of frames from the camera to do some OCR(Optical Character Recognition) when an image target is detected. Thus I am not overly concerned about the performance issues since I do not need the image in realtime. A bonus would be also getting the pixel coordinates of the detected image(AR Marker) inside of the larger raw camera image.

Any help achieving all or part of this would be greatly appreciated. So far I am loving the simplicity of the sdk.

Thanks.

kennedyryan commented 5 years ago

@manbod False alarm. I believe I have figured out a workaround that will work for me.

For any that follow me looking for a possible solution I am planning on just using the takeScreenshot method that exists on the ViroARSceneNavigator object. I wouldn't recommend attempting to do this for every frame but since I only need a single frame to do some OCR before moving on to another screen it should work in my case.