search

mrousavy / react-native-vision-camera

📸 A powerful, high-performance React Native Camera library.
https://react-native-vision-camera.com
MIT License
6.64k stars 983 forks source link

‼️‼️‼️‼️ ✨ VisionCamera V3 ‼️‼️‼️‼️‼️ #1376

Closed mrousavy closed 8 months ago

mrousavy commented 1 year ago

We at Margelo are planning the next major version for react-native-vision-camera: VisionCamera V3 :sparkles:

For VisionCamera V3 we target one major feature and a ton of stability and performance improvements:

  1. Write-back frame processors. We are introducing a new feature where you can simply draw on frame in a Frame Processor using RN Skia. This allows you to draw face masks, filters, overlays, color shadings, shaders, Metal, etc..)
    • Uses a hardware accelerated Skia layer for showing the Preview
    • Some cool examples like inverted colors shader filter, VHS filter (inspired by Snapchat's VHS + distortion filter), and a realtime text/bounding box overlay
    • Realtime face blurring or license plate blurring
    • Easy to write color correction, beauty filters
    • All in simple JS (RN Skia) - no native code and hot reload while still maintaining pretty much native performance!
  2. Sync Frame Processors. Frame Processors will now be fully synchronous and run on the same Thread as the Camera is running.
    • Pretty much on-par with native performance now.
    • Run frame processing without any delay - everything until your function returns is the latest data.
    • Use runAtTargetFps(fps, ...) to run code at a throttled FPS rate inside your Frame Processor
    • Use runAsync(...) to run code on a separate thread for background processing inside your Frame Processor. This can take longer without blocking the Camera.
  3. Migrate VisionCamera to RN 0.71. Benefits:
    • Much simpler build setup. The CMakeLists/build.gradle files will be simplified as we will use prefabs, and a ton of annoying build errors should be fixed.
    • Up to date with latest React Native version
    • Prefabs support on Android
    • No more Boost/Glog/Folly downloading/extracting
  4. Completely redesigned declarative API for device/format selection (resolution, fps, low-light, ...)
    • Control exactly how much FPS you want to record in
    • Know exactly if a desired format is supported and be able to fall back to a different one
    • Control the exact resolution and know what is supported (e.g. higher than 1080, but no higher than 4k, ...)
    • Control settings like low-light mode, compression, recording format H.264 or H.265, etc.
    • Add reactive API for getAvailableCameraDevices() so external devices can become plugged in/out during runtime
    • Add zero-shutter lag API for CameraX
  5. Rewrite the native Android part from CameraX to Camera2
    • Much more stability as CameraX just isn't mature enough yet
    • Much more flexibility with devices/formats
    • Slow-motion / 240 FPS recording on Android
  6. Use a custom Worklet Runtime instead of Reanimated
    • Fixes a ton of crashes and stability issues in Frame Processors/Plugins
    • Improves compilation time as we don't need to extract Reanimated anymore
    • Doesn't break with a new Reanimated version
    • Doesn't require > Reanimated v2 anymore
  7. ML Models straight from JavaScript. With the custom Worklet Runtime, you can use outside HostObjects and HostFunctions. This allows you to just use things like TensorFlow Lite or PyTorch Live in a Frame Processor and run ML Models fully from JS without touching native code! (See proof of concept PR: https://github.com/facebookresearch/playtorch/pull/199 and working PR for Tensorflow: https://github.com/mrousavy/react-native-vision-camera/pull/1633)
  8. Improve Performance of Frame Processors by caching FrameHostObject instance
  9. Improve error handling by using default JS error handler instead of console.error (mContext.handleException(..))
  10. More access to the Frame in Frame Processors:
    • toByteArray(): Gets the Frame data as a byte array. The type is Uint8Array (TypedArray/ArrayBuffer). Keep in mind that Frame buffers are usually allocated on the GPU, so this comes with a performance cost of a GPU -> CPU copy operation. I've optimized it a bit to run pretty fast :)
    • orientation: The orientation of the Frame. e.g. "portrait"
    • isMirrored: Whether the Frame is mirrored (eg in selfie cams)
    • timestamp: The presentation timestamp of the Frame

Of course we can't just put weeks of effort into this project for free. This is why we are looking for 5-10 partners who are interested in seeing this become reality by funding the development of those features. Ontop of seeing this become reality, we also create a sponsors section for your company logo in the VisionCamera documentation/README, and we will test the new VisionCamera V3 version in your app to ensure it's compatibility for your use-case. If you are interested in that, reach out to me over Twitter: https://twitter.com/mrousavy or email: me@mrousavy.com

Demo

Here's the current proof of concept we built in 3 hours:

const runtimeEffect = Skia.RuntimeEffect.Make(`
  uniform shader image;
  half4 main(vec2 pos) {
    vec4 color = image.eval(pos);
    return vec4((1.0 - color).rgb, 1.0);
  }
`);

const paint = paintWithRuntimeEffect(runtimeEffect)

function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    frame.drawPaint(paint)
  }, [])

  return <Camera frameProcessor={frameProcessor} />
}




Progress

Currently, I spent around ~60 hours to improve that proof of concept and created the above demos. I also refined the iOS part a bit and created some fixes, did some research and improved the Skia handling.

Here is the current Draft PR: https://github.com/mrousavy/react-native-vision-camera/pull/1345

Here's a TODO list:

I reckon this will be around 500 hours of effort in total.

Update 15.2.2023: I just started working on this here: feat: ✨ V3 ✨ #1466. No one is paying me for that so I am doing all this in my free time. I decided to just ignore issues/backlash so that I can work as productive as I can. If someone is complaining, they should either offer a fix (PR) or pay me. If I listen to all issues the library will never get better :)

xts-bit commented 1 year ago

@mrousavy 500mb the size of the package is a lot it increases the app size by a lot if you do npm install -g cost-of-modules and then Run cost-of-modules in the directory you are working in the size is 500mb

mrousavy commented 1 year ago

@syblus446 you're looking at buildcache. The package size is not 500MB.

On npmjs.org you can see that the size of the react-native-vision-camera package is 747kB, and the actual JS bundle part of that is even smaller:

image

levipro commented 1 year ago

Is there some trick to get tap to focus working? I have been integrating v3 into an app as a replacement for an older camera library but on my iPhone 13 I cannot get the tap to focus working

<TapGestureHandler
  numberOfTaps={1}
  onEnded={async ({ nativeEvent }) => {
  if (device?.supportsFocus) {
     await camera.current!.focus({ x: nativeEvent.x, y: nativeEvent.y });
  }
}}>
  <Camera ... />
</TapGestureHandler>
levipro commented 1 year ago

Is there some trick to get tap to focus working? I have been integrating v3 into an app as a replacement for an older camera library but on my iPhone 13 I cannot get the tap to focus working

<TapGestureHandler
  numberOfTaps={1}
  onEnded={async ({ nativeEvent }) => {
  if (device?.supportsFocus) {
     await camera.current!.focus({ x: nativeEvent.x, y: nativeEvent.y });
  }
}}>
  <Camera ... />
</TapGestureHandler>

https://github.com/mrousavy/react-native-vision-camera/blob/1ddea178ae4ec07213372844821f90ce3a94455a/ios/CameraView%2BFocus.swift#L83-L87

Fix: https://github.com/mrousavy/react-native-vision-camera/pull/1541

levipro commented 1 year ago

Another issue: this new method does not match what videoPreviewLayer.captureDevicePointConverted(fromLayerPoint: point) would generate. This means any tap to focus event will not be sent to the same spot that the touch event happened at

https://github.com/mrousavy/react-native-vision-camera/blob/1ddea178ae4ec07213372844821f90ce3a94455a/ios/CameraView%2BFocus.swift#L25-L26

xts-bit commented 1 year ago

@mrousavy It's possible that this package has a large number of dependencies or includes large files that contribute to its size. you can create a new react native project and install react-native-vision-camera and then do npm install -g cost-of-modules and then cost-of-modules it's too large

It also increases app bundle size when it comes to doing production

mrousavy commented 1 year ago

@levipro yea I left out tap to focus on Android now as well. On iOS, the code should work though, it did work in previous versions.. 🤔 Does videoPreviewLayer.captureDevicePointConverted(fromLayerPoint: point) not do what it did previously?

mrousavy commented 1 year ago

@mrousavy It's possible that this package has a large number of dependencies or includes large files that contribute to its size. you can create a new react native project and install react-native-vision-camera and then do npm install -g cost-of-modules and then cost-of-modules it's too large

It also increases app bundle size when it comes to doing production

@syblus446 again, this is off-topic. Create a separate issue for this. This is the V3 discussion.

It's possible that this package has a large number of dependencies

It has zero runtime dependencies. It has peerDependencies on react and react-native, which are packages you always have installed if you use VisionCamera. Maybe you're looking at devDependencies: https://github.com/mrousavy/react-native-vision-camera/blob/ba099e4044714a2178eff9c6ef148420124cd416/package.json#L71-L88

It also increases app bundle size when it comes to doing production

By how much? Do you have a before and after comparison?

levipro commented 1 year ago

@levipro yea I left out tap to focus on Android now as well. On iOS, the code should work though, it did work in previous versions.. 🤔 Does videoPreviewLayer.captureDevicePointConverted(fromLayerPoint: point) not do what it did previously?

The old version (2.15.4) definitely did not work. The continuous autofocus overrides everything and the tap does nothing. The secondary problem that was introduced in v3 (I am assuming to make accommodation for the skia preview option) is that the custom captureDevicePointConverted method was added to replace calling ...videoPreviewLayer.captureDevicePointConverted(fromLayerPoint: point). This is a problem because it is calculating an entirely different coordinate. So when the videoPreviewLayer is actually available that built in method should be used. I have made that change as well. Now the tap to focus works properly and the coordinate system for that event is also working again

xts-bit commented 1 year ago

@mrousavy You can see its size any solution to fix it? Capture

bglgwyng commented 1 year ago

I wrote a custom frame processor in object-oriented style on iOS(https://github.com/mrousavy/react-native-vision-camera/commit/622d3830f1e2e728255abfcf3ee237bc4267e487). However, FrameProcessorPlugins is an empty object and it seems that my frame processor is not detected. Here is the repo. It contains ExamplePlguin in the RNVC3's example. Do I have additional steps to register a frame processor? I just created the same files that the RNVC3's example has and this way worked in RNVC V2.

Also, the latest version in v3 branch throws a build error on iOS.

/.../node_modules/react-native-vision-camera/ios/CameraError.swift:264:21: error: cannot find type 'SystemError' in scope
  case system(_ id: SystemError)

It can be easily fixed by commenting on that line. I made a patch script, though I'm not sure this is the right way. I wonder if others saw the same error. I'm using XCode 14.2.

mrousavy commented 1 year ago

@bglgwyng ah yep, this requires one additional step now - you need to manually register the plugin, just like on Android. This can be done either with a + load method, or in your AppDelegate start func. So call [ExamplePlugin registerPlugin] Exactly here

Also - yep the SystemError not found thing is already fixed but not yet released, thx :)

johnvsmit31 commented 1 year ago

@mrousavy Hello Marc VisionCamera is a great package but I use your package react-native-jsi-image But unfortunately, it's not working with the latest version of react native Can you please give me a solution to fix that I tried creating an issue on the repo but I can't found any solution of that problem CMake Error at CMakeLists.txt:6 (add_library): Cannot find source file: ../cpp/TypedArray/TypedArray.cpp

After Chatgpt it gives me this response when none of the solution work can you please help me? Capture

mrousavy commented 1 year ago

@bglgwyng did registering the FP in AppDelegate work for you?

bglgwyng commented 1 year ago

@bglgwyng did registering the FP in AppDelegate work for you?

Yes, it worked! I'm sorry I didn't let you know. I just thought a reaction was enough.

mrousavy commented 1 year ago

All good, glad to hear it works! Currently don't have a lot of free time but I'll continue the work on V3 soon thanks to some new sponsorships I got!

guthriet1230 commented 1 year ago

you're a wizard harry - thanks for all your generosity!

ratz6 commented 1 year ago

Any ETA reg V3 ?

rkmackinnon commented 1 year ago

I have recently been working on updating https://github.com/rodgomesc/vision-camera-code-scanner/ to support VisionCamera V3. @bglgwyng and I came across an interesting compile error when trying to build projects with use_frameworks! :linkage => :static. It is the same error referenced here: https://github.com/mrousavy/react-native-vision-camera/issues/1043.

I found I could work around this error by patching in an empty VisionCamera.h file and referencing it in the podspec. Some auto-generated code during the XCode build process apparently expects this header to exist. I'm not sure if this is the right way to go about solving this problem, but it does fix the compile error. Perhaps the community can comment on whether it is reasonable or if there is a preferable alternative? If there are no objections, I could submit a pull request with the fix.

mrousavy commented 1 year ago

Interesting, shouldn't this header be generated by CocoaPods? If nothing breaks, I'm happy to use this header.

bglgwyng commented 1 year ago

I made a simple patch script that makes react-native-skia@^0.1.176 work with vision camera V3. It's just a revert of the diff between react-native-skia@0.1.175(the last version that works with vision camera V3) on the podspec file and react-native-skia@0.1.185. You can use the latest version of react-native-skia with vision camera V3 by applying this patch. Since it's a revert of a library, I'm very suspicious of this being a proper solution. Does anyone have a better solution?

mrousavy commented 1 year ago

Good idea @bglgwyng! I'm talking with @chrfalch to figure this one out and make sure the newest version stays compatible, it's probably gonna be a change on my end, not on Skia's end.

Another exciting update: I'm flying in @thomas-coldwell to our Vienna office next week and we'll do a week long hackathon on VisionCamera V3. We'll attempt to rewrite the Android part to Camera2, try the RN Skia part implementation there, and ideally fix a few bugs (like the build error with Skia) and create a few examples, I'll keep you guys updated. 🚀

bglgwyng commented 1 year ago

I have discovered that it is not possible to directly access reanimated's shared value from the frame processor. As an example, the code snippet provided below:

const rotationSensor = useAnimatedSensor(SensorType.ROTATION);

const frameProcessor = useFrameProcessor((frame) => {
  'worklet';
  console.info(rotationSensor.sensor.value);
}, []);

results in the following error message:

Error: Reading from `_value` directly is only possible on the UI runtime, js engine: hermes

It appears that this error occurs because the frame processor is not considered to be reanimated's worklet.

I am left wondering if there is currently, or will be in the future, a way to read and write the value of shared values across different types of worklets.

xts-bit commented 1 year ago

@mrousavy When will the Face Detector Module will launch with react-native-vision-camera

mrousavy commented 1 year ago

I am left wondering if there is currently, or will be in the future, a way to read and write the value of shared values across different types of worklets.

@bglgwyng so this is because I am using react-native-worklets instead of Reanimated. Since Reanimated 3, we have similar benefits as react-native-worklets, however I cannot use Reanimated because they don't provide the API's that I need in VisionCamera:

I talked with @tomekzaw about this, but at the moment I don't have the time to implement this for Reanimated. If the Software Mansion team implements those APIs, I can easily switch to REA :)

mrousavy commented 1 year ago

@thomas-coldwell and I just finished building a really cool demo.

This shows how groundbreaking VisionCamera V3 really is for the entire mobile camera industry - instead of building two highly complex native apps and integrating shaders, canvases, Texture Views, TFLite models, etc. into your app, you can simply use VisionCamera's easy to use JavaScript APIs.

For running the model, we have a Tensorflow Lite abstraction. The model can easily be tweaked and swapped out at runtime using hot-reload.

For drawing the blur and the hands, we use @shopify/react-native-skia. That's all C++/GPU based operations as well.

For running the Frame Processor we use the native Camera APIs which are fast and synchronous.

The entire abstraction is 1ms slower than a fully native app, which is nothing in my opinion.

Here's the tweet

https://user-images.githubusercontent.com/15199031/236502632-2c8dcd6b-49a5-4481-96fe-a27da795b342.mov

See code ```tsx const blurEffect = Skia.RuntimeEffect.Make(FACE_PIXELATED_SHADER); if (blurEffect == null) throw new Error('Shader failed to compile!'); const blurShaderBuilder = Skia.RuntimeShaderBuilder(blurEffect); const blurPaint = Skia.Paint(); const linePaint = Skia.Paint(); linePaint.setStrokeWidth(10); linePaint.setStyle(PaintStyle.Stroke); linePaint.setColor(Skia.Color('lightgreen')); const dotPaint = Skia.Paint(); dotPaint.setStyle(PaintStyle.Fill); dotPaint.setColor(Skia.Color('red')); // load the two Tensorflow Lite models - those can be swapped out at runtime and hot-reloaded - just like images :) const faceDetection = loadModel(require("../assets/face_detection.tflite")) const handDetection = loadModel(require("../assets/hand_detection.tflite")) const frameProcessor = useFrameProcessor((frame) => { 'worklet' // runs native TensorFlow Lite model on GPU - fast! const { faces } = faceDetection.run(frame) // blur faces using Skia for (const face of faces) { const centerX = (face.x + face.width / 2); const centerY = (face.y + face.height / 2); const radius = Math.max(face.width, face.height) / 2; blurShaderBuilder.setUniform('x', [centerX]); blurShaderBuilder.setUniform('y', [centerY]); blurShaderBuilder.setUniform('r', [radius]); const imageFilter = Skia.ImageFilter.MakeRuntimeShader(blurShaderBuilder, null, null); blurPaint.setImageFilter(imageFilter); frame.render(blurPaint); } // runs native TensorFlow Lite model on GPU - fast! const { hands } = handDetection.run(frame) // show hand outlines using Skia for (const hand of hands) { // green lines: for (const line of hand.lines) { frame.drawLine( line.from.x, line.from.y, line.to.x, line.to.y, linePaint ) } // red dots: for (const dot of hand.dots) { frame.drawCircle( dot.x, dot.y, dot.size, dotPaint ) } } }, [linePaint, dotPaint]) return ```
idrissakhi commented 1 year ago

nice work @mrousavy , when it will be available v3 i mean ? or at least a rc-3

xts-bit commented 1 year ago

@mrousavy From where can we get face_detection.tflite? is there github repo for this demo code?

bglgwyng commented 1 year ago

@mrousavy From where can we get face_detection.tflite? is there github repo for this demo code?

It might be Blazeface. The pre-trained model file can be obtained at https://github.com/ibaiGorordo/BlazeFace-TFLite-Inference/tree/main/models.

However, as far as I know, since ML Kit face detector also uses the same model, I'm not sure you can see the difference in accuracy or performance compared to vision-camera-face-detector. Also, you need to write the post-processing of inference, such as filtering the bounding box. This is where improvements can be made. It is also worth mentioning that ML Kit's face tracking is not very stable.

xts-bit commented 1 year ago

@bglgwyng vision-camera-face-detector is good but fails my release build on Android every time I try will this work with react native its written in python?

bglgwyng commented 1 year ago

@xts-bit ~I don't know about the Android build issue and perhaps it'd be better to be discussed in a separate issue. You can find .tflite file in the link I provided.~

Sorry for the misinformation. I just found the branch of that cool demo https://github.com/mrousavy/react-native-vision-camera/pull/1586.

xts-bit commented 1 year ago

@bglgwyng I checked this branch there are other face detector files however I can't find that face_detection.tflite"file as code example

xts-bit commented 1 year ago

@mrousavy When will vision camera v3 will be launch? will it support ios simulators in v3? When will face detector modules will will be launch?

mrousavy commented 1 year ago

Thanks @bglgwyng, yep it uses BlazeFace but for now the FP Plugin was built with MediaPipe. I wanted to experiment more with a general purpose TFLite FP Plugin, but I focused on Android the past few days.

xts-bit commented 1 year ago

@mrousavy from where did you get face_detection.tflite? can i get this file?

timothygorer commented 1 year ago

Would it be possible to plug in your own text recognition model and have it detect text using that?

ZakirBangash commented 1 year ago

@mrousavy You are doing amazing work, hats off to you, currently we are using in production ready app, but when we record video on front camera it flip the video in preview, while ios is working fine, we have fixed it but it takes much time while processing flipping issue., looking forward to build in support for this

bglgwyng commented 1 year ago

Have you considered the possibility of passing a Skia surface to the frame processor alongside the existing ImageProxy on Android and CMSampleBufferRef on iOS? This would enable the utilization of Skia for image processing while avoiding unnecessary memory allocation and copying.

To provide some context, I encountered a situation where I needed to perform preprocessing on an image for ML inference. Due to the limited image processing functions available in Android (specifically, cropping an image by a rotated rectangle), I resorted to using OpenGL. This involved creating an OpenGL texture from the video frame by obtaining the buffer and converting the color space from YUV to RGB. However, I realized that the vision camera already performs a similar operation. To avoid redundancy and optimize performance, I began exploring the possibility of using the same buffer for preview purposes.

Given my limited knowledge of Android and iOS image objects, I wanted to confirm my understanding of the current situation before suggesting any changes. Currently, I use the Bitmap object on Android and CIImage on iOS for image processing, both obtained from the frame object passed to the frame processor. I would like to know if using these objects results in additional memory allocation and copying, or if they simply act as wrappers for the original buffer without allocating extra memory. If they are just wrappers, then my current approach doesn't affect performance, despite its suboptimal nature of duplicating boilerplate code. However, if extra memory allocation or copying occurs, many frame processors would make it happen unnecessarily.

I referred to it as a Skia surface earlier in my comment, but I'm uncertain if it's the correct object for the described usage. Does the current implementation of the V3 camera have an object that can be utilized for this purpose?

mrousavy commented 1 year ago

Hey @bglgwyng

Have you considered the possibility of passing a Skia surface to the frame processor alongside the existing ImageProxy on Android and CMSampleBufferRef on iOS? This would enable the utilization of Skia for image processing while avoiding unnecessary memory allocation and copying.

not sure if I understand correctly, but the purpose of V3 is to allow you to draw on a Skia context. The Skia canvas/surface/context is passed to the Frame Processor, and you can draw on it straight from JS.

On the native side (in a FP Plugin) you receive the Frame. The Frame object will hold the native buffer. You can directly use that buffer, without making any copies. Afaik CIImage does not copy, not sure about Bitmap.

Converting from YUV to RGB does copy, but often you need to do that. On iOS, the Frame Processor Frames are already in RGB when using Skia, but YUV when not using Skia. I think I should also add a prop to the Camera to configure it for which colorspace to use, RGB or YUV.

bglgwyng commented 1 year ago

@mrousavy Thank you for your response! I apologize for any confusion caused by my previous question. Allow me to clarify my query.

I did not intend to suggest drawing on the Skia surface within the native FP plugin. I understand that it may not have any immediate practical value. Instead, I was considering utilizing the Skia texture that contains the original frame to preprocess the image and obtain the tensor object.

To provide a more detailed overview of my current situation, I have shared a face recognition demo. The current flow of the demo, without incorporating Skia surface, is as follows:

graph TD;
    CMSampleBuffer-->CVImageBuffer-1;
    CVImageBuffer-1-->SkiaTexture;
    SkiaTexture--drawback frame proceessor-->SkiaSurface-Preview;
    CMSampleBuffer-->CVImageBuffer-2;
    CVImageBuffer-2-->CIImage;
    CIImage--crop face with CoreImage operations-->CGImage;
    CGImage-->Tensor;

Since I am not satisfied with the functionality of CIImage, I contemplated using Skia to preprocess the image and obtain the tensor object. This would result in the following modified flow:

graph TD;
    CMSampleBuffer-->CVImageBuffer-1;
    CVImageBuffer-1-->SkiaTexture-1;
    SkiaTexture-1--drawback frame proceessor-->SkiaSurface-Preview;
    CMSampleBuffer-->CVImageBuffer-2;
    CVImageBuffer-2-->SkiaTexture-2;
    SkiaTexture-2--crop face with Skia operations-->Tensor;

I have concerns about the potential creation of an additional copy when generating the Skia texture from CVImageBuffer. Although this issue may not arise in iOS, I wonder how it would work when creating the texture from ImageProxy on Android. It's possible that creating a Bitmap object first to create the texture, and converting the ImageProxy to a Bitmap object, may involve additional copies due to the YUV to RGB translation process.

Considering these factors, I arrived at the following alternative approach:

graph TD;
    CMSampleBuffer-->CVImageBuffer;
    CVImageBuffer-->SkiaTexture;
    SkiaTexture--drawback frame proceessor-->SkiaSurface-Preview;
    SkiaTexture--crop face with Skia operations-->Tensor;

In this scenario, the native FP plugin receives the Skia texture as a parameter, such as frame.skiaTexture, and can utilize it internally.

I would appreciate your insights on the feasibility of this approach. If my understanding of the flow is incorrect, please kindly correct me.

mrousavy commented 1 year ago

huh, that's interesting. The native skiaTexture would be a void* and you'd have to go into C++, then make sure all Skia bindings and linkings are set up properly, and then you can use the Texture, but I guess that could work.

What you could do in the latest Skia version is to just do frame.takeSnapshot() and you'd get an SkImage of the currently rendered Canvas - but again, this is a copy. I'll think about this!

pateljigar7210 commented 11 months ago

yarn add react-native-vision-camera@rc yarn add react-native-worklets@https://github.com/chrfalch/react-native-worklets yarn add @shopify/react-native-skia

shubham-stan commented 11 months ago

hi @mrousavy, Looking to support this awesome library. Have been following it for a while and wondering is the development for this dead? saw a lot happening around feb-march. if this is alive and kicking would love to support with at least $20 a month to support it

mrousavy commented 11 months ago

is the development for this dead?

No, it's not dead, I'm exploring different things on other branches and working on it from time to time. I'm not working on VisionCamera fulltime obviously, but it's coming along. The challenge is a bit harder than I originally anticipated, Skia interop is quite a complex task

yadigarbz commented 11 months ago

On RN 0.71.7 I was trying to use rc2 version then suddenly I faced with this issue

/Users/yadigarberkayzengin/projects/yabu/RNBugMapper/android/app/build/generated/rncli/src/main/java/com/facebook/react/PackageList.java:83: error: cannot find symbol
import com.mrousavy.camera.example.CameraPackage;
                                  ^
  symbol:   class CameraPackage
  location: package com.mrousavy.camera.example
/Users/yadigarberkayzengin/projects/yabu/RNBugMapper/android/app/build/generated/rncli/src/main/java/com/facebook/react/PackageList.java:169: error: cannot find symbol
      new CameraPackage(),
          ^
  symbol:   class CameraPackage
  location: class PackageList

I'm pretty aware it's in the early development phase, just wanted to contribute if you find this useful. I'd like to give a hand too, I wish I had fewer burry on my shoulders at the moment

pritish-thub commented 11 months ago

Try this https://www.youtube.com/playlist?list=PLQhQEGkwKZUrempLnmxjt7ZCZJu1W3p2i

stemyke commented 11 months ago

@mrousavy Hi Mark!

I also wanted to try out v3 rc2 in my project but with no luck. I got the following error:

> Configure project :react-native-vision-camera
react-native-vision-camera: Skia integration is enabled!
5 actionable tasks: 5 up-to-date

FAILURE: Build failed with an exception.

* Where:
Build file 'D:\Work\RNEngravingApp\node_modules\react-native-vision-camera\android\build.gradle' line: 150

* What went wrong:
A problem occurred evaluating project ':react-native-vision-camera'.
> Project with path ':react-native-worklets' could not be found in project ':react-native-vision-camera'.

However it is present in my package.json

{
  "name": "RNEngravingApp",
  "dependencies": {
    "react": "18.2.0",
    "react-native": "0.71.10",
    "react-native-reanimated": "2.17.0",
    "react-native-worklets": "0.0.1-alpha",
    "react-native-vision-camera": "3.0.0-rc.2",
    "@shopify/react-native-skia": "0.1.193"
  },
}

I used yarn to install the package and deleted the lock file also. I also reached out to you in e-mail to support the project.

fukemy commented 11 months ago

you

@mrousavy Hi Mark!

I also wanted to try out v3 rc2 in my project but with no luck. I got the following error:

Configure project :react-native-vision-camera react-native-vision-camera: Skia integration is enabled! 5 actionable tasks: 5 up-to-date

FAILURE: Build failed with an exception.

  • Where: Build file 'D:\Work\RNEngravingApp\node_modules\react-native-vision-camera\android\build.gradle' line: 150
  • What went wrong: A problem occurred evaluating project ':react-native-vision-camera'.

Project with path ':react-native-worklets' could not be found in project ':react-native-vision-camera'.

However it is present in my package.json

{ "name": "RNEngravingApp", "version": "0.0.1", "private": true, "scripts": { "android": "react-native run-android", "ios": "react-native run-ios", "lint": "eslint .", "start": "react-native start", "test": "jest" }, "dependencies": { "react": "18.2.0", "react-native": "0.71.10", "react-native-reanimated": "2.17.0", "react-native-worklets": "0.0.1-alpha", "react-native-vision-camera": "3.0.0-rc.2", "@shopify/react-native-skia": "0.1.193" }, "devDependencies": { "@babel/core": "^7.20.0", "@babel/preset-env": "^7.20.0", "@babel/runtime": "^7.20.0", "@react-native-community/eslint-config": "^3.2.0", "@tsconfig/react-native": "^2.0.2", "@types/jest": "^29.2.1", "@types/react": "^18.0.24", "@types/react-test-renderer": "^18.0.0", "babel-jest": "^29.2.1", "eslint": "^8.19.0", "jest": "^29.2.1", "metro-react-native-babel-preset": "0.73.9", "prettier": "^2.4.1", "react-test-renderer": "18.2.0", "typescript": "4.8.4" }, "jest": { "preset": "react-native" } }

I used yarn to install the package and deleted the lock file also. I also reached out to you in e-mail to support the project.

you need to install exactly this version "react-native-worklets": "github:chrfalch/react-native-worklets#d62d76c",

stemyke commented 11 months ago

@fukemy Thanks, now it goes along with the previous Issue, now I have the same problem as @yadigarbz on Android. Maybe the current version only works on iOS? I thought only the features will be less on Android and will not have build errors. But still I am excited for this release and cant wait to see it working on Android too.

fukemy commented 11 months ago

I am testing for IOS first, I got error build on Android too belong of caknzckaovjbsdovbsdovjb... problem, i have to disable auto linki for visioncamera on Android, and waiting for news from v3