‼️‼️‼️‼️ ✨ VisionCamera V3 ‼️‼️‼️‼️‼️

[mrousavy]

We at Margelo are planning the next major version for react-native-vision-camera: VisionCamera V3 :sparkles:

For VisionCamera V3 we target one major feature and a ton of stability and performance improvements:

1. Write-back frame processors. We are introducing a new feature where you can simply draw on frame in a Frame Processor using RN Skia. This allows you to draw face masks, filters, overlays, color shadings, shaders, Metal, etc..)
   • Uses a hardware accelerated Skia layer for showing the Preview
   • Some cool examples like inverted colors shader filter, VHS filter (inspired by Snapchat's VHS + distortion filter), and a realtime text/bounding box overlay
   • Realtime face blurring or license plate blurring
   • Easy to write color correction, beauty filters
   • All in simple JS (RN Skia) - no native code and hot reload while still maintaining pretty much native performance!
2. Sync Frame Processors. Frame Processors will now be fully synchronous and run on the same Thread as the Camera is running.
   • Pretty much on-par with native performance now.
   • Run frame processing without any delay - everything until your function returns is the latest data.
   • Use runAtTargetFps(fps, ...) to run code at a throttled FPS rate inside your Frame Processor
   • Use runAsync(...) to run code on a separate thread for background processing inside your Frame Processor. This can take longer without blocking the Camera.
3. Migrate VisionCamera to RN 0.71. Benefits:
   • Much simpler build setup. The CMakeLists/build.gradle files will be simplified as we will use prefabs, and a ton of annoying build errors should be fixed.
   • Up to date with latest React Native version
   • Prefabs support on Android
   • No more Boost/Glog/Folly downloading/extracting
4. Completely redesigned declarative API for device/format selection (resolution, fps, low-light, ...)
   • Control exactly how much FPS you want to record in
   • Know exactly if a desired format is supported and be able to fall back to a different one
   • Control the exact resolution and know what is supported (e.g. higher than 1080, but no higher than 4k, ...)
   • Control settings like low-light mode, compression, recording format H.264 or H.265, etc.
   • Add reactive API for getAvailableCameraDevices() so external devices can become plugged in/out during runtime
   • Add zero-shutter lag API for CameraX
5. Rewrite the native Android part from CameraX to Camera2
   • Much more stability as CameraX just isn't mature enough yet
   • Much more flexibility with devices/formats
   • Slow-motion / 240 FPS recording on Android
6. Use a custom Worklet Runtime instead of Reanimated
   • Fixes a ton of crashes and stability issues in Frame Processors/Plugins
   • Improves compilation time as we don't need to extract Reanimated anymore
   • Doesn't break with a new Reanimated version
   • Doesn't require > Reanimated v2 anymore
7. ML Models straight from JavaScript. With the custom Worklet Runtime, you can use outside HostObjects and HostFunctions. This allows you to just use things like TensorFlow Lite or PyTorch Live in a Frame Processor and run ML Models fully from JS without touching native code! (See proof of concept PR: https://github.com/facebookresearch/playtorch/pull/199 and working PR for Tensorflow: https://github.com/mrousavy/react-native-vision-camera/pull/1633)
8. Improve Performance of Frame Processors by caching FrameHostObject instance
9. Improve error handling by using default JS error handler instead of console.error (mContext.handleException(..))
10. More access to the Frame in Frame Processors:
    • toByteArray(): Gets the Frame data as a byte array. The type is Uint8Array (TypedArray/ArrayBuffer). Keep in mind that Frame buffers are usually allocated on the GPU, so this comes with a performance cost of a GPU -> CPU copy operation. I've optimized it a bit to run pretty fast :)
    • orientation: The orientation of the Frame. e.g. "portrait"
    • isMirrored: Whether the Frame is mirrored (eg in selfie cams)
    • timestamp: The presentation timestamp of the Frame

Of course we can't just put weeks of effort into this project for free. This is why we are looking for 5-10 partners who are interested in seeing this become reality by funding the development of those features. Ontop of seeing this become reality, we also create a sponsors section for your company logo in the VisionCamera documentation/README, and we will test the new VisionCamera V3 version in your app to ensure it's compatibility for your use-case. If you are interested in that, reach out to me over Twitter: https://twitter.com/mrousavy or email: me@mrousavy.com

---

Demo

Here's the current proof of concept we built in 3 hours:

const runtimeEffect = Skia.RuntimeEffect.Make(`
  uniform shader image;
  half4 main(vec2 pos) {
    vec4 color = image.eval(pos);
    return vec4((1.0 - color).rgb, 1.0);
  }
`);

const paint = paintWithRuntimeEffect(runtimeEffect)

function App() {
  const frameProcessor = useFrameProcessor((frame) => {
    'worklet'
    frame.drawPaint(paint)
  }, [])

  return <Camera frameProcessor={frameProcessor} />
}

---

Progress

Currently, I spent around ~60 hours to improve that proof of concept and created the above demos. I also refined the iOS part a bit and created some fixes, did some research and improved the Skia handling.

Here is the current Draft PR: https://github.com/mrousavy/react-native-vision-camera/pull/1345

Here's a TODO list:

• [ ] iOS
  • [x] Set up Skia Preview
  • [x] Pass Skia Canvas to JS Frame Processor
  • [x] Make sure we use high performance Skia drawing operations
  • [x] Make sure Frames are not out of sync of the screen refresh rate (60Hz / 120Hz)
  • [x] Do some performance profiling and see if we can improve something
  • [x] Make sure everything continues to work when not using Skia
  • [x] Swap the REA Runtime with the custom Worklet Runtime
  • [x] Implement synchronous Frame Processors
  • [x] Implement runAtTargetFps
  • [x] Implement runAsync
  • [x] Implement toByteArray(), orientation, isMirrored and timestamp on Frame
  • [x] Add orientation to Frame
  • [ ] Convert it to a TurboModule/Fabric
  • [ ] Rewrite to new simple & declarative API
• [ ] Android
  • [x] Set up Skia Preview
  • [x] Pass Skia Canvas to JS Frame Processor
  • [x] Make sure we use high performance Skia drawing operations
  • [ ] Make sure Frames are not out of sync of the screen refresh rate (60Hz / 120Hz)
  • [x] Do some performance profiling and see if we can improve something
  • [x] Make sure everything continues to work when not using Skia
  • [x] Swap the REA Runtime with the custom Worklet Runtime
  • [x] Implement synchronous Frame Processors
  • [x] Implement runAtTargetFps
  • [x] Implement runAsync
  • [x] Implement toByteArray(), orientation, isMirrored and timestamp on Frame
  • [x] Add orientation to Frame
  • [ ] Convert it to a TurboModule/Fabric
  • [x] Rewrite from Camera2 to CameraX
  • [ ] Rewrite to new simple & declarative API
• [ ] Documentation
  • [ ] Create documentation for write-back/Skia Frame Processors
  • [x] Create documentation for synchronous Frame Processors
  • [x] Create documentation for runAtTargetFps
  • [x] Create documentation for runAsync
  • [ ] Create a realtime face blurring example
  • [ ] Create a realtime license plate blurring example
  • [ ] Create a realtime text recognition/overlay example
  • [ ] Create a realtime color grading/beauty filter example
  • [ ] Create a realtime face outline/landmarks detector example

I reckon this will be around 500 hours of effort in total.

Update 15.2.2023: I just started working on this here: feat: ✨ V3 ✨ #1466. No one is paying me for that so I am doing all this in my free time. I decided to just ignore issues/backlash so that I can work as productive as I can. If someone is complaining, they should either offer a fix (PR) or pay me. If I listen to all issues the library will never get better :)

[mrousavy]

Write-Back Frame Processors will shape the future of realtime image processing on mobile. For reference, let's compare this example, w/ native iOS/Android vs w/ RN VisionCamera:

Imagine how you'd do that in a native app:

• ~700 lines of code across ~5 files to set up the Camera accordingly
• ~300 lines of very low-level C-style Metal code across ~3 files to set up Metal
• ~40 lines of Metal Shader code to draw the Frame from a Texture (with the box) to the Layer
• ~30 lines of code to set up the Face Detector Module
• Then do the same thing again for Android.

In VisionCamera, it's just:

• ~4 lines of code to set up the <Camera>
• ~35 lines of code to set up the Face Detector Frame Processor Plugin
• ~13 lines of code to set up the Frame Processor that detects faces and draws on the view

Plus it already works on both platforms, you have way more flexibility (third party FP Plugins on npm) and can draw more (entire Skia API is available for drawing stuff), and you write basic JS (Command + S to instantly see your changes appear on your device, no need to rebuild).

[todorone]

if VisionCamera will become Fabric only, it will reduce amount potential users drastically, say 10x

[mrousavy]

Making it Fabric only will make a lot of things way simpler for me, especially talking about the C++/native code buildscripts. For legacy architecture support it requires a bunch of code, making it harder for me to maintain VisionCamera - and VisionCamera is a huge repository that I maintain myself in my free time.

Also, all older versions of VisionCamera still work on the legacy architecture. :)

EDIT: VisionCamera V3 will support the oldarch (Paper) too, so not only Fabric! It will require RN 0.71+ tho :)

[todorone]

Making it Fabric only will make a lot of things way simpler for me, especially talking about the C++/native code buildscripts. For legacy architecture support it requires a bunch of code, making it harder for me to maintain VisionCamera - and VisionCamera is a huge repository that I maintain myself in my free time.

Also, all older versions of VisionCamera still work on the legacy architecture. :)

Sure, i totally understand that it's quite a burden to support both archs, thanx a lot for such an amazing library! I just wanted to mention, that it's already obvious that migration of real-world existing production apps to Fabric is the question of pretty far future w/o any intention to put request or pressure on You.🐱

[mrousavy]

I just wanted to mention, that it's already obvious that migration of real-world existing production apps to Fabric is the question of pretty far future w/o any intention to put request or pressure on You.

I disagree, a lot of our clients at Margelo are running Fabric in production. Facebook is too. We can help with the migration of your app if you want -> hello@margelo.io ;)

[patrickkabwe]

@mrousavy I really appreciate the effort you and the team at margelo.io are putting into this package. i think it would be great to also add barcode/qrcode scanning by default, because some of the community packages that have this implementation do not maintain them any more for example this.

Thanks

[kesha-antonov]

I just wanted to mention, that it's already obvious that migration of real-world existing production apps to Fabric is the question of pretty far future w/o any intention to put request or pressure on You.

I disagree, a lot of our clients at Margelo are running Fabric in production. Facebook is too. We can help with the migration of your app if you want -> hello@margelo.io ;)

Hi I guess many projects can't migrate to Fabric because they depend on libraries that aren't Fabric compatible yet

Here's a table of those libs https://github.com/reactwg/react-native-new-architecture/discussions/6

[mrousavy]

i think it would be great to also add barcode/qrcode scanning by default

I have received hundreds of requests like this seems like a much wanted feature haha... It definitely has to be a Frame Processor Plugin the way that VisionCamera is built. But how about this; if we manage to get enough partners/companies to sponsor the VisionCamera V3 project, I'll build a QR Code scanner plugin into VisionCamera or as a separate package maintained by me :)

[jonxssc]

Very nice of you to maintain this library, i am a big fan but from using this library i have some improvements that should also be looked at:

• Flash (When using the takesnapshot method the flash i think fires to early and the picture gets ruined)
• Zoom (It would be way easier to take a look at how expo camera did this by just adding a simple option true/false)
• Performance (The overall performance on Android is pretty bad, expo camera package is about 0,5 seconds faster)
• Documentation (The Documentation is not really god, to understand or even find much functions you have to take a look deep in the examples to understand most of it and see all functions/features)

To be clear, that is absolutely no hate and i am a big fan of this package and would love to use it in my apps, but for now i stay at expo camera, because it is faster and the flash is working properly. But i would love to switch, because expo camera has not much features and also has its problems like mirrored images on the front camera :)

[mrousavy]

Hi @jonxssc,

When using the takesnapshot method the flash i think fires to early and the picture gets ruined

takeSnapshot does not have a flash. It is literally just taking a snapshot of the view.

It would be way easier to take a look at how expo camera did this by just adding a simple option true/false

What? How does zoom={true} or zoom={false} make sense?

The overall performance on Android is pretty bad, expo camera package is about 0,5 seconds faster

The performance is pretty optimized, this is what VisionCamera is about. On iOS, it's really fast and optimized especially for streaming camera frames and initial launch time. What exactly is 0,5 seconds faster in expo-camera? Open time? Photo time? Can you create a repo to reproduce this? Remember that it's hard to compare Camera speeds if you're using a different format (e.g. if you use a higher-res camera device/format, it will obviously take longer to open the Camera).

The Documentation is not really god, to understand or even find much functions you have to take a look deep in the examples to understand most of it and see all functions/features

What functions do you mean specifically?

[Bamorem]

I agree for the QR code scanner

[amochkin]

Isn't RN SKia is still proof-of-concept?

[obasille]

Thank you for working on this library! It saved me a ton of work :) Looking forward to V3!

[mrousavy]

Thank you for working on this library! It saved me a ton of work :) Looking forward to V3!

Thank you! Happy to hear it helps :)

[MadsonChavante]

Thank you for working on this library! ... Native QR code scanner is most welcome ...

[jpike88]

Many opened issues against this project haven't been addressed, and I've had to fork because maintainers have been unresponsive/haven't followed up. It's hard to get excited about V3 when there's still outstanding issues with v2 which the above v3 plan doesn't address. I think addressing clear sticking points (even if it's a 'no, we aren't interested in including this in the project right now'), or at least moving issues that are marked 'question' to discussions to clean things up, would allow people like myself who are potential contributors to understand whether continuing to maintain our own forks is the only reasonable option, or if there is enough alignment between main project and our own goals/expectations to make the forks redundant.

[mrousavy]

I think addressing clear sticking points (even if it's a 'no, we aren't interested in including this in the project right now'), or at least moving issues that are marked 'question' to discussions to clean things up, would allow people like myself who are potential contributors to understand whether continuing to maintain our own forks is the only reasonable option, or if there is enough alignment between main project and our own goals/expectations to make the forks redundant.

@jpike88 I agree, issues should be handled individually and flagged appropriately - either a "wont-fix" (not on roadmap), or put into backlog and restructured. Spam issues should be closed, and duplicate ones should be linked / locked.

This is unfortunately really time consuming, and I already have very limited time. Cameras are a pretty complex topic, and I know my way around the VisionCamera codebase since I wrote it, so I think my time is best spent actually writing code - so that's what I do - when I have some free time I will work on the code and improve VisionCamera. E.g. today, I spent 5 hours already to work on V3 (see #1466).

Frankly, most issues are spam or annoying, and I don't have the nerves to process each of those. My library works in all the apps I use it in, and if something breaks for me, I fix it. Luckily for everybody else, I publish that code as open-source on GitHub (here), and if something doesn't work for you, submit a well written PR to fix it and I'll find some time to test that and integrate it if nothing breaks.

If someone wants to help me maintain VisionCamera, I think the best place to start is by triaging/flagging issues, answering questions, closing spam or duplicate issues, and discussing features/fixes with me internally. If anyone wants to do that, let me know :)

I am willing to help you guys use VisionCamera, but I can only do so much. If someone posts a screenshot of a build error and the screenshot just says "Build failed in 65 seconds", I will consider it rude to not even try to fill out the issue appropriately and won't reply myself.

Many opened issues against this project haven't been addressed

Which ones are those exactly? Again, I use VisionCamera in production in a few apps, and it successfully builds, starts, shows Camera at 30 or 60 FPS, takes photos, records videos, and runs a Frame Processor (private code). Both on iOS and Android.

[amochkin]

I agree with Marc, developers are becoming more and more demanding. It is an open source project, fork it if you don't like how things are handled here.

Lately I had another problem and had no time to fix it myself, so I hired a freelancer to help me. No big deal. I will PR soon.

@mrousavy As a side note, I think it would be better to create an open collective project so corporate users, like myself in certain cases, will be able to support the project financially.

[mrousavy]

I will PR soon.

❤️❤️❤️

@mrousavy As a side note, I think it would be better to create an open collective project so corporate users, like myself in certain cases, will be able to support the project financially.

Oh yea, interesting. I thought GitHub sponsors was more convenient and modern, but I'll take a look at open collective!

[amochkin]

I thought GitHub sponsors was more convenient and modern, but I'll take a look at open collective!

My thinking is Github Sponsors is more tailored for P2P support. While Open Collective is more like corporate level. I guess Open Collective is more demanding in terms of legal requirements because certain projects can be considered charity foundations and are tax exempt.

[zzz08900]

I agree with @mrousavy on the part that vision camera works fine on most production Apps. Yeah you might need some patch-package or some dependency version locking or other not-so-fancy workarounds but you got the camera library for free.

Most "issues" related to build fails or whatnot are not issues, those are counterparts of rookie C programmer hit by a SIGSEGV and trying to blame on the compilers/IDEs.

(By the way vision camera is the best library my team encounter in the RN world. My team couldn't make a camera library that's both very extensible and feature rich. Some people these days really need to learn to appreciate what they have, not demanding to be babysitted all day long.)

[mrousavy]

By the way vision camera is the best library my team encounter in the RN world. My team couldn't make a camera library that's both very extensible and feature rich. Some people these days really need to learn to appreciate what they have, not demanding to be babysitted all day long.

Much appreciated @zzz08900 ❤️❤️❤️❤️❤️❤️❤️

Yeah you might need some patch-package or some dependency version locking

Working on V3 right now, and it seems that this solves most of the build issues! :)

[zzz08900]

Well, I've noticed most of those "issues" with build errors occure after upgrading RN version so I would say many of them are not caused by vision camera or reanimated, but originates from a bad RN upgrading.

I had those problems before, like library X stops working after RN upgrade, so now I just ditch the whole android/iOS folders, bump dependencies and re-configure new RN version from base template - that always work.

I guess the above method should be in the RN official documentation.

[bglgwyng]

@mrousavy Thank you for this great project! It's hard to wait for the new RNVC. I read the commits in v3 branch, and there seem to be more updates than items checked in the to-do list here. Could you please update the to-do list? I want to test the new features of RNVC available now.

[mrousavy]

Yeah I figured a few more features that are cool to add in v3, I'll update todo list maybe today

[mrousavy]

I just updated the task list!

Added the two points:

• synchronous Frame Processors (which I just merged into feat: ✨ V3 ✨ #1466, woohoo!!! 🥳)
• the PyTorch Live integration which comes with the REA -> RN Worklets rewrite

I just released a pre-release for v3 here: v3.0.0-rc.1 You can test the latest V3 release by creating a new RN project with RN 0.71 and installing VisionCamera + RNWorklets:

yarn add react-native-vision-camera@3.0.0-rc.1
yarn add react-native-workets@https://github.com/chrfalch/react-native-worklets#15d52dd

Things to test:

• New Android build script. This should drastically speed up the build time! 💨
• New Worklet library. This replaces Reanimated Worklets. Should be faster and more stable :)
• New synchronous Frame Processors. Should be faster :)
• runAtTargetFps and runAsync in Frame Processors
• Using HostObjects or HostFunctions (like models from PyTorch) inside a Frame Processor. This will probably require a few native bindings on PyTorch's end to make the integration work (cc @raedle)

Overall V3 is close to completion. I have a few things to do the coming days so not sure how much work I can put into this. If anyone wants to support the development of v3, I'd appreciate donations / sponsors: https://github.com/sponsors/mrousavy ❤️ :)

[migueldaipre]

I tried create a new RN project with RN 71 and add the dependencies but it doesn't build.

See

Example folder works fine.

I just updated the task list!

Added the two points:

• synchronous Frame Processors (which I just merged into feat: ✨ V3 ✨ #1466, woohoo!!! 🥳)
• the PyTorch Live integration which comes with the REA -> RN Worklets rewrite

I just released a pre-release for v3 here: v3.0.0-rc.1 You can test the latest V3 release by creating a new RN project with RN 0.71 and installing VisionCamera + RNWorklets:

yarn add react-native-vision-camera@3.0.0-rc.1
yarn add react-native-workets@https://github.com/chrfalch/react-native-worklets#15d52dd

Things to test:

• New Android build script. This should drastically speed up the build time! 💨
• New Worklet library. This replaces Reanimated Worklets. Should be faster and more stable :)
• New synchronous Frame Processors. Should be faster :)
• runAtTargetFps and runAsync in Frame Processors
• Using HostObjects or HostFunctions (like models from PyTorch) inside a Frame Processor. This will probably require a few native bindings on PyTorch's end to make the integration work (cc @raedle)

Overall V3 is close to completion. I have a few things to do the coming days so not sure how much work I can put into this. If anyone wants to support the development of v3, I'd appreciate donations / sponsors: https://github.com/sponsors/mrousavy ❤️ :)

[mrousavy]

I had those problems before, like library X stops working after RN upgrade, so now I just ditch the whole android/iOS folders, bump dependencies and re-configure new RN version from base template - that always work.

@zzz08900 I agree, that's a good approach to upgrading. I always do that. :)

[mrousavy]

@migueldaipre please don't post issues here, only high-level feedback. Thanks. I'll look into this later though

[kesha-antonov]

Well, I've noticed most of those "issues" with build errors occure after upgrading RN version so I would say many of them are not caused by vision camera or reanimated, but originates from a bad RN upgrading.

I had those problems before, like library X stops working after RN upgrade, so now I just ditch the whole android/iOS folders, bump dependencies and re-configure new RN version from base template - that always work.

I guess the above method should be in the RN official documentation.

No need to. They have upgrade tool where you can look at all necessary changes that you need to do.

https://react-native-community.github.io/upgrade-helper/

[bglgwyng]

I got some questions and ideas on the new frame processors. They are a bit long so I made a new discussion here. https://github.com/mrousavy/react-native-vision-camera/discussions/1481

[bctt]

I just released a pre-release for v3 here: v3.0.0-rc.1 You can test the latest V3 release by creating a new RN project with RN 0.71 and installing VisionCamera + RNWorklets:

Will V3 require RN 0.71?

[mrousavy]

Will V3 require RN 0.71?

Yes. Due to the significant simplification of the buildscript, I fixed a bunch of build errors and made the whole building process much more robust. I had to drop backwards compatibility for this, otherwise this would've gotten too complicated.

However I managed to not rely on the new arch, so it works on both old and new arch! (Paper and Fabric)

[mrousavy]

Another surprise for v3 - I added three new props and one new function to the Frame object:

• toByteArray(): Gets the Frame data as a byte array. The type is Uint8Array (TypedArray/ArrayBuffer). Keep in mind that Frame buffers are usually allocated on the GPU, so this comes with a performance cost of a GPU -> CPU copy operation. I've optimized it a bit to run pretty fast :)
• orientation: The orientation of the Frame. e.g. "portrait"
• isMirrored: Whether the Frame is mirrored (eg in selfie cams)
• timestamp: The presentation timestamp of the Frame

see https://github.com/mrousavy/react-native-vision-camera/pull/1487

[AlkanV]

hello @mrousavy, i really like what you are doing here and i am fig fan of the project. couple of months ago, just before you released road map for v3, i have give it a go over rn-camera to see if we can integrate it to our project or not. since this library produce big media files in android, we parked the integration process. this was due to cameraX integration over camera2, where we are not able to 'really' modify bitrates, instead we can only give some preset props, which was not working as we expected. i got notification that there is a RC version of v3, and i immediately check if you re-write the camera2 module, unfortunately not :(. do you think the above implementation #1487 , would fix this problem on android?

best wishes, a.

[mrousavy]

i really like what you are doing here and i am fig fan of the project.

Thank you! :)

do you think the above implementation https://github.com/mrousavy/react-native-vision-camera/pull/1487 , would fix this problem on android?

No, this has nothing to do with media writing. This exposes new props to the Frame Processor.

i got notification that there is a RC version of v3, and i immediately check if you re-write the camera2 module, unfortunately not :(

Read the things I have planned for V3 at the top of this issue:

4. ...
5. Rewrite the native Android part from CameraX to Camera2
6. ...

I have it planned, but I think this will be a big project. This will take like 100 to 300 hours worth of effort, and frankly I don't have that time right now unless I decide to become a cocaine addict.

So I'm thinking to release V3 as a kind of beta and just have some features not work on Android, and if people get together to raise some money for getting those features implemented on Android as well I can more freely allocate that time through my agency (Margelo) to work on this.

[faroutchris]

i think it would be great to also add barcode/qrcode scanning by default

I have received hundreds of requests like this seems like a much wanted feature haha... It definitely has to be a Frame Processor Plugin the way that VisionCamera is built. But how about this; if we manage to get enough partners/companies to sponsor the VisionCamera V3 project, I'll build a QR Code scanner plugin into VisionCamera or as a separate package maintained by me :)

Or offer it as a paid tier for enterprise usage. OSS for personal usage. The ones I work for don't understand open source, they figure it's just free. They understand license fees though.

[jpike88]

We use this for our Cordova apps, works like a charm https://github.com/phonegap/phonegap-plugin-barcodescanner

[Vednus]

Just an FYI. You need to pin skia at 0.1.175 for the rc2 to work. I was getting the error 'include/gpu/GrRecordingContext.h' file not found when trying to build on XCode with 0.1.176.

[bglgwyng]

I suppose that VisionCamerea V3 records the video just as shown on the screen. This means, a drawback frame processor also affects the video recording.

Sometimes those need to differ, the preview on the screen and the video to record. For example, I want to watermark the video to record, but I don't like the watermark to be shown on the screen.. If the video processing should be done on the original video after the user finishes the recording, it would be quite inconvenient for the user who needs to wait until the saving is done. It's ok with the photo, we'll be able to manipulate the photo before saving it thanks to the new API of V3. And the photo processing is cheap enough not to bother the user. So I think it would be better to have a way to explicitly pass back the frame to be save to the video.

Is it achievable? There might be API design issues and we need time to think about it. Before that, I'd like to know if it is possible with the current internal of VisionCamera V3.

[mrousavy]

Hi @bglgwyng, interesting point. Let me quickly explain:

Cameras work by combining a set of inputs to a set of outputs.

For example, those are the inputs that we have:

• Video Input (a Camera device)
• Audio Input (a Microphone)

And those are the outputs:

• Preview View
• Photo Capture
• Record Video
• Frame Processor

This is too much for a Camera to handle at the same time, so we do some tricks here.

On Android, we can skip the Photo Capture and do a snapshot/screenshot of the view instead, so that we only have three outputs (Preview, Video, Frame Processor, but no Photo Capture). On iOS, I combine Video Recording and Frame Processing into one, so that I get one Frame, which runs the Frame Processing algorithm, and then records it to a file if there is an active recording.

In other words; I don't have two pipelines for streaming frames here. I might be able to restructure this a bit, especially with the Skia Preview (maybe we can combine FP + Preview instead?)

Anyways, right now in the current V3 version, you can only draw to the Preview View, not to a Recorded Video or Photo. That still has to be done in post-processing. I want to implement that however, but no one is paying me to do so, so I just wait until I have the free time. If you want to accelerate this, consider sponsoring me on GitHub, or reaching out to me for a more corporate deal thingy

[xts-bit]

@mrousavy When Vision Camera V3 wil Launch?

[mrousavy]

I don't have a timeline for this. I work on this in my freetime, so I'm not setting any deadlines, I work on this whenever I want to and whenever I have some free time - if you want to accelerate this, consider sponsoring me on GitHub.

With that being said, idk, could be 1 month could be 3 months. Still a fair bit of work to do, we're talking about ~500 hours of effort here just for a V3 version

[xts-bit]

@mrousavy What is that 35 lines of code to set up the Face Detector Frame Processor Plugin? will it be prebuilt in camera?

[mrousavy]

will it be prebuilt in camera?

No, VisionCamera's focus is to be as lean as possible. No opinionated frame processor code, so the user can tweak it however he wants to.

Prebuilding stuff into the Camera just makes it bloated and messy, that's why I keep it modular/plugin-based.

[idrissakhi]

hello,

thank you for the effort and time invested in this lib. i just tested the rc-2 on android the CameraPackage is coming from the example package => // react-native-vision-camera import com.mrousavy.camera.example.CameraPackage; i think there is a small typo in generation scripts

[mrousavy]

Hi yep I fixed this in one of the latest commits, I didn't publish a new version yet. Will work on Android first, then publish a new prerelease.

[xts-bit]

@mrousavy Do VisionCamera V3 will reduce its size? Its 500MB increases react native app size by a lot beside that its great!

[mrousavy]

What? What's 500MB?

[Bayramito]

is this still valid ? because cannot resolve the react-native-worklets from the source