tdurand
commented
3 years ago Update 🔥🙌🎉 !!
First POC working !!! yay !!! Was able to run OpenDataCam on my android !!
I kept exploring the "mobile node.js" road the past days and here are some more updated thoughts:
Upsides
- Be able to reuse code from the main github.com/opendatacam/opendatacam .. To really evaluate if this will be a win for future maintenance.. The other option I am considering for now is to have all the code from node.js being ported to the client side and having next.js export a static app.. which can be done also in a way we can maintain a common part from what I see... As anyways if we run node.js on the mobile we also need to diverge from the main opendatacam code / build adapters for Mobile ( not launching darknet.. not using mongodb, etc etc )
After putting more thoughs (& action) on this.. with going the node.js way on the Android app, we should be able to have just one repository 😁 . And have build task that creates the "darknet" version, the "android" version.. which is not only only appealing to ease the future maintenance but making this step for "android" will be a big step towards the vision of having OpenDataCam beeing able to hook to anything (hardware and software) that is able to give us a video feed, run a neural network framework and run node.js : Raspberry + DepthAI, Jetsons + Deepstream, iOS + NCNN, Android + NCNN, Darknet + Jetsons, NCNN + Jetsons, Tensorflow + any hardware, Pytorch + any hardware.... etc etc..
- Be faster to deliver the app ... as the first version deadline is a bit time sensitive. This is not really a clear upside.. I think it is the same amount of work to port the node.js code to the client only as to adapt the node.js code to work on mobile...
turns out it was "fast" (or not, we will never know for sure until I don't do the client side OpenDataCam version :-) ..)
-
Persistence layer: won't run mongodb on android (or may I ? 🤷♂️😁), need to do something else that can be done on node.js mobile
-
the coordinates of the box are still a bit off.. need to work this out.. some magic video resolution and aspect ratio debugging to do here
-
the canvas drawing for the counting lines have some issue.. need to debug
-
For remote operating.. for now we don't "forward" the camera view into node.js via a MJPEG stream.. we just use the trick of rendering the camera preview behind the html webview .. not prioritary for first release
- Use the Mobile device with the opendatacam app as a "server" like we do with Jetson... This is big feature I didn't think about at first.. but if we can run the node.js server on Android.. then we can connect to OpenDataCam from another device that is on the same network and operate OpenDataCam remotely like we do now. This enables more use cases..
Keep the ability to operate OpenDataCam remotely with the HTTP APIs (using android phone "headless" like a jetson) is a huge feature.. didn't fully valide this yet.. We might need to open a firewall port on Android to make this work.. but it should + the mjpeg stream needs to be figured out.
Updated Drawback
More CPUs consumption.. node.js running means less power for other things.. not sure if this affects the YOLO performances..
this does not seem to be the case
- Adds 50 MB to the app size
still true ! but not problematic IMO
At app install and for each subsequent update.. we need to copy the app assets so node mobile can access them: .. and from first tests it seems it is slow as we are copying all the node_module folder.. like 30s at least right now.. but I'm sure this can be optimized ..
Next.js is not really made for a node-mobile deployment.. so right now need to hardcode the path where the files are copied to start : https://github.com/opendatacam/opendatacam-mobile/blob/acba05fadc6851e7501f188ae30423de3f84bdaf/android/app/src/main/assets/nodejs-project/server.js#L37 , that was hard to figure out and I'm not sure if this is super portable.. would need more work..
Right now it takes 1 minute 40 seconds to start the app after an update or on first install.. (otherwise, just 5 second after subsequent opening).. I didn't work on this but I'm almost sure this copy can be optimized a lot by reducing to the absolute minimum the node_modules dependency we ship and maybe pack them together.. And even if it's not possible.. we need to be creative on the loading screen, that's all !
More complexity.. more problems.. less dependency is always best ;-)
Yeah.. sorry 🤷♂️ ! in this case the "less is more" isn't doing it.. But it is very true that we need to test is long term reliability is there (like running for several hours).. a "critical" piece of software we rely on is "nodejs-mobile" https://github.com/JaneaSystems/nodejs-mobile , which is tagged 0.3.2 . This very much fits into our "OpenDataCam is very alpha and we do not provide any guarantee that this will work for your use case, but we conceived it as a starting point from where you can build-on & improve." motto
If a Web version (client side) of OpenDataCam with neural network running via WebAssembly is something we want to do at some point.. already working on a client side only version of OpenDataCam would be a big step forward this
💔 ! Will do it anyways one day I know 😂
Developer experience
need to investigate better, but it is obviously harder.. we need to precompile the node.js app before copying it on android.. and we can't set debug points when running in android.. (or I don't know how to do it yet, we might be able to run the node.js server on the android in development mode and get a nice experience).. If it was just a Webview we could debug "as usual" using chrome remote devtool . + right now every time I touch the code I have to wait 2 min the app starts
b-g
This started as a casual question from @b-g .. "are you gonna run node.js" on android.. which I answered.. "no no, will not, don't think it is possible..."
And yet here I am 😂.. Exploring the https://github.com/JaneaSystems/nodejs-mobile framework: https://code.janeasystems.com/nodejs-mobile
Turns out you can run Node.js on mobile .. I'm evaluating it on the
node-mobilebranch : https://github.com/opendatacam/opendatacam-mobile/tree/node-mobileAfter battling a lot to integrate it along the NCNN framework.. I think I spent 6 hours on obscure and frustrating CMake builds errors yesterday ( it is also a C++ dependency).. I got a first version running. (not OpenDataCam yet.. but nodejs)
The idea behind this is to:
Be able to reuse code from the main github.com/opendatacam/opendatacam .. To really evaluate if this will be a win for future maintenance.. The other option I am considering for now is to have all the code from node.js being ported to the client side and having next.js export a static app.. which can be done also in a way we can maintain a common part from what I see... As anyways if we run node.js on the mobile we also need to diverge from the main opendatacam code / build adapters for Mobile ( not launching darknet.. not using mongodb, etc etc )
Use the Mobile device with the opendatacam app as a "server" like we do with Jetson... This is big feature I didn't think about at first.. but if we can run the node.js server on Android.. then we can connect to OpenDataCam from another device that is on the same network and operate OpenDataCam remotely like we do now. This enables more use cases..
Be faster to deliver the app ... as the first version deadline is a bit time sensitive. This is not really a clear upside.. I think it is the same amount of work to port the node.js code to the client only as to adapt the node.js code to work on mobile...
Potential/ Confirmed Drawbacks (for now)
Adds 50 MB to the app size .. Not that problematic I think as NCNN and yolo weights are at least 50 MB already.. so the app won't be lightweight anyways..
At app install and for each subsequent update.. we need to copy the app assets so node mobile can access them: .. and from first tests it seems it is slow as we are copying all the node_module folder.. like 30s at least right now.. but I'm sure this can be optimized ..
Next.js is not really made for a node-mobile deployment.. so right now need to hardcode the path where the files are copied to start : https://github.com/opendatacam/opendatacam-mobile/blob/acba05fadc6851e7501f188ae30423de3f84bdaf/android/app/src/main/assets/nodejs-project/server.js#L37 , that was hard to figure out and I'm not sure if this is super portable.. would need more work..
More CPUs consumption.. node.js running means less power for other things.. not sure if this affects the YOLO performances..
If a Web version (client side) of OpenDataCam with neural network running via WebAssembly is something we want to do at some point.. already working on a client side only version of OpenDataCam would be a big step forward this
More complexity.. more problems.. less dependency is always best ;-)