Time to start up kevoree on PI and Edison

[kehusa]

Starting my fairly simple kevoree js node on a Raspberry Pi Model B+ 700MHz took me exactly 4 minutes and 20 seconds !!! And that was not the first time when compiling and downloading libraries which made it take even longer time.

On Edison it took 1 minute and 20 seconds, less time but still..

What are your thoughts about this? Is node well suited to run on devices like Edison and PI or is it our framework which makes it so slow? Edison and PI are not a PC, but still quite powerful devices.

[maxleiko]

I think I could improve the start-up time in general. Didn't put that much thoughts in it for now because I'm running on a killer laptop machine, but we definitely need to look into this.

[barais]

Is it for nodeJs node ?

2015-03-13 14:08 GMT+01:00 Maxime Tricoire notifications@github.com:

I think I could improve the start-up time in general. Didn't put that much thoughts in it for now because I'm running on a killer laptop machine, but we definitely need to look into this.

— Reply to this email directly or view it on GitHub https://github.com/HEADS-project/forum/issues/11#issuecomment-78963252.

[kehusa]

This is the nodejs on PI: http://node-arm.herokuapp.com/node_latest_armhf.deb This is the nodejs on Edison: aptitude -t wheezy-backports install nodejs-dev

[brice-morin]

Also we should try to identify if the performances of node are ok once everything has been deployed. One alternative is of course to go directly in C/C++, as would recomment @ffleurey, but we would have a less flexible support with Kevoree (though something like the "sintefboard" could be achieved.

[kehusa]

Doing as @maxleiko proposed: starting grunt with option --no-reinstall reduced the startup time to: Intel Edison: 36 seconds PI: 1 minute 45 seconds ¨ :-), but can we perform even better?

[maxleiko]

Using the latest grunt-kevoree task runner coupled with some modifications in your Gruntfile you could save some time.

You need to update your grunt-kevoree module to 5.4.3 and modify your build task to:

    grunt.registerTask('build', 'Build Kevoree module', function () {
        if (process.env.KEVOREE_RUNTIME !== 'dev') {
            grunt.tasks([
                'kevoree_genmodel'
            ]);
        }
    });

Note that, for browser-ready component/channel/group/node you will also need to add the browser task to the array:

    grunt.registerTask('build', 'Build Kevoree module', function () {
        if (process.env.KEVOREE_RUNTIME !== 'dev') {
            grunt.tasks([
                'kevoree_genmodel',
                'browser'
            ]);
        }
    });

This trick will prevent npm to run the build task for the current module while it is installed by the runtime (which was useless in the end, and could be time consuming, specially if you have to browserify/uglify your module)

[maxleiko]

Another great way to save time is to always use the same directory for your deploy units installations.

Using -p /a/path for the kevoreejs command-line client, or the grunt task initConfig options for grunt-kevoree:

        kevoree: {
            options: {
                modulesPath: '/a/path'
            }
        }

[kehusa]

@maxleiko When starting kevoree with the following command: kevoreejs -p . -n node0 --kevscript main.kevs it takes a lot of time. It seems like the node has downloaded about 42 MB to get started. However, this happened the first time I started the node after not restarting it for several days. Next time it went much faster and it did not download much data. Is there a way to make the node not reload all the components and rebuild?

[maxleiko]

Did you checked before if there was a node_modules/ directory with the Node.js modules inside ? Because this is what npm will check and then decide if it needs to install or not.
Maybe there is something like a "timeout" on that. I will dig into it :)

[brice-morin]

I have made some benchmarks using the HEADS weather station:

• Standalone JS code generated by ThingML:
  • node_modules: 526 KB (+ any other NPM lib you would use in your ThingML code)
  • RAM: ~20 MB
  • CPU: ~0
• In Kevoree (wrapping the above ThingML code):
  • node_modules: 56.9 MB (roughly x100 overhead)
  • RAM: ~300MB (roughly x15 overhead)
  • CPU: ~0

Regarding the node_modules, it has been discussed.

Regarding the RAM, a x15 (or +280 MB) overhead seems quite heavy. I understand the dynamicity brought by KevoreeJS needs RAM, but the price to pay is quite high.... I did the experiments on my PC, but if we assume they are the same on Edison, a simple Weather station would consume 33% of the RAM there...

@maxleiko any idea why you need so much RAM? Is that needed or more like accidental?

[maxleiko]

Did you benchmark in dev mode or not ? I think there are significant changes between a classic prod node_modules and the dev node_modules.

For the RAM consumption I didn't investigate that much the issues. I will though as I really want the runtime to start faster and take less RAM.
I will keep you updated

[maxleiko]

@brice-morin Looks like npm is the trouble maker for the RAM consumption. Each time Kevoree calls npm.install to add a DeployUnit dynamically, the heapUsed grows (~20mo each time). Apparently npm keep the whole dependency tree in memory.
I will still dig into that, but it looks like we are stuck with that problem unless npm changes the way they handle module installation or we change the way we install DeployUnit (which, I think, is not a good idea because we would have to re-create a npm-like thing)

[barais]

10 Mb for the dependency tree is a bit huge. Isn't it ?

2015-07-23 9:43 GMT+02:00 Maxime Tricoire notifications@github.com:

Looks like npm is the trouble maker for the RAM consumption. Each time Kevoree calls npm.install to add a DeployUnit dynamically, the grows (~20mo each time). Apparently npm keep the whole dependency tree in memory.

I will still dig into that, but it looks like we are stuck with that problem unless npm changes the way they handle module installation or we change the way we install DeployUnit (which, I think, is not a good idea because we would have to re-create a npm-like thing)

— Reply to this email directly or view it on GitHub https://github.com/HEADS-project/forum/issues/11#issuecomment-124007623.

[maxleiko]

╭─leiko@kevtop /tmp/test-npm  
╰─➤  node app.js 
without npm

======
mem usage:
  rss          13mo
  heapTotal    9mo
  heapUsed     3mo
=======

now loading NPM
npm loaded

======
mem usage:
  rss          35mo
  heapTotal    30mo
  heapUsed     15mo
=======

now installing: kevoree-comp-consoleprinter
kevoree-comp-consoleprinter@5.2.9 node_modules/kevoree-comp-consoleprinter
└── kevoree-entities@8.0.5 (pseudoclass@1.0.3, kevoree-commons@2.1.2, kevoree-library@5.0.9, kevoree-kevscript@2.2.3)
kevoree-comp-consoleprinter installed

======
mem usage:
  rss          97mo
  heapTotal    72mo
  heapUsed     40mo
=======

This is the output. If you keep on installing module, you will see the heap growing and growing.
I might "resolve" the memory comsumption by installing the DeployUnits in a child process, but this is pretty ugly, don't you think ?

[maxleiko]

Okay, so my guess was ok-ish
https://github.com/npm/npm/issues/9042#issuecomment-124432920

[kehusa]

Got Raspberry Pi 2 today and tested the HomeGW on it. Startup time went down from 2 minutes (Raspberry PI B+) to 20 seconds :-)