Question: Implemeting cpu-bound LDA algorithm with Web Worker

[loretoparisi]

Hi, I'm running a LDA algorithm in JavaScript with a given number of documents, that will result in a cpu intensive task when the number of tokens (words) for a document is high. This will block the CPU and the VM, so when running on a web server like express, everything will be stuck for some time. The LDA algorithm I'm running (you can look at it here) does not need any external module, so according to your docs it could use this web workers implementation since it is using native threads. Do you think this is a good use case to use node-webworker-threads rathen then tiny-worker that is using child_process instead?

Thank you.

[davisjam]

This seems like a good use case. You should get less overhead and more portability (across server/client) with node-webworker-threads than with tiny_worker.

[loretoparisi]

@davisjam thanks. Supposed to need to import external libraries in the task, can I use node-webworker-threads? Let's say I have to do some require in the Thread code...

[davisjam]

@loretoparisi Does #173 explain this for you?

[loretoparisi]

@davisjam ok this means that it works like phantomjs execution block i.e. the code within the worker is running isolated and you cannot import external modules for that reason. This means that for numerical execution I can get rid of it, but if I need external resources (like an access to redis, etc.) I need to serialize/deserialize data via postMessage, on('message') worker api.

When using WebKit web workers I can always use the script import via the importScript api like in this simple worker that do numeric calculations on a csv file via d3.js api:

importScripts('https://cdnjs.cloudflare.com/ajax/libs/d3/4.13.0/d3.min.js');
var readFiles = function (files,whendone) {
    // do some magic using D3.JS
}
self.addEventListener('message', function (e) {
    var files = e.data;
    readFiles(files, function (datasetsList) { // done
        postMessage(datasetsList);
    });
}, false);