Unclear if the API allows workers to be generated from strings

[rileyjshaw]

TL;DR: Using raw workers, I can generate a Blob from a string. Is there any affordance for this in operative's API? Can I alias the callback to a function that's called in the string?

The reason: I'm writing a tool that needs to evaluate user-generated code. It's a programming challenge, so when a user thinks they've got the correct result they call verify(result) and see if it passes.

An example of a simple challenge: Calculate 1+1, then pass the result into the verify function.

verify is defined by me behind the scenes as function (n) { return n == 2; }. After reading the challenge, the user might type something like this into a textarea and hit run.

var result = 1 + 1;
verify(result);

Thoughts so far: It seems like this should work well with the operative API... I've got a chunk of code to run, and a single callback to send back to the main thread. I'm just not sure how to deal with stringified user input. One possible solution is:

var myOperative = operative(function (__code__, verify) {
  eval(__code__);
});

...but I was hoping to avoid using eval like that. Any thoughts on how best to handle this?

Thanks for all of your hard work on this library!!

[padolsey]

Blobs give you a tad more debuggability, but beyond that, not much. Also, to get that debuggability, you'd have to form the worker from the blob itself ... so, a brand new operative for every piece of user code. Would that work for you? I'd like to experiment with this a bit, but was wondering if there's something specific about blobs that makes you want to use them here? Eval seems bad, but that's what's effectively going to be happening either way.

[rileyjshaw]

... so, a brand new operative for every piece of user code. Would that work for you?

It would certainly work, since there's no state held between tests (i.e. whenever run is hit, we're starting from scratch).

...wondering if there's something specific about blobs that makes you want to use them here? Eval seems bad, but that's what's effectively going to be happening either way.

You're right; the difference between Blobs and eval in this case is minimal. I was just wondering if Blob string support was baked-in, because if so I'd use it instead :)

[rileyjshaw]

Slight tangent: Is it possible for an operative's callback to accept function arguments? For example, if I have the following prompt:

Write a function that returns the number two and pass it into the verify function.

And the following verify method:

function verify (fn) { return fn() === 2; }

Running the following will return an error:

var worker = operative(function (__code__, verify) {
  eval(__code__);
});

worker('verify(function () { return 2; });', verify);
// Uncaught DataCloneError: Failed to execute 'postMessage' on 'WorkerGlobalScope': An object could not be cloned.

In this toy example, we could just run verify(fn())... but in more complex examples, passing a function back is very useful. Just wondering if you've dealt with this in the past.

[padolsey]

There is no way to pass a function in either direction. If there was, operative would just be toString'ing it and then eval'ing it on the other side. This is generally a limitation of workers, and makes sense since a function is never a self-contained thing, but has ties to its execution context and scope, etc.

[dulichan]

@padolsey is it possible to pass Float32 array buffers between the web worker and the main thread. I want to run a merge buffer operation for Float32 arrays.

[padolsey]

@dulichan Have a look at https://github.com/padolsey/operative/wiki/Transfers-API -- you should be able to pass through the buffer directly (though do note that you'll be transferring ownership of it). See https://developer.mozilla.org/en/docs/Web/API/Worker/postMessage (specifically the transferList argument), to see how it works.

[dulichan]

@padolsey thank you for pointing me in that direction. What I want to do is to transfer an array of Float32Arrays. Does the spec support away to do this?

[dulichan]

Basically I am trying to run the below function -

var mergeBuffer = operative(function(obj, callback) {
      const result = new Float64Array(obj.length);
      let offset = 0;
      var buffer = obj.buffer;
      for(let i = 0; i < buffer.length; i++) {
        const inner = buffer[i];
        result.set(inner, offset);
        offset += inner.length;
      }
      callback.transfer(result.buffer, [result.buffer]);
    });

What I have as inputs is an array of Float32 Arrays. If I try below invocation it fails with Failed to execute 'postMessage' on 'Worker': Value at index 0 does not have a transferable type.. This is mainly because buffers[0] is a plain old array. Any solution that you can think of to enable this transfer?

mergeBuffer.transfer(buffers[0], buffers[0], function(result) {
      console.log(result);
    });

[padolsey]

Try:

mergeBuffer.transfer(buffers[0], [buffers[0]], ...);

(The second argument to transfer() must be an Array of Transferables.)

If this still doesn't work I suggest opening a separate issue as this is technically not related to the issue thread we've been speaking on.