Hive Computing

[harshkhandeparkar]

gpujs-hive-compute is a tool that takes a gpu.js kernel (js code run on a gpu, we use it in some of the modules, eg: blur) and distribute the work over multiple GPUs of multiple devices. All we need is a server to relay the communication between two devices.

This can be used for: 1) IS hives 2) This thing can connect IS to a server like nobody's business. 3) It can run on ANY device. A phone (via browser), a nodejs headless server, another PC, ANYTHING. Any OS, any platform. This means that IS which earlier would just crash with larger images can connect to a server or other devices to reduce the load and each device will use less memory, processing power, it can effectively half, 1/3, 1/4... the computation time. 4) image-sequencer-app can benefit from this. 5) It is possible (and I have tried and succeeded) to convert pixel-manipulation which basically the core of IS, to use gpu.js which means that entire IS can run on a hive.

Caveats:

• It needs a server, there is no way two browsers can directly communicate but once there is a server, there is no limit to how many devices can connect.
• It is a bit network intensive, read more

---

Thank you!

Your help makes Public Lab better! We deeply appreciate your helping refine and improve this site.

To learn how to write really great issues, which increases the chances they'll be resolved, see:

https://publiclab.org/wiki/developers#Contributing+for+non-coders

[harshkhandeparkar]

cc @jywarren

[harshkhandeparkar]

I experimented a bit on this. I found that image sequencer is meant to be synchronous but hive computing works on async promises. We will have to completely change the extraManipulation method and make it async or find out another way...

[harshkhandeparkar]

There is an easy way to do that and that is to create a new callback similar to extraManipulation or pixelManipulation. I tried this locally but unfortunately, it doesn't seem to work :(

[jywarren]

hmmmm i like the idea -- we've tried to centralize and standardize module code to make optimizations like this possible... but i understand it may not be easy on the first try. Thanks for giving this a go, Harsh!

[harshkhandeparkar]

Would it be useful if IS becomes asynchronous?

[jywarren]

I'm not sure - i guess it would still end up synchronous on the level of module-to-module connection, wouldn't it? So, within a module, it seems like there could be substantial potential. This is one example - what other uses might it have? Parallel processing?

On Fri, Oct 30, 2020 at 2:22 PM Harsh Khandeparkar notifications@github.com wrote:

Would it be useful if IS becomes asynchronous?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/1716#issuecomment-719720829, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAF6J6HGQAEQFUVW5CIILLSNL76XANCNFSM4RNTPIJA .

[harshkhandeparkar]

Parallel, yes! Other than that, performance benefits. We can let a module run in a web worker for hours without affecting the main window. Show a spinner until then, and once the output is ready and the promise resolves, display the output.

[jywarren]

OK, maybe it's worth opening a new issue to discuss. Things like, are there modules that /wouldn't/ work asynchronously, is there a graceful fallback, is this something like gpu acceleration where we could mark asynchronous-friendly modules, which files would need to be modified, would older modules need an upgrade to work, any breaking changes... stuff like that? Thanks Harsh!

On Fri, Oct 30, 2020 at 4:14 PM Harsh Khandeparkar notifications@github.com wrote:

Parallel, yes! Other than that, performance benefits. We can let a module run in a web worker for hours without affecting the main window. Show a spinner until then, and once the output is ready and the promise resolves, display the output.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/1716#issuecomment-719774996, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAF6J5H5R6BQ42UISIHMXDSNMNDZANCNFSM4RNTPIJA .

[harshkhandeparkar]

Everything synchronous can be async, async can't be sync.

[jywarren]

Maybe if we tackled this in chunks it's be best- what if we started by breaking out an issue to try running pixel manipulation on gpu.js, which itself would be useful?

Then we could try to make it so you can press the gpu.js icon in any module to generate a link to share with other devices, which would make a hive?

[harshkhandeparkar]

A hive needs some form of a server. To allow for communication between devices. Have you read the docs of gpujs-hive-compute? (Not too long) It will give you a better insight...

[jywarren]

Ah I will... Does it even require them if using peer.js style webrtc peerConnection? That can do serverless and even NAT traversal if u use a TURN server. Cool!

On Fri, Nov 27, 2020, 11:02 AM Harsh Khandeparkar notifications@github.com wrote:

A hive needs some form of a server. To allow for communication between devices. Have you read the docs of gpujs-hive-compute? (Not too long) It will give you a better insight...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/1716#issuecomment-734898107, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAF6J3UJN6KGWXCEWL4FQ3SR7EQTANCNFSM4RNTPIJA .

[harshkhandeparkar]

I have no idea what the last two terms mean... lol.

[harshkhandeparkar]

The only required thing is communication between the two devices. Since I couldn't find an (easy to use) protocol that would allow for direct communication between two browsers. The only thing I can think of is a WebSocket relay server.

I was thinking of creating a special package for it anyway... Instead of just relaying the between two devices, it will only assign them tasks, get the data from each device, generate the final output and send it back.

[harshkhandeparkar]

Currently how it works: Helper 1 <-> Leader <-> Helper 2 1) Leader will tell each helper what to do. Each helper will render a piece of the final image. 2) The helpers send their respective outputs back to the leader. 3) Leader also calculates a part of it. 4) Leader combines the outputs to generate one mega output.

What I want to do: I want to add an option to gpujs-hive-compute that skips step 3. Will that be useful?

[jywarren]

I guess i wonder if it's adaptable to a model where the initiating sequencer is the host (formerly the server) and anyone who connects to it is a helper peer, using a library like this: https://rtcmulticonnection.herokuapp.com/demos/

On Fri, Nov 27, 2020 at 12:46 PM Harsh Khandeparkar < notifications@github.com> wrote:

Currently how it works: Helper 1 <-> Leader <-> Helper 2

1. Leader will tell each helper what to do. Each helper will render a piece of the final image.
2. The helpers send their respective outputs back to the leader.
3. Leader also calculates a part of it.
4. Leader combines the outputs to generate one mega output.

What I want to do: I want to add an option to gpujs-hive-compute that skips step 3. Will that be useful?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/publiclab/image-sequencer/issues/1716#issuecomment-734935802, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAF6J5Z5RCGWXEHK3Q5W2LSR7QZBANCNFSM4RNTPIJA .

[harshkhandeparkar]

We'll see... Currently, async doesn't work very well with IS. Is it possible to block the js thread to make async sync? (But ig that is not desirable)

[endurance21]

@jywarren @HarshKhandeparkar Hey !

I came across this and found it amusing, so as far the the concern of making async action to sync , I saw something relevant https://github.com/tc39/proposal-top-level-await , This top level await thing is awesome. I would like to hear your views !

[harshkhandeparkar]

We still don't use a lot of ES6 features due to browser compatibility. That is ES10! We might have to use babel for that.

Although async...await is very widely supported, just not top-level await.

[endurance21]

@harshithpabbati though i am not very sure but It appears we are using web assembly and that for a reasons a way faster , also gpu.js requires gpu which is absent in most of devices we gonna deliver our apis.

my second thoughts are on using python image processing capacity which would higher than any js script processing as it supports parallel processing in action. In particular i am taking about Open CV and we can use Rest APIS to connect the frontend with backend using flask or Django?

I am just curious about this topic and it would be nice to see some amazing conclusion over this!

[harshkhandeparkar]

1. Parallel processing requires GPU no matter whether it is python or js
2. We use gpu.js anyway, and it can fall back to CPU mode.
3. I am not sure about the REST APi though.

[harshkhandeparkar]

What kind of image processing do you need for your API?

[endurance21]

Particularly I don't need any of them , I was saying we can increase speed of image processing if we use python on backend and send those results to frontend using REST APIS.

[endurance21]

However , I think that would be too much and it seems too much away from current implementation that is done in javascript.😬

[harshkhandeparkar]

If you want to do something quickly, you can use OpenCV.js although that library is massive.

[harshkhandeparkar]

Also, see https://github.com/publiclab/image-sequencer-app