processing / processing-website

Repository for the processing.org website
https://processing.org
GNU General Public License v2.0
68 stars 95 forks source link

Prevent mouse events from being captured by a live sketch when the cursor is not hovering it #310

Open SableRaf opened 2 years ago

SableRaf commented 2 years ago

Current behavior Mouse clicks register in interactive sketches even when the cursor is not over the sketch window (like clicking on links anywhere in the page). This is particularly problematic in examples like FileIO/SaveOneImage where any click triggers a file save.

Expected behavior Mouse clicks only register when the cursor is over the sketch window

Possible solution Embed the sketches into an iFrame. See @trikaphundo's first comment below

The following examples are using MousePressed to capture clicks and may need to be fixed

trikaphundo commented 2 years ago

Current behavior Mouse clicks register in interactive sketches even when the cursor is not over the sketch window (like clicking on links anywhere in the page). This is particularly problematic in examples like FileIO/SaveOneImage where any click triggers a file save.

Actually, I realized this when I was writing some examples in P5js (PR #309). Moreover, sketches are aware and receive any input event—either from keyboard or mouse— that occurs in the webpage.

This is expected, since the canvas is wrapped in a div element and directly inserted in the html element of the page. This can be checked with the browser inspector.

Expected behavior Mouse clicks only register when the cursor is over the sketch window

A possible solution (I am still investigating and experimenting on this) would be to wrap the canvas in an inner html document, that is, wrap it in an iframe element and insert that in the html element of the example page. This is what the P5js web editor does.

To make myself clearer, go over to the P5js web editor, paste the following code, play around and inspect the page's code. Note how the sketch stops updating the line when the mouse leaves the iframe element it is wrapped in.

function setup() {
  createCanvas(400, 400);
}

function draw() {
  background(220);

  line(0, 0, mouseX, mouseY);
}

Summing up, this is a problem that has very little to do with sketches; but with the html structure that holds the canvas the sketches draw on.

SableRaf commented 2 years ago

Thanks @trikaphundo! Shall I assign this one to you? Otherwise I can tag this as "help wanted"

trikaphundo commented 2 years ago

Thanks @trikaphundo! Shall I assign this one to you? Otherwise I can tag this as "help wanted"

I am writing some tests and tinkering with them, but for now I am making no promises ^^'.

Althought I have knowledge of web development technologies, never actually used an iframe. My plan is to get a small proof-of-concept repo (if I succeed) that everyone can copy and test locally, and then decide whether (and how) to take that into account for the actual processing's website.

So for now leave this unassigned, or at least do not assign it to me ^^'. I will do my best

trikaphundo commented 2 years ago

Hello, I have been reading the code of the P5js library that manages input events and, in particular, updating the mouse position. According to the documentation for the mouseX property

The system variable mouseX always contains the current horizontal position of the mouse, relative to (0, 0) of the canvas. The value at the top-left corner is (0, 0)

What the code actually does is computing the difference between the mouse position in the window (clientX and clientY properties) and the top left corner of the bounding rectangle of the canvas, which is a point in the page of the document.

Therefore the question is: where is our p5 sketch getting the events from? That is, who are the event handlers being registered to?

The 'problem' with events

The events P5js offers are binded to the window object of the html document (see the code). Note that this applies to both, mouse events and keyboard events. Therefore, it is perfectly normal that sketches respond to mouse and keyboard events occuring in the window, irrespective of whether the canvas has the focus or the mouse is over it.

It is interesting to note, though, that p5 objects register their event handlers to the window of the document they belong to, instead of the canvas' container div, because registering them to the div makes keyboard events not work. This is stated in a comment in the code linked to by the first link of the previous paragraph.

// Bind events to window (not using container div bc key events don't work)

This suggest a solution: give the sketch its own window.

Solutions

I see two possible solutions:

  1. Purposely ignore events in every single sketch we write, when the mouse is not over the canvas, or the canvas has not the focus. Really ugly solution but does not require modifying the website, just the sketches with boilerplate code.
  2. Load P5js library and the sketch in a separate window embedded in the document, so that the p5 object registers the event handlers to that inner embedded window. This can be done with the html element iframe.

I am dismissing the former and explaining how to carry out the latter. For a working example of this latter solution, see further below.

The changes to the existing code of Processing's website will be the following (I have no idea about React, so more changes may be in order)

There is no actual need to store in the server a new html document with just a pair of script tags, one to load the p5js library, the other to load the sketch, to later on give its URL to the src attribute of the iframe element. The document is so simple it can be embedded in the iframe's srcdoc attribute, which allows feeding in html content as a string.

Thus you can create the html document as a string in your react code and inject that string in the iframe's srcdoc attribute. This document consists of

Note: as the sketches are loaded in example pages by means of useEffect, the script tag for loading the sketch may be dropped? Someone with knowledge of react should work this out.

Minimal example

I have created a repository with a minimal working example. In this example I have not used the srcdoc attribute, but src. Anyway you should have no trouble using it by following through with the instructions given above.

Sandbox

In my example repository no Internet connection is required, but if you do load p5js from the Internet from within the iframe, you will encounter some problems. They have to do with security measures html enforces for embedded content; in such case you have to lift the appropiate restrictions on the iframe element via its sandbox attribute. Which restrictions to lift is up to the maintainers of the website.


I hope this is clear enough so that someone can turn my working example into working code for the Processing's website. There is the detail of how to load p5js and the sketch in the iframe with react; that will have to be worked out by whoever takes on this.

SableRaf commented 2 years ago

Thanks for the detailed investigation!

Regarding your point about createCanvas(windowWidth, windowHeight) I suppose we would also need to do the following to support the responsive layout of the website:

function windowResized(){
    resizeCanvas(windowWidth,windowHeight)
}

I think I'll implement the ugly solution for the FileIO/SaveOneImage example as a quick fix, but it's good to know that we have a path for a more robust solution.

trikaphundo commented 2 years ago

Thanks for the detailed investigation!

Whether Processing ends up adopting this proposal or not, I believe it is best to write down why and how it works; so that anyone can use this information in the future :).


Regarding your point about createCanvas(windowWidth, windowHeight) I suppose we would also need to do the following to support the responsive layout of the website:

function windowResized(){
    resizeCanvas(windowWidth,windowHeight)
}

Not unless the size of the iframe changes. The size of the iframe is fixed in absolute units (640x360px), and the canvas is instructed to fill its (document's) window (the iframe). Therefore it does not matter if the top window is resized, the canvas only cares about its window (the iframe), and as long as the iframe's size remains unchanged there is no need to use that function.

You can try this with the web browser inspector: open the inspector and manually change the dimensions width and height of the iframe element (via CSS or HTML attributes, it does not matter). You will see that, either scroll bars appear, or the canvas does not resize to fill the resized iframe.


I think I'll implement the ugly solution for the FileIO/SaveOneImage example as a quick fix, but it's good to know that we have a path for a more robust solution.

In issue #314 I mentioned this very issue. I did so because the version I wrote in P5js for the scale shape example, does not exhibit this behaviour of responding to events happening in the whole window, in that example, instead, it only responds to the mouse movement when it hovers over the canvas.

That is interesting because in that example, the onmousemove event handler is not registered with P5, but with Paperjs Tool object. Therefore I think there may be another way of accomplishing this with P5js sketches without having to modify the website code, but I have to look up the Paperjs code.