copy() method has an high performance impact in Android mode

[Silenoid]

I know I should give more infos, but it's really as simple as the title: whenever I use the method in a sketch, even in very simple projects, the performances on Android drop drastically. If I limit the framerate to 60, for example, it usually drop lower than 30 FPS.

I'm using Processing 3.3.7 on Windows 10 64-bit. I'd add code or screenshots, but I think it'd just be redundant. I'm sure it does not depend on devices: I use Processing on multiple PCs and on different mobile phones. I've had this problem also with projects in P2D and P3D.

[Silenoid]

I've tried obtaining the copy() effect with a very simple magnifiyng fragment shader. Same performances problem using the filter() method: regardless of the complexity of some shaders, performances drops drastically when calling the filter() method, but #going very well when just drawing shapes with the shader() method.

It seems like any post-processing effect procedure has some sort of bottleneck.

[codeanticode]

@Scemenzo could you post some of the shaders that are causing a drop in performance? Thank you!

[Silenoid]

@codeanticode Thanks for answering, but it is not releated to the fragment shader body at all. I deeply researched the problem and finally understood that this has something to do with the way Pgraphics manipulation is done in Processing's Android mode throught the Android enviroment itself. The copy(), filter() and shader() methods could probably have some common sub-method call or procedure that breaks things up, but I cannot precisely point the problem as long as I don't know the source code of Processing itself and how it communicates with the android enviroment.

I've tried tinkering with some Pgraphics objects in very simple projects, not even including shaders, and I obtain all sorts of misbehaviour (bad graphical artifacts) and perfromances problems on the Android version, while everything works just fine on the Java version. I'm not totally ignorant on the matter (referring to the high level Processing knowledge, not the source's one), I've used these tools in Java mode and I made comparisons during time (different versions of Processing, everythong up to date, same device for testing) and I can confirm the discrepancy in more than a single occurrency.

I'm very sorry I can't help more than this and sorry for my bad english.

[codeanticode]

@Scemenzo Not a problem. What's the exact sub-method you are referring to that is breaking things up? I will be happy to investigate that further.

It would also be very helpful for me to try determining the cause of these issues if you could post a simple sketch demonstrating the graphical artifacts you are referring to.

Thanks again!

[Silenoid]

I'm sorry, I've edited "methods have some common sub-method" with "methods could probably have some common sub-method". As I said I've not read the source code, so I can only speak hypotetically (I'm very sorry).

Here are two very simple sketches that tries to sum up what I've described:

Sketch 1: drastic FPS drop between Java and Android mode (and between image() and copy())

PImage img;

void setup(){
  size(400,400);
  frameRate(60);
  img = loadImage("https://i.ytimg.com/vi/oHg5SJYRHA0/hqdefault.jpg");    //give INTERNET permission
}

void draw(){
  background(0);
  copy(img,300,300,50,50,0,15,100,100);  //double scaling image fraction
  text("Framerate: "+ frameRate, 10,10);

Java mode: 60 FPS, everything is fine. Android mode: about 34 FPS. 60 FPS if using image() instead of copy()

Sketch 2: Extreme FPS drop and misbehaviour between Java and Android mode

PImage img;
PGraphics pg;

void setup(){
  size(400,400);
  frameRate(60);
  img = loadImage("https://toppng.com/public/uploads/preview/melon-11528323303czwzg9l0rf.png");    //give INTERNET permission
  pg = createGraphics(400,400);
}

void draw(){
  pg.beginDraw();
  pg.background(0,0,0,0);
  pg.copy(img,400,300,100,100,0,15,200,200);
  pg.stroke(255);
  pg.fill(255,255,255,100);
  pg.ellipse(50,50,50,50);
  pg.endDraw();

  background(0);
  image(pg,0,0);
  text("Framerate: "+ frameRate, 10,10);
}

Java mode: 60 FPS, everything is fine. Android mode: about 10 FPS and the alpha value of the fill is not respected.

I hope this could be usefull and still hope I'm not talking about stupid things in stupid ways.

[codeanticode]

Ok, this is great and explains part of the problem. Thanks a lot!

The reason for the FPS slowdown is that the functions copy() and filter() (when using the default filters, not the shader-based ones) can result in a large performance penalty since they manipulate the pixels array for the entire screen. With this kind of operations, you can see that mobile devices can be significantly slower than a laptop/desktop computer.

In general, I would recommend trying to avoid large pixel operations on Android as much as possible, Some of effects obtained by manipulating the pixels array can be achieved with GLSL shaders, and doing that should not slow down your sketch, at least to the extent you have observed with copy().

Do you have a shader example that also results in a large FPS drop?

The fact that alpha value of the fill is not respected may actually be pointing to an actual bug, will look into it when I get the chance :-)

[Silenoid]

Thanks a lot, you're awesome. I forgot to mention I've also tried to manipulate the screen pixel matrix directly, a well known method in Java mode to optimize performances, but I had still bad result and now I have the certainty that this is a problem that goes beyond my knowledge and possibilities.

As for the shaders example, is it really usefull to post it? Everything I did was just defining a shader variable and loading a very very light threshold filter on a glsl fragment shader (hoping that it'd perform better than the built-in filters), also following some of your awesome documents. I, then, pass the shader instance in the filter() method to obtain a post processing effect on the whole screen (i honestly didn't find any other way to do it, so I was forced to use filter()), but the results are pretty identical in terms of performances. So, if you said that the filter() method is already compromized, it's not so much different to discuss about its behaviour working with a shader instead of standard filters.

[codeanticode]

Posting a minimal example using filter with a shader could help us figuring out the problem. From what you are describing, the FPS should not be impacted that much, but maybe there is an error somewhere else.

[Silenoid]

Ok, I'm going to post a simple sketch but I have some news after testing: it seemed I didn't have problems with Android mode this time. Performances seemed proportionally consistent between the two mode without any obvious malicious bottleneck. Then I remembered that, weeks ago when I discovered the filter(PShader) problem, I was focusing on live wallpaper coding.

Sketch

PImage img;
PShader sh;

void setup(){
  size(400,400, P2D);
  frameRate(60);
  img = loadImage("https://i.ytimg.com/vi/oHg5SJYRHA0/hqdefault.jpg"); //give INTERNET permission
  sh = loadShader("shader.glsl");
  sh.set("amount", 0.5);
  sh.set("falloff", 0.5);
}

void draw(){
  background(0);
  image(img,0,15);
  text("Framerate: "+ frameRate, 10,10);
  filter(sh);
}

A simple vignette Shader with "amount" and "falloff" to set

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif

uniform sampler2D texture;

uniform float amount;
uniform float falloff;

varying vec4 vertColor;
varying vec4 vertTexCoord;

void main() {
    vec4 color = texture2D(texture, vertTexCoord.xy);

    float dist = distance(vertTexCoord.xy, vec2(0.5, 0.5));
    color.rgb *= smoothstep(0.8, falloff * 0.799, dist * (amount + falloff));

    gl_FragColor = color;
}

I've done tests and so I can confirm: Java mode: 60 FPS, works like a charm. Android mode - app: stable Android mode - live wallpaper: about 30 FPS (very similar to the copy() and the filter(built-in filter) performances drop) caused by calling the filter(PShader) method.