HiDPI scaling creates flickering and scaling artifacts

[sebastianland]

We are trying to integrate the library into our open source Data Science Platform. Our Application runs fine on windows with high dpi scaling, with the exception of the plots.

What happens: When the Scaling is set to 150% in windows, the canvas will have a logical size of say 1000x1000. But in fact it should be backed by 1500 x1500 pixel image. When drawn, it will only fill up the space by 2/3rd. Then it seems to be scalled by some Java internals to 1500 but creating obvious artifacts.

What we tried: We retrieved the current scaling factor and extended the FBOCanvas to apply it to the Frame Buffer and the image painted. We simply made it bigger and painted on the right position. It worked and gives us a crisp image without flickering.

Whats still the problem: While the above worked for painting, the mouse events are still off. Even if correcting the mouse event coordinates, the calculations for selection will always use the Canvas size, which is not reliably related to the open gl image size. There are probably more like that and we would have to rewrite the entire code. So my idea was to get in connection with you as maintainers.

[hageldave]

Right, I remember I had this issue too when I worked on a machine with HiDPI. I think I tracked it down to the lwjgl-awt dependency being responsible for the bug. I commented on the respective issue on their repository, but that was almost 2 years ago, so I should check if that's now solved, or if I can work around it.

However, I think when using BlankCanvasFallback instead of BlankCanvas it should work correctly (can be used as drop in replacement when using their common interface JPlotterCanvas). But you won't get GPU accelerated draw calls then.

[sebastianland]

Thanks for having a look into it, David. The Fallback works, but the entire purpose of it was to get GPU acceleration. Otherwise we could simply stick to JFreeChart, which we already have integrated, as the speed is comparable. We tested that and only with GPU it is as responsive as a Data scientist would like if one has reasonable and realistic sized data sets.

[hageldave]

I pushed a fix but can't test it on Windows right now. Would you mind checking it out? The way it works is that the FBOCanvas will use the regular component size for it's FBO, but when the FBO is transferred (blit) to screen, it will use the actual framebuffer (on-screen) size for rendering.

[sebastianland]

Hi David, thanks for looking into it. Unfortunately it didn't work because of two things: The FBO is recreated during paint if dimensions don't match, which happens right after creating it first time a there the scaling was not takein into account. Second the paint(Graphics g) method needs to target the component getWidth()/getHeight() and read from the entire frame buffer. Then we have a crisp sharp image that's not flickering. I attached a patch, sorry for the reformatting. JPlotter.patch

But the coordinate calculation for handling events like drawing a zoom rectangle with the mouse is still entirely off as it seems not to take the default transform into account. Any chance you could look into that or tell me where to look myself?

[hageldave]

Thanks, I will have a look at this next week (need to borrow a device with high DPI screen). Based on what you are reporting, I'm anticipating that it will require more changes down the road. My idea was to simply draw onto the FBO in the regular resolution reported by the component's size, and then scale it up for displaying on screen since OpenGL draws on the actual device resolution. However, this will not give you a crisp looking image in the sense of what is actually achievable on a high DPI display. To get that, the FBO needs to be the size of the actual resolution in pixels on the device and then the shaders that draw the primitive elements (triangles, lines, points, and so on) need to be made aware of the scaling factor. Why is that? Because when I specify line thickness or size of a Glyph for a Point, then I want these elements to appear equally large whether I use a high DPI device or not.

Let me look into this next week and see whether it requires a lot of changes and if there can be a quick dirty temporary fix to make it at least display correctly.

[sebastianland]

Thanks! And just an Idea, but not sure if that's possible in OpenGL: Maybe there's something similar to the Affine Transformation that could be placed in front of the FBO, so that everything is scaled in the same way as AWT does it for HighDPI?

[hageldave]

So I finally found some time to test on a hidpi screen (razer blade 15 with 3840x2160 and nvidia quadro 5000 rtx). However, to me it seems that the fix that I provided does indeed work. I made a screen recording to demonstrate (first the bahavior before the fix, then the behavior after).

https://github.com/hageldave/JPlotter/assets/2974361/433f0a1d-3b27-4f57-ac6d-4a742b7d3606

The recording is done with 200% scaling and looks good because the upscaling is done with nearest neighbor interpolation. What I noticed is that using 150% scaling does not look crisp, cause the NN-interpolation does not simply quadruple every pixel as in the 200% case. Linear interpolation smears the pixels and also does not help for crispiness.

Next I will check if the shaders can be easily adapted to include a scaling parameter.

[sebastianland]

Hi, thanks for looking into it. For some reason I cannot play the file, so cannot say anthing about that. Did you try to zoom in? However, I'm indeed on 150%. Anyway, if we just scale that by using 4 px instead of 1, then we loose a lot of potential for visual details. And as most users will probably use this OpenGL solution to speed up renderings as they have large data sets, that could be quite important.

[hageldave]

I looked into the code and I have to say, it is kind of a nightmare to support non integer scaling such as 150% or 175%. This is mainly because the scaling factor also has to be taken into account for the viewport calculations (e.g. the content inside the coordinate system has a viewport so the content is not bleeding into the axes and axes labels). The glViewport calls however only take interger valued coordinates. Scaling the integer coordinates up with 150% would require some clamping afterwards, and in turn require a recalculation of the resulting x and y scaling due to the clamping, but also creating areas at the borders of a viewport where pixels can mix with other adjacent areas.

But what could be supported is rendering at 200% or 300% scaling and then later scaling it down to 150% for example to display.

[sebastianland]

What about the glViewportIndex method that was added with OpenGL4. Seems to me to do the same and accepting floating points. If not, I don't really see any benefit of the proposed solution to our very first approach to not modify open gl at all execpt painting on a bigger buffer and rewriting location calculation code to use buffer instead of java canvas size. Because we would have to adjust font sizes, thickness, etc anyway. And having to adjust them on basis of a more complex formular (first rendering larger, then dividing again) doesn't make much sense to me. Or is there any benefit in the rendering process, I don't see?

[hageldave]

I wasn't aware of glViewportIndexdf, thanks for pointing it out, however it seems that the actual support for subpixel precision is opengl imlementation and hardware dependent (GL_VIEWPORT_SUBPIXEL_BITS can be 0).

The first approach (which is currently in place) only upscales the image after rendering and looks okayish for 200% scaling. What I am proposing now is actually rendering on a screenbuffer that is twice the canvas resolution, which gives us crisp results, not just an image upscaling. Later downscaling to 150% will preserve that crispness. Of course, some changes have to be made to viewport calls and telling the shaders that they need to render at twice the size, but it will all be happening behind the curtains and no manual changes to font sizes or line thickness need to be made.

So the things that would need to be touched are:

• creation of the framebuffer object
• calls to glViewport (and methods that also adjust viewport like setRenderTargets)
• projection matrices passed to the shaders that take care of scaling to pixel coordinates (like orthoMX in GenericRenderer)

[sebastianland]

But wouldn't the downscaling from 200% to 150% will again require interpolation? I don't see the reason why we do that? I think it combines the worst of the two ways:

• A: Scaling the Open GL painting using viewport
• B: Rendering into a buffer fitting the real underlying pixel number and then paint that In A we might have sub pixel problems, in B we will have to adjust size of fonts, tick marks, symbols, etc. The way you do it, we have to deal with both problems once you want to support 150% which is a quite often used scaling factor on laptops and on 4K Displays.

[hageldave]

I think I miscommunicated this a little. In fact I think i already got the solution and it actually only required scaling up the FBO and adapting the glViewport calls.

Left is old (without scaling up the FBO), right is new.

There are still some loose ends that I need to take care of, e.g. reading pixel values from the canvas at specified location (for picking), and check if export (svg,pdf,image) is still working correctly. I also could not test it with fractional scaling yet, cause I can only do integer scaling on Ubuntu for AWT/Swing.

[sebastianland]

No problems, I actually don't have to understand it if it works... If you want, I can test that on windows, if you push your changes to a branch here.

[hageldave]

I just tested it on windows and it seems to work. Would be great if you could have a look. It's on the hidpi branch.

[sebastianland]

I can confirm that it works like a charm! Thanks very much. I will now be able to re-create our data visualization around this library.

[hageldave]

Great! Feel free to ask questions if you encounter any problems