search

javafxports / openjdk-jfx

The openjfx repo has moved to:
https://github.com/openjdk/jfx
GNU General Public License v2.0
1.01k stars 145 forks source link

JDK-8217605: Add support for e-paper displays #339

Closed jgneff closed 5 years ago

jgneff commented 5 years ago

Report Classification

Issue Type: Request for Enhancement
Component: JavaFX
Subcomponent: graphics: JavaFX Graphics
Operating System: Ubuntu
Java Release: 12

Report Details

Synopsis

Add support for e-paper displays.

Description

Summary

This is a request for JavaFX to support devices with electrophoretic displays, commonly known as electronic paper displays, with a new Monocle platform called “EPD.”

Goals

The ultimate goal of this request is to be able to run a JavaFX application on a device with an e-paper display by simply installing the Ubuntu OpenJFX package for the armhf architecture. That goal includes:

Non-Goals

The following are specifically out of scope for this request:

Success Metrics

One measure of success would be running a JavaFX animation program on an e-paper display at eight frames per second and delivering the correct touch screen input events to the program.

Motivation

The motivation for this request comes from a work of art by Joseph Farbrook exhibited at the SIGGRAPH 2014 Art Gallery and part of a series called Lineographs. See the artist’s website for a brief description of the work and a one-minute video demonstration. The video is also available directly on YouTube as Dancer.

The artwork was remarkable in at least two ways. First, it was stunningly simple, using a device that emits no light, makes no sound, and employs the most minimal user interface. Among the bright screens and loud noises of the SIGGRAPH Conference, it was a passive and silent wonder—an ink drawing that came to life with a touch. Secondly, few people knew at the time that e-paper could display animations. Electronic ink was invented in 1997, and the first commercial e-reader introduced in 2004, but it wasn’t until 2012 that a patent application described the various waveform modes, including one for fast black-and-white animation.

When I looked into writing my own software for e-paper, I found the information buried deep in archives of documents for embedded processors. The documents are helpful, and improve with each revision, but they describe only a C-language interface to the Linux frame buffer device and some IOCTL codes. I was looking for a way to write the software in a higher-level language with libraries for graphics, threads, and networking. I was hoping to test the programs on my workstation and run them unmodified on the device without having to use cross-compilers or emulators. In short, I wanted an interface in Java.

Description

The device driver for an Electrophoretic Display Controller supports the programming interface of the Linux frame buffer device with one key caveat: the contents of the frame buffer are not automatically updated to the display. The Monocle Linux platform in JavaFX already provides software rendering to the Linux frame buffer and can be extended to provide support for e-paper displays.

I propose a new Monocle platform, named EPD, that allows a JavaFX application to run on a device with an Electrophoretic Display Controller (EPDC) frame buffer driver. To select the platform, the application is started with the system properties:

$ java -Dglass.platform=Monocle -Dmonocle.platform=EPD -Dprism.order=sw ...

Additional system properties for the EPD platform are described below in three groups: settings for the Linux frame buffer device, general display update settings, and update flags.

Frame Buffer Device
Display Updates
Update Flags

The EPDC drivers on the NXP i.MX6SL and i.MX6SLL processors support software dithering, while the i.MX7D supports dithering in hardware. Dithering is not available on the i.MX507 and i.MX508 processors.

Messages are logged using the “javafx” system logger with the following levels:

Alternatives

There are three main alternatives to providing the support in JavaFX:

Among the popular cross-platform toolkits of Qt, Swing, JavaFX, and Flutter, only JavaFX supports both desktop computers and embedded devices with a single executable file.

Testing

The closest I have found to an inexpensive general-purpose computer with an e-paper display is one of the Rakuten Kobo e-readers, shown on that page with pricing in U.S. Dollars. (See the Canada website for pricing in Canadian Dollars.) Older used Kobo e-readers, such as the Kobo Touch, can be found for less than 40 USD, and newer refurbished models sell for less than 70 USD from Rakuten on eBay.

On the other hand, the E Ink ICE evaluation kit with a 6-inch display costs 449 USD and is not intended for personal or household use. “Orders placed by consumers will not be accepted,” says the notice at the E Ink Shop.

The advantage of a Rakuten Kobo over an Amazon Kindle is that you don’t need to jailbreak the device for access. A single setting added to a configuration file (“Kobo eReader.conf”) enables root access with the telnet command:

[DeveloperSettings]
EnableDebugServices=true

Once logged in, it’s relatively easy to create a chroot environment with a small Ubuntu Base image and to set up Ethernet over USB. You then have a stable high-speed connection with over 45,000 open-source packages ready to install—including OpenJDK and OpenJFX—already compiled for the armhf architecture. If necessary, you can emulate everything but the display on a regular Linux workstation by running Ubuntu in a QEMU virtual machine with the armv7l architecture.

Furthermore, all of the source code for the Linux kernel, the EPDC frame buffer drivers, and related software is posted in the Kobo-Reader repository on GitHub. Of course, you can also use the Kobo e-reader to read books.

Risks and Assumptions

There is the risk of Kobo devices being locked down or discontinued, making the development of this enhancement more difficult and expensive. In addition, this request is tied to the current technology for electronic paper, the microencapsulated electrophoretic display, creating the risk of a new technology taking over the market and making the implementation obsolete.

In the long run, both of these events are likely to occur as companies, markets, and technologies change. Meanwhile, the Kobo e-reader is distributed globally, has the leading market share in Canada, and is second to the Kindle in market share worldwide. The current E Ink technology was invented in 1997 and has yet to be replaced, despite many well-funded efforts to do so.

Dependencies

The implementation of this request should avoid making changes to the current JavaFX source code so that it may be added without any risk of causing errors in the existing features.

System / OS / Java Runtime Information

I propose starting with the Kobo Touch Models N905B and N905C, which have either an NXP i.MX507 or i.MX508 processor based on the ARM Cortex-A8 core. These are the oldest and least powerful devices still supported by Kobo with system updates. Once the implementation is stable, other more powerful models can be added, such as the Kobo Glo HD with an i.MX6SL processor and the Kobo Clara HD with an i.MX6SLL processor, each based on the ARM Cortex-A9 core.

Given that plan, the first test system is a Kobo Touch Model N905C with an 800-MHz i.MX507 Multimedia Application Processor Model MCIMX507CVM8B and 256 MB of RAM. This processor has an ARM Cortex-A8 core implementing the ARMv7-A architecture. The system runs Ubuntu Base 14.04.5 LTS (Trusty Tahr) in a chroot environment.

$ uname -a
Linux koboa 2.6.35.3-850-gbc67621+ #619 PREEMPT
Thu Dec 22 15:29:00 CST 2016 armv7l armv7l armv7l GNU/Linux

$ ldd --version
ldd (Ubuntu EGLIBC 2.19-0ubuntu6.14) 2.19

$ getconf GNU_LIBPTHREAD_VERSION
NPTL 2.19

$ $HOME/opt/jdk-12-client/bin/java -version
openjdk version "12-internal" 2019-03-19
OpenJDK Runtime Environment (build 12-internal+0-adhoc.ubuntu.jdk)
OpenJDK Client VM (build 12-internal+0-adhoc.ubuntu.jdk, mixed mode)

The screen is a 6-inch E Ink Pearl display panel with a resolution of 800 × 600 pixels and a density of 167 pixels per inch, supporting up to 16 levels of gray (4-bit grayscale).

guaporocco commented 5 years ago

Awesome, very exciting proposal.

johanvos commented 5 years ago

Since you have -Dprism.order=sw do I understand that there is no OpenGL possible?

jgneff commented 5 years ago

Since you have -Dprism.order=sw do I understand that there is no OpenGL possible?

As far as I can tell, there's no 3D graphics support in any of the devices I plan to test. The processors for the devices have the hardware accelerators listed below, where EPDC is the Electrophoretic Display Controller, and PXP is the pixel processing pipeline.

My current test device has the i.MX507 processor.

jgneff commented 4 years ago

The second part of supporting e-paper displays is described by Issue #521 and implemented by Pull Request openjdk/jfx#60, and completes the following plans in this issue:

Once the implementation is stable, other more powerful models can be added, such as the Kobo Glo HD with an i.MX6SL processor and the Kobo Clara HD with an i.MX6SLL processor, each based on the ARM Cortex-A9 core.