Need OLED Graphics API

[shaggygi]

We recently had a new binding PR (#188) for the SSD1306 OLED display controller. However, it is somewhat limited as it doesn't offer a good solution for text/images. We need a graphics library that can produce the respective bytes and possibly work with various other displays.

Also referencing #178 as it is somewhat related.

The Adafruit GFX graphics library might be a good place to start to understand what is needed.

[maloo]

Hi, I can port the ADA Gfx library if you want. Or you want a .Net Core designed one? Do you want just bindings or a pure C# library?

[joperezr]

What we are looking for here would be trying to port the GFX library so that we can use it with other bindings like the one that @shaggygi pointed out.

[maloo]

I could take a look at it. Maybe just use a Span as target for the GFX lib, then it could work with fbdev et al too. Since the color space on these displays is quite limited it might be enough to start with RGB565 support which most I2C/SPI displays support. Do you have any guidelines for contributions/PRs? Coding style, unit test requirements, process etc

[joperezr]

Do you have any guidelines for contributions/PRs? Coding style, unit test requirements, process etc

I replied to this question on this other issue. Regarding Coding style, we have an .editorConfig that will cause VS to have you follow some guidelines and we follow the same guidelines that corefx enforces

[BrianA-DK]

@maloo You might take a peek at https://github.com/Bodmer/TFT_eSPI and https://github.com/Bodmer/TFT_eFEX It has some extra TFT functions that is quite handy. Like support for sprites, antialiased fonts, BMP & JPEG load etc.

/Brian

[maloo]

Current plan is to write a bitmap/image class like RenderTargetBitmap with some basic draw ops like Adafruit Gfx lib. Then maybe add some bitblt ops. As for images I plan a command line tool to convert bmp/png/jpg etc to a .cs file in native format. Anyone that want a full blown graphics lib should probably just use System.Drawing with the Skia backend. Output will be to a byte array/span. Any display specific stuff is already available in the respective display binding. Then we could add som helpers/wrappers that combine rendering and display bindings for ease of use or RAD. I would love to run Miguel's Text UI framework on top of this...

[krwq]

I personally can live without drawing utils since you can do all of that in few lines of shader-like code but definitely font utils and common interface for all similar devices is a must for me.

I think we should find a common interface first before doing GFX stuff (although this does not necessarily conflict each other).

I think the interface should be based on the Color (or RGB888) class and the encoding to RGB565 should be done by the library itself and we should provide utils for that - I'm open for discussion since this might be a bottleneck for larger displays but I think either way we should have a way to draw using Color and on the side we can also have RGB565 utils. Perhaps we should have both and have an easy way of converting between them.

@maloo's Bitmap class in the PR above resembles of reasonable common interface - we would need to polish it up with perf and external libraries in mind.

[maloo]

@krwq Yes, bitmap fonts is a must. Event for "larger" SBC displays like the 3-4" TFTs. Above that people should use System.Drawing/Skia/GLES/etc. For Fonts I think a single fixed font should be enough. And then a tool for users to generate their own fonts when needed. That is, pay for what you use. After looking around in .NET System.Drawing (Forms) and System.Windows.Media (WPF) I realize they both use different Color classes, different Rect/Rectangle etc. So I don't see any harm in doing the same for Iot SW rendering. But if we have to select camp, I would also go for System.Drawing which is also designed for SW rendering. And until .NET Core has a decent tree shaking linker (promised for many years now) System.Drawing is an expensive addon. I have some BitmapImage classes for XRGB8888, RGB888 and RGB565 that I will add when needed. I can see two parts of API surface for these. The native ones (sealed, nonvirt for perf on specific class) and the BitmapImage inherited ones for simplicity/unity. But I don't see the need to add optimized versions of the operations before the API is stable. I have worked several years writing and optimizing SW rendering libs for ARM mobile platforms (targeting JBenchmark et al) and the amount of code needed for performance is huge. I also think we should skip the DrawLine/Circle/Ellipse/Polygon/Arc/Curve/etc until someone really needs it. I think System.Drawing/Skia is a better choice for them. Instead I suggest focusing on DrawVLine/DrawHLine/DrawRect/FillRect/BitBlt and other operations useful in embedded small screen GUIs and TUIs.

[krwq]

@maloo I suggest to update the first post with API proposal and we can iterate on that. I'd move GFX to separate issue since it will get lost if we have 2 separate conversations in one thread.

[joshfree]

@tarekgh @JeremyKuhne

[krwq]

@joshfree we are currently investigating the common interface with @tarekgh and @JeremyKuhne

[maloo]

@joshfree @tarekgh @JeremyKuhne @krwq Any progress on this? I've not had much spare time, but hope to propose some of the functions I think is appropriate for this API/Util before next week.

[krwq]

@maloo we have talked about it a bit and currently we do not have yet clear agreement.

We are thinking of something taking System.Drawing.Bitmap (i.e. DrawBitmap) but at the same time we don't want to take System.Drawing as a dependency.

We will probably allow this to wait a little bit until we got clearer understanding what's needed.

Here is the alternative proposal:

Color -> eventually will be replaced with something of size of uint. (Color contains string fields and some other stuff which should never be there)

// alternative names
//  - IDrawingDevice
//  - IDisplay (perhaps with some suffix such as Device)
// also idea of this being a generic interface which takes color type as a generic argument has been considered
interface IDrawable
{
    // coordinate system
    // every X, Y in this interface refer to the device coordinate
    // (0, 0) => Left top corner (y goes down)

    // Dimensions of the device
    // For chainable devices this should be total width/height
    int Height { get; }
    int Width { get; }

    // Fills rectangle with specific color
    // width, height == 0 => default to device dimension
    void FillRect(Color color, int x = 0, int y = 0, int width = 0, int height = 0);

    // if outputWidth == 0 => outputWidth = Width - x
    // if stride == 0 => stride = (updated) outputWidth
    // stride allows skipping parts of the buffer (stride - outputWidth bytes) after finishing drawing the row
    // outputHeight is not needed because it can be infered from the buffer length and the remaining parameters
    void DrawRect(int x, int y, ReadOnlySpan<Color> buffer, int outputWidth = 0, int stride = 0);
}

// SetPixel can be an extension method
public static class DrawableExtensions
{
    public void SetPixel(this IDrawable drawable, int x, int y, Color color)
    {
        Span<Color> buf = stackalloc Color[1] { color };
        drawable.DrawRect(x, y, buf);
    }
}

• only for low level -each display must be able to implement it
• more functionalities supported by the devices can be exposed in the device (chaining options etc.)
• we can add more functionalities to the interface later if we find we need it for more performance oriented scenarios (or modify it)

cc: @richlander

[shaggygi]

PR #308 uses the following from https://github.com/SixLabors...

using SixLabors.Fonts; using SixLabors.ImageSharp; using SixLabors.ImageSharp.PixelFormats; using SixLabors.ImageSharp.Processing;

[krwq]

me, @JeremyKuhne and @tarekgh have discussed today: What about something like this for now - once we identify issues with System.Drawing.Bitmap we will either address them or propose something better in the future (in the corefx itself - this process may take a while because corefx APIs are stable - once they are public we cannot remove them anymore so we have to bounce the ideas by more people). Until then perhaps something like this:

// graphics display driver is recommended (not enforced) to do following
class MyDisplayDriver
{
  // in some cases like persistent vision this may be
  // i.e. a float value since you got virtually infinite resolution
  // this is what I mean by recommendation vs enforcement by i.e. interface
  public int Width = 123 { get; }
  public int Height = 234 { get; }
  public void DrawBitmap(System.Drawing.Bitmap);
}

// implementation of the graphics library is recommended to do following
class MyGraphicsLib
{
  // alternative names:
  // - Update
  public void DrawBitmap(System.Drawing.Bitmap);
}

This way user can quickly combine both worlds (i.e. display this image, generated by this library on this device).

As I said eventually we will figure out all the issues and inconveniences with Bitmap and we will try to address them in corefx.

cc: @gloveboxes @Ellerbach @DemoS76 @maloo @garciaolais

[joperezr]

I don't think @tarekgh is working on this anymore, so flagging as up-for-grabs so that community can pick this item up.

[tarekgh]

The drawing APIs in https://github.com/dotnet/iot/blob/master/src/devices/RGBLedMatrix/RgbLedMatrix.cs is good start in general to look at.

[krwq]

To be more specific: Width, Height, SetPixel, Fill, DrawBitmap covers most of the scenarios - for tiny matrices also DrawText since System.Drawing text utils looks good only on tiny bit larger matrices

[maloo]

@krwq What is the status on graphics APIs? I have seen some demos by @richlander in the On .Net Show. What APIs are they using? Do you have a list of requirements/primitives? Do we still have time to drop System.Drawing dependency before 1.0? Just using Color from System.Drawing?

[krwq]

@maloo likely it's one of these two, not sure which demo did you see: https://github.com/dotnet/iot/tree/master/samples/led-matrix-weather https://github.com/dotnet/iot/tree/master/src/devices/RGBLedMatrix/samples

For small stuff graphics like this I have it in my backlog to add some utilities to make it easier to re-use these components but not sure when I'll have time to pick it up.

[maloo]

I could start with the primitives you got defined above when I start playing with #796. Once we got what we need to do a basic UI, we could go back to this issue.

[krwq]

@maloo let's start with writing down what we need. I.e.:

• abstract "drawable" element
• layouts (i.e. horizontal/vertical) which define list of "drawable"s and lay it out according to its own rules
• geometric figures: circle, square, ...
  • properties: filled?; line thickness; line smoothness
• text
  • properties: background color (includes alpha channel); font; font color; ...
• abstract way of grouping children stuff in a rectangle region etc.

[krwq]

We should start with replacing System.Drawing with ImageSharp (see: https://github.com/dotnet/iot/issues/1403) and see if we can cover some abstraction from ImageSharp as part of the "drawable" abstraction

[maloo]

Don't you think IMemory is enough? This would allow interop to the framebuffer, drm, managed memory, unmanaged memory etc. https://github.com/SixLabors/ImageSharp/blob/master/src/ImageSharp/Image.WrapMemory.cs

[krwq]

@maloo I haven't personally had time to play around with ImageSharp yet, possibly that's the right abstraction

[krwq]

[Triage] https://github.com/dotnet/iot/issues/1644 replaces this one. We also now use ImageSharp which makes it so that we less need this abstraction. Closing.