We should replace dependency on System.Drawing with ImageSharp

[krwq]

System.Drawing is officially deprecated, we should use ImageSharp instead. See: https://docs.microsoft.com/en-us/dotnet/api/system.drawing?view=dotnet-plat-ext-5.0#remarks

[Ellerbach]

[Triage] We need to list all the bindings that uses System.Drawing and identify groups which need to be adjusted together. We also need to make sure that the person who'll do some of the work has the proper hardware to test it.

[Ellerbach]

Here is the list of references to System.Drawing in the repo:

• samples\led-matrix-weather
• src\devices\Apa102 -> seems to only use Color
• src\devices\Bh1745 -> seems to only use Color
• src\devices\Common\Iot\Device\Graphics -> seems to only use Color
• src\devices\GoPiGo3 -> seems to only use Color
• src\devices\Media\VideoDevice -> uses Color and PixelFormat, Size and maybe few others. This will be more complicated to migrate
• src\devices\PiJuice -> seems to only use Color
• src\devices\RGBLedMatrix -> Color, Rectangle and most likely few others. This may require quite some efforts
• src\devices\SenseHat -> seems to only use Color
• src\devices\Ssd1351 -> uses Color, Bitmap, Point. This will require quite some efforts
• src\devices\Tcs3472x -> seems to only use Color
• src\devices\Ws28xx -> seems to only use Color

Those only using Color are quite straight forward to migrate. They require always the same pattern to transform the color space to the right one. A limited hardware can be needed to test but the pattern will always be the same. So that can limit the errors when the hardware is not available.

The problematic ones will be VideoDevice, RGBLedMAtrix and Ssd1351. They will for sure require the hardware to test and make sure all is working perfectly.

[MaxPrimeAERY]

VideoDevice doesn't work for me properly. I get an error in this part of code video = device.CaptureContinuous(); Bitmap myBitmap = new Bitmap(video);

System.ArgumentException: Parameter is not valid. at System.Drawing.Image.InitializeFromStream(Stream stream) at System.Drawing.Bitmap..ctor(Stream stream)

[Ellerbach]

@MaxPrimeAERY can you please create a separate issue for this problem? This is not related to this specific issue.

[JimBobSquarePants]

We're gonna need help implementing Font Hinting to be able to render test properly on low resolution devices. Any assistance there would be most welcome.

https://github.com/SixLabors/Fonts/issues/30

[krwq]

@JimBobSquarePants notice we currently use BDF fonts for very low resolution (couple pixels by couple pixels) but I agree it would be good to have that support

[A-J-Bauer]

I understand the reason for moving away from System.Drawing. System.Drawing.Common only supported on Windows

Is there a particular reason for picking ImageSharp over e.g. SkiaSharp?

[krwq]

@A-J-Bauer I guess at this point we've already started using it here in some bindings, I can't remember exactly now but I think it might have been also the only alternative in the article at the time this issue got created (now I can see it mentions other alternatives, either overlook or it wasn't there before). I think this is still not set in stone but at this point folks got most familiar with it so might be the least work. I'm happy to reiterate on this decision, I personally don't have strong feelings to any library but we will need to test it on all devices if we plan to change this again. I'll be happy to hear any good comparison between couple of alternatives. In most of the devices I doubt library choice will be a bottleneck - I suspect SPI/I2C transmission will be much slower than any library choice we have.

[raffaeler]

I am not very familiar with SkiaSharp, but the two APIs are quite different and porting code could take a while.

Another reason to stay with System.Drawing is the amount of examples/code available in the wild.

I agree with @krwq that any library would be faster than I2C/SPI devices. But since this is an abstraction, it could make sense to use a library that can also be adopted when drawing on a regular display over HDMI. Anyway, I am not aware of any perf gain with SkiaSharp on the RPi for HDMI. Any comment on this is also welcome.

my 2 cents

[A-J-Bauer]

@krwq The bottleneck analogy suggests that drawing functions are done in parallel and the program flow is constantly waiting for the bus to deliver bytes to the display's chip. In practice the drawing functions add significantly to the time needed to show a frame on the display since it is usually done sequentially (game loop). Even if using back buffers, doing things like 'Graphics DrawImage' from GDI+ or BitBlt from GDI in pure managed code without an underlying unmanaged custom memcpy (not Buffer.Blockcopy) somwhere is just very slow.

@raffaelerI think trying to draw directly over HDMI is not applicable. For larger screens people use a browser (possibly in kiosk mode if on the same device) as frontend and a web server on the device for dealing the HTML and Script (interestingly most browsers use Skia for the rendering). The question should probably be what is the best option for a graphics engine/library for small I2C/SPI displays with a dedicated chip when using C# for multiple plattforms.

SkiaSharp (backed by Microsoft Xamarin), Skia (backed by Google Chromium), ImageSharp is not nearly finished it seems..

SkiaSharp is more like System.Drawing than ImageSharp actually and provides Alias fonts without fiddeling around on the target plattform font (via configs) which is especially helpful for readability on black/white displays and small displays in general.

alias antialias text sample code

ran on a Raspberry Pi W2:   ```C# // declare an array of colors SKColor[] colors = new SKColor[] { SKColors.White, SKColors.Green, SKColors.Blue, SKColors.Red, // SKColors.Cyan, // SKColors.Yellow }; // get the default font manager SKFontManager fontManager = SKFontManager.Default; // output the font names returned by the font manager to console foreach (string fontFamilyName in fontManager.FontFamilies) { Console.WriteLine(fontFamilyName); } // create a paint object SKPaint paint = new SKPaint(); // create two back buffer bitmaps high enough to print all font family names on SKBitmap bitmapFontsAlias = new SKBitmap(128, fontManager.FontFamilyCount * 14, SKColorType.Rgb565, SKAlphaType.Premul); SKBitmap bitmapFontsAntiAlias = new SKBitmap(128, fontManager.FontFamilyCount * 14, SKColorType.Rgb565, SKAlphaType.Premul); // create two canvases to be able to draw on the bitmaps SKCanvas canvasFontsAlias = new SKCanvas(bitmapFontsAlias); SKCanvas canvasFontsAntiAlias = new SKCanvas(bitmapFontsAntiAlias); // write the font names returned by the font manager to the canvases (and automatically to the underlying bitmaps) int y = 0; foreach (string familyName in fontManager.FontFamilies) { SKFont font = new SKFont(SKTypeface.FromFamilyName(familyName), 14); // pick a 'random' color from the color array paint.Color = colors[Random.Shared.Next(colors.Length)]; font.Edging = SKFontEdging.Alias; canvasFontsAlias.DrawText(familyName, 0, y, font, paint); font.Edging = SKFontEdging.Antialias; canvasFontsAntiAlias.DrawText(familyName, 0, y, font, paint); y += 14; } // save the alias bitmap using (var sKFileWStream = new SKFileWStream("fontAlias.png")) { if (sKFileWStream != null) { bitmapFontsAlias.Encode(sKFileWStream, SKEncodedImageFormat.Png, 50); } } // save the anti alias bitmap using (var sKFileWStream = new SKFileWStream("fontAntiAlias.png")) { if (sKFileWStream != null) { bitmapFontsAntiAlias.Encode(sKFileWStream, SKEncodedImageFormat.Png, 50); } } ```

The Ssd1351 implementation below would be used like this:

using System.Diagnostics;
using System.Threading.Tasks;
using Iot.Device.Ssd1351;
using SkiaSharp;

// wait for debugger
// while (!Debugger.IsAttached) { await Task.Delay(1000).ContinueWith(dummy => { }); }

// create a paint object
SKPaint paint = new SKPaint()
{
    Color = SKColors.White,
    IsStroke = true,
    StrokeWidth = 5,
    IsAntialias = true
};

// create a display using SPI 0 and an initial rotation of 3 * 90°
Ssd1351 display = new Ssd1351(0, 3);

// turn the display on
display.PowerOn();

// get the 128x128 canvas from the display
SKCanvas canvas = display.SKCanvas;

// draw something on the canvas
canvas.Draw...

// update the display
display.Update();

await Task.Delay(6000);

display.Dispose();

Ssd1351.cs

```C# // Licensed to the .NET Foundation under one or more agreements. // The .NET Foundation licenses this file to you under the MIT license. using System; using System.Threading; using System.Threading.Tasks; using System.Device.Gpio; using System.Device.Spi; using SkiaSharp; namespace Iot.Device.Ssd1351 { ///

/// class for up to two 128 x 128 pixel SPI OLED displays with Solomon SSD1351. /// .NET 6.0 uses SkiaSharp /// ----------- /// NuGet /// SkiaSharp /// SkiaSharp.NativeAssets.Linux /// System.Device.Gpio /// ----------- /// raspberry pi: /// edit /boot/config.txt /// dtparam=spi=on /// dtoverlay=spi1-1cs // only needed if second display is used ///

public class Ssd1351 : IDisposable { // ------ rpi physical pins ------ // // _0 for SPI 0 and _1 for SPI 1 // // 3.3V -- 1 2 -- 5V // 3 4 -- 5V // 5 6 -- gnd // 7 8 // gnd -- 9 10 // 11 12 -- cs_1 // rst_1 -- 13 14 -- gnd // dc_1 -- 15 16 -- dc_0 // 17 18 -- rst_0 // din_0 -- 19 20 // 21 22 // clk_0 -- 23 24 -- cs_0 // gnd -- 25 26 // 27 28 // 29 30 -- gnd // 31 32 // 33 34 -- gnd // 35 36 // 37 38 -- din_1 // gnd -- 39 40 -- clk_1 // // ------------------------------- private const int UNMODIFIED_PIN = -1; private const int SPICLOCKFREQUENCY = 24_000_000; private const UInt16 SPIBLOCKLENGTH = 0x1000; private enum SSD1351_CMD : byte { SetColumn = 0x15, WriteRam = 0x5C, SetRow = 0x75, SetRemap = 0xA0, SetStartLine = 0xA1, SetDisplayOffset = 0xA2, SetNormalDisplay = 0xA6, SelectFunction = 0xAB, SleepIn = 0xAE, SleepOut = 0xAF, SetPrecharge = 0xB1, SetClockDiv = 0xB3, SetVSL = 0xB4, SetGPIO = 0xB5, SetPrecharge2 = 0xB6, SetDeselectVoltageLevel = 0xBE, SetContrastABC = 0xC1, SetContrastMasterCurrent = 0xC7, SetMultiplexorRatio = 0xCA, SetCommandLocks = 0xFD } private static readonly int[] rpiDataCmdOledDcPin = new int[] { 16, 15 }; // low command, high data private static readonly int[] rpiResetOledRstPin = new int[] { 18, 13 }; // low active, keep high during normal operation private byte _remap = 0b01100100; // 565, C->B->A, Solomon SSD1351 manual private byte _startLine; private byte[] _spiBlock = new byte[SPIBLOCKLENGTH]; private SKBitmap _bitmap = new SKBitmap(128, 128, SKColorType.Rgb565, SKAlphaType.Premul); private SKCanvas _canvas; private int _spiClockFrequency = SPICLOCKFREQUENCY; private int _dataPin = UNMODIFIED_PIN; private int _resetPin = UNMODIFIED_PIN; private GpioController _gpioController; private SpiConnectionSettings _spiConnectionSettings; // ls /dev/ -l | grep spi private SpiDevice _spiDevice; private bool _initialized = false; ///

/// Constructor ///

/// SPI bus number (0 or 1) /// initial rotation (0,..,3) public Ssd1351(byte spiBus, byte rotation) { switch (spiBus) { case 0: // spidev0.0 case 1: // spidev1.0 _spiConnectionSettings = new SpiConnectionSettings(spiBus, 0) { ClockFrequency = _spiClockFrequency }; if (_dataPin == UNMODIFIED_PIN) { _dataPin = rpiDataCmdOledDcPin[spiBus]; } if (_resetPin == UNMODIFIED_PIN) { _resetPin = rpiResetOledRstPin[spiBus]; } break; default: throw new ArgumentException("0 and 1 allowed (spidev0.0 and spidev1.0)", "spiBus"); } switch (rotation % 4) { case 0: _remap |= 0b00010000; _startLine = 128; break; case 1: _remap |= 0b00010011; _startLine = 128; break; case 2: _remap |= 0b00000010; _startLine = 0; break; case 3: _remap |= 0b00000001; _startLine = 0; break; } _gpioController = new GpioController(PinNumberingScheme.Board); _spiDevice = SpiDevice.Create(_spiConnectionSettings); _canvas = new SKCanvas(_bitmap); } ///

/// Constructor ///

/// SPI bus number (0 or 1) /// initial rotation (0,..,3) /// spi clock frequency /// custom data pin /// custom reset pin public Ssd1351(byte spiBus, byte rotation, int spiClockFrequency, int dataPin, int resetPin) : this(spiBus, rotation) { _spiClockFrequency = spiClockFrequency; _dataPin = dataPin; _resetPin = resetPin; } ///

/// SKCanvas to draw on ///

public SKCanvas SKCanvas { get { return _canvas; } } private bool _isOn; ///

/// Display status ///

public bool IsOn { get { return _isOn; } } private void SendCommand(SSD1351_CMD cmd, byte[]? data) { _gpioController.Write(_dataPin, PinValue.Low); _spiDevice.WriteByte((byte)cmd); if (data == null) { return; } _gpioController.Write(_dataPin, PinValue.High); if (data.Length <= SPIBLOCKLENGTH) { _spiDevice.Write(data); } else { int blocks = data.Length / SPIBLOCKLENGTH; int remain = data.Length - blocks * SPIBLOCKLENGTH; byte[]? spiBlockRemain = remain > 0 ? new byte[remain] : null; for (int i = 0; i < blocks; i++) { Buffer.BlockCopy(data, i * SPIBLOCKLENGTH, _spiBlock, 0, SPIBLOCKLENGTH); _spiDevice.Write(_spiBlock); } if (spiBlockRemain != null) { Buffer.BlockCopy(data, blocks * SPIBLOCKLENGTH, spiBlockRemain, 0, spiBlockRemain.Length); _spiDevice.Write(spiBlockRemain); } } } private void Initialize() { _gpioController.OpenPin(_dataPin, PinMode.Output); _gpioController.OpenPin(_resetPin, PinMode.Output); _gpioController.Write(_resetPin, PinValue.Low); Thread.Sleep(200); _gpioController.Write(_resetPin, PinValue.High); Thread.Sleep(20); SendCommand(SSD1351_CMD.SetCommandLocks, new byte[] { 0x12 }); SendCommand(SSD1351_CMD.SetCommandLocks, new byte[] { 0xB1 }); SendCommand(SSD1351_CMD.SleepIn, null); SendCommand(SSD1351_CMD.SetClockDiv, new byte[] { 0xFF }); SendCommand(SSD1351_CMD.SetMultiplexorRatio, new byte[] { 127 }); SendCommand(SSD1351_CMD.SetDisplayOffset, new byte[] { 0x00 }); SendCommand(SSD1351_CMD.SelectFunction, new byte[] { 0x01 }); SendCommand(SSD1351_CMD.SetPrecharge, new byte[] { 0x32 }); SendCommand(SSD1351_CMD.SetDeselectVoltageLevel, new byte[] { 0x05 }); SendCommand(SSD1351_CMD.SetNormalDisplay, null); SendCommand(SSD1351_CMD.SetContrastABC, new byte[] { 0xD0, 0xD0, 0xD0 }); SendCommand(SSD1351_CMD.SetContrastMasterCurrent, new byte[] { 0b1111 }); SendCommand(SSD1351_CMD.SetPrecharge2, new byte[] { 0x01 }); SendCommand(SSD1351_CMD.SleepOut, null); SendCommand(SSD1351_CMD.WriteRam, new byte[128 * 128 * 2]); SendCommand(SSD1351_CMD.SetColumn, new byte[] { 0, 127 }); SendCommand(SSD1351_CMD.SetRow, new byte[] { 0, 127 }); SendCommand(SSD1351_CMD.SetRemap, new byte[] { _remap }); SendCommand(SSD1351_CMD.SetStartLine, new byte[] { _startLine }); _isOn = true; _initialized = true; } ///

/// Turn the display on /// (enabled the internal Vdd regulator - more than 30mA current draw) ///

public void PowerOn() { if (!_initialized) { Initialize(); } else if (!_isOn) { SendCommand(SSD1351_CMD.SelectFunction, new byte[] { 0x01 }); Thread.Sleep(10); SendCommand(SSD1351_CMD.SleepOut, null); _isOn = true; } } ///

/// Turn the display off /// (disable internal Vdd regulator - less than 1mA current draw) ///

public void PowerOff() { if (_isOn) { SendCommand(SSD1351_CMD.SleepIn, null); SendCommand(SSD1351_CMD.SelectFunction, new byte[] { 0x00 }); _isOn = false; } } ///

/// Send the underlying bitmap of the canvas to the device. ///

public void Update() { byte[] pixels = _bitmap.GetPixelSpan().ToArray(); // low byte byte of 565 16-bit expected first byte bucket; for (int i = 0; i < pixels.Length / 2; i++) { bucket = pixels[i * 2]; pixels[i * 2] = (byte)(pixels[i * 2 + 1]); pixels[i * 2 + 1] = bucket; } SendCommand(SSD1351_CMD.WriteRam, pixels); } ///

/// Send the underlying bitmap of the canvas to the device. /// Same as the synchronous Update function making it possible to run two Update functions (two displays) in parallel with await Task.WhenAll. ///

/// public async Task UpdateAsync() { await Task.Run(() => Update()); } ///

/// Power off and clean up ///

public void Dispose() { PowerOff(); _bitmap.Dispose(); _canvas.Dispose(); _spiDevice.Dispose(); _gpioController.Dispose(); } } } ```

Two displays on the RpiZero2


https://dev.to/mindplay/comment/aka9 https://swharden.com/CsharpDataVis/alt/drawing-with-ImageSharp.md.html

[raffaeler]

@A-J-Bauer I am not sure to understand your point on HDMI. You can (and I am doing it via a library I don't remember now) use the Rpi graphics/gpu capabilities to output as you would do on a normal pc. The transfer rate is higher of course than spi/i2c devices and GPU-driven manipulation could make a huge difference. Anyway, this is less relevant in conjuction with the iot library. My point was about adopting the same abstraction regardless using the GPU or simpler spi/i2c devices. I have no strong opinion on that anyway.

[A-J-Bauer]

@raffaeler no offense, not saying you can't, just thinking people that are using (usually bigger) displays > 1080x720 pixels probably want to have widgets, textboxes, input a.s.o. and doing all this from scratch is just not justified, but for custom visualization or a game there is spezial builds of SKIA for OpenGL, not entirely sure but for iot devices this might or might not be useful.

[raffaeler]

@A-J-Bauer It is very much use-case dependent. You do not always need a controls library. For example, if you are using the display to just show some data being acquired (graphs are the typical example), you may want to avoid the browser and html at all (which requires the whole desktop environment and running the browser in kiosk mode, sacrificing a lot of memory). Instead it would be far easier using a standard drawing library. I am stressing that it would be useful to provide the same abstraction for both i2c/spi devices and the normal GPU/HDMI displays. For example I am using 3.5" and 7" displays that are both attached to the RPi via HDMI.

Ideally, the best library from this perspective would be the one providing an accelerated "driver" (OpenGL?) and also a simple way to create "custom drivers" for i2c/spi devices. I am not proficient with Skia to say if it fits in this scenario.

[A-J-Bauer]

@raffaeler "It is very much use-case dependent" Totally agree, "For example I am using 3.5" and 7" displays that are both attached to the RPi via HDMI" Do you know which library you are using for this, raylib? SKIA might have the potential for future utilization of the GPU if the host supports this,

I have the feeling that SkiaSharp would be the better choice for making the necessary move away from System.Drawing, that's all.

[A-J-Bauer]

@raffaeler ok you convinced me, so I found some promising links to dig thru regarding directly drawing over HDMI (which I now know on Linux is into a frame buffer device), the last two look especially interesting.

https://medium.com/@avik.das/writing-gui-applications-on-the-raspberry-pi-without-a-desktop-environment-8f8f840d9867 https://github.com/mono/SkiaSharp/issues/492 https://stackoverflow.com/questions/53360854/skiasharp-drawing-image-with-gpu-acceleration

[krwq]

I don't want to get into a very lengthy thread on this issue, can you guys create a new thread and try to compile some table (or some other format) where we can easily compare benefits of any proposed library? I'd suggest at minimum include:

• license
• support level / responsiveness on github (i.e. in case of blocking issue or security issue how long will we wait for the update; how long if we contribute fix ourselves)
• small fonts support (i.e. will it look good with 32x32 LED matrix or some other tiny display)
• transition difficulty
• anything relevant to dotnet/iot scenarios you can think of

Alternatively if compiling table is hard, once there is enough data we can perhaps schedule some short meeting with everyone on the thread + everyone attending triage meeting where we can discuss this subject.

[krwq]

or you can hijack #displays channel on discord

[Ellerbach]

@A-J-Bauer we can use this Discord channel to discuss and then do a summary here (link here: https://discord.gg/GyEfttux)

[raffaeler]

@A-J-Bauer ok, just to wrap-up the discussion for future readers:

• I don't remember the library name, sorry. I don't have the repo available at this time
• If Skiasharp fits the requirement, my thumb is up

I will open a different thread for the requirements so that it gets indexed. For any other discussion, let's do it on discord (my handle is raf).

[joperezr]

Hey @michaelgsharp some time ago in an offline conversation I believe you were considering creating a drawing abstraction layer for ML.NET, and at that time I told you that if you were to do this we (dotnet/iot) would be interested in potentially use that abstraction as well. Are we still planning on doing this?

[michaelgsharp]

We are still planning on doing it, though we haven't figured out exactly what we will do or when we will do it. Worst case scenario is that it needs to be done before the release of .NET 7 in November (since .NET 7 no longer supports System.Drawing on Unix). My guess is that it will be much sooner than that, but I can't guarantee that.

Do you have any thoughts on what you would like it to look like?

[joperezr]

That's great to hear! We haven't really thought about it too much, but we are currently using ImageSharp so the closer it is to that the easier it would be for us to adapt our code. That said, I don't really think it would not be too bad to adapt to the design that you guys decide on. I'd be interested to be on the loop for API reviews where this gets proposed just to make sure that the design would accommodate our needs.

[joperezr]

Closing this in favor of https://github.com/dotnet/iot/issues/1767