dotnet / iot

This repo includes .NET Core implementations for various IoT boards, chips, displays and PCBs.
MIT License
2.18k stars 586 forks source link

We should replace dependency on System.Drawing with ImageSharp #1403

Closed krwq closed 2 years ago

krwq commented 3 years ago

System.Drawing is officially deprecated, we should use ImageSharp instead. See: https://docs.microsoft.com/en-us/dotnet/api/system.drawing?view=dotnet-plat-ext-5.0#remarks

Ellerbach commented 3 years ago

[Triage] We need to list all the bindings that uses System.Drawing and identify groups which need to be adjusted together. We also need to make sure that the person who'll do some of the work has the proper hardware to test it.

Ellerbach commented 3 years ago

Here is the list of references to System.Drawing in the repo:

Those only using Color are quite straight forward to migrate. They require always the same pattern to transform the color space to the right one. A limited hardware can be needed to test but the pattern will always be the same. So that can limit the errors when the hardware is not available.

The problematic ones will be VideoDevice, RGBLedMAtrix and Ssd1351. They will for sure require the hardware to test and make sure all is working perfectly.

MaxPrimeAERY commented 3 years ago

VideoDevice doesn't work for me properly. I get an error in this part of code video = device.CaptureContinuous(); Bitmap myBitmap = new Bitmap(video);

System.ArgumentException: Parameter is not valid. at System.Drawing.Image.InitializeFromStream(Stream stream) at System.Drawing.Bitmap..ctor(Stream stream)

Ellerbach commented 3 years ago

@MaxPrimeAERY can you please create a separate issue for this problem? This is not related to this specific issue.

JimBobSquarePants commented 3 years ago

We're gonna need help implementing Font Hinting to be able to render test properly on low resolution devices. Any assistance there would be most welcome.

https://github.com/SixLabors/Fonts/issues/30

krwq commented 3 years ago

@JimBobSquarePants notice we currently use BDF fonts for very low resolution (couple pixels by couple pixels) but I agree it would be good to have that support

A-J-Bauer commented 2 years ago

I understand the reason for moving away from System.Drawing. System.Drawing.Common only supported on Windows

Is there a particular reason for picking ImageSharp over e.g. SkiaSharp?

krwq commented 2 years ago

@A-J-Bauer I guess at this point we've already started using it here in some bindings, I can't remember exactly now but I think it might have been also the only alternative in the article at the time this issue got created (now I can see it mentions other alternatives, either overlook or it wasn't there before). I think this is still not set in stone but at this point folks got most familiar with it so might be the least work. I'm happy to reiterate on this decision, I personally don't have strong feelings to any library but we will need to test it on all devices if we plan to change this again. I'll be happy to hear any good comparison between couple of alternatives. In most of the devices I doubt library choice will be a bottleneck - I suspect SPI/I2C transmission will be much slower than any library choice we have.

raffaeler commented 2 years ago

I am not very familiar with SkiaSharp, but the two APIs are quite different and porting code could take a while.

Another reason to stay with System.Drawing is the amount of examples/code available in the wild.

I agree with @krwq that any library would be faster than I2C/SPI devices. But since this is an abstraction, it could make sense to use a library that can also be adopted when drawing on a regular display over HDMI. Anyway, I am not aware of any perf gain with SkiaSharp on the RPi for HDMI. Any comment on this is also welcome.

my 2 cents

A-J-Bauer commented 2 years ago

@krwq The bottleneck analogy suggests that drawing functions are done in parallel and the program flow is constantly waiting for the bus to deliver bytes to the display's chip. In practice the drawing functions add significantly to the time needed to show a frame on the display since it is usually done sequentially (game loop). Even if using back buffers, doing things like 'Graphics DrawImage' from GDI+ or BitBlt from GDI in pure managed code without an underlying unmanaged custom memcpy (not Buffer.Blockcopy) somwhere is just very slow.

@raffaelerI think trying to draw directly over HDMI is not applicable. For larger screens people use a browser (possibly in kiosk mode if on the same device) as frontend and a web server on the device for dealing the HTML and Script (interestingly most browsers use Skia for the rendering). The question should probably be what is the best option for a graphics engine/library for small I2C/SPI displays with a dedicated chip when using C# for multiple plattforms.

SkiaSharp (backed by Microsoft Xamarin), Skia (backed by Google Chromium), ImageSharp is not nearly finished it seems..

SkiaSharp is more like System.Drawing than ImageSharp actually and provides Alias fonts without fiddeling around on the target plattform font (via configs) which is especially helpful for readability on black/white displays and small displays in general.

alias antialias text sample code
ran on a Raspberry Pi W2: ![fontAlias](https://user-images.githubusercontent.com/20378441/150100388-77933bbb-80fa-4280-8f1a-6886a15bcfdf.png) ![fontAntiAlias](https://user-images.githubusercontent.com/20378441/150102399-1916e535-01dd-4e3e-800b-d3e6d5b72bae.png) ```C# // declare an array of colors SKColor[] colors = new SKColor[] { SKColors.White, SKColors.Green, SKColors.Blue, SKColors.Red, // SKColors.Cyan, // SKColors.Yellow }; // get the default font manager SKFontManager fontManager = SKFontManager.Default; // output the font names returned by the font manager to console foreach (string fontFamilyName in fontManager.FontFamilies) { Console.WriteLine(fontFamilyName); } // create a paint object SKPaint paint = new SKPaint(); // create two back buffer bitmaps high enough to print all font family names on SKBitmap bitmapFontsAlias = new SKBitmap(128, fontManager.FontFamilyCount * 14, SKColorType.Rgb565, SKAlphaType.Premul); SKBitmap bitmapFontsAntiAlias = new SKBitmap(128, fontManager.FontFamilyCount * 14, SKColorType.Rgb565, SKAlphaType.Premul); // create two canvases to be able to draw on the bitmaps SKCanvas canvasFontsAlias = new SKCanvas(bitmapFontsAlias); SKCanvas canvasFontsAntiAlias = new SKCanvas(bitmapFontsAntiAlias); // write the font names returned by the font manager to the canvases (and automatically to the underlying bitmaps) int y = 0; foreach (string familyName in fontManager.FontFamilies) { SKFont font = new SKFont(SKTypeface.FromFamilyName(familyName), 14); // pick a 'random' color from the color array paint.Color = colors[Random.Shared.Next(colors.Length)]; font.Edging = SKFontEdging.Alias; canvasFontsAlias.DrawText(familyName, 0, y, font, paint); font.Edging = SKFontEdging.Antialias; canvasFontsAntiAlias.DrawText(familyName, 0, y, font, paint); y += 14; } // save the alias bitmap using (var sKFileWStream = new SKFileWStream("fontAlias.png")) { if (sKFileWStream != null) { bitmapFontsAlias.Encode(sKFileWStream, SKEncodedImageFormat.Png, 50); } } // save the anti alias bitmap using (var sKFileWStream = new SKFileWStream("fontAntiAlias.png")) { if (sKFileWStream != null) { bitmapFontsAntiAlias.Encode(sKFileWStream, SKEncodedImageFormat.Png, 50); } } ```

The Ssd1351 implementation below would be used like this:

using System.Diagnostics;
using System.Threading.Tasks;
using Iot.Device.Ssd1351;
using SkiaSharp;

// wait for debugger
// while (!Debugger.IsAttached) { await Task.Delay(1000).ContinueWith(dummy => { }); }

// create a paint object
SKPaint paint = new SKPaint()
{
    Color = SKColors.White,
    IsStroke = true,
    StrokeWidth = 5,
    IsAntialias = true
};

// create a display using SPI 0 and an initial rotation of 3 * 90°
Ssd1351 display = new Ssd1351(0, 3);

// turn the display on
display.PowerOn();

// get the 128x128 canvas from the display
SKCanvas canvas = display.SKCanvas;

// draw something on the canvas
canvas.Draw...

// update the display
display.Update();

await Task.Delay(6000);

display.Dispose();
Ssd1351.cs ```C# // Licensed to the .NET Foundation under one or more agreements. // The .NET Foundation licenses this file to you under the MIT license. using System; using System.Threading; using System.Threading.Tasks; using System.Device.Gpio; using System.Device.Spi; using SkiaSharp; namespace Iot.Device.Ssd1351 { /// /// class for up to two 128 x 128 pixel SPI OLED displays with Solomon SSD1351. /// .NET 6.0 uses SkiaSharp /// ----------- /// NuGet /// SkiaSharp /// SkiaSharp.NativeAssets.Linux /// System.Device.Gpio /// ----------- /// raspberry pi: /// edit /boot/config.txt /// dtparam=spi=on /// dtoverlay=spi1-1cs // only needed if second display is used /// public class Ssd1351 : IDisposable { // ------ rpi physical pins ------ // // _0 for SPI 0 and _1 for SPI 1 // // 3.3V -- 1 2 -- 5V // 3 4 -- 5V // 5 6 -- gnd // 7 8 // gnd -- 9 10 // 11 12 -- cs_1 // rst_1 -- 13 14 -- gnd // dc_1 -- 15 16 -- dc_0 // 17 18 -- rst_0 // din_0 -- 19 20 // 21 22 // clk_0 -- 23 24 -- cs_0 // gnd -- 25 26 // 27 28 // 29 30 -- gnd // 31 32 // 33 34 -- gnd // 35 36 // 37 38 -- din_1 // gnd -- 39 40 -- clk_1 // // ------------------------------- private const int UNMODIFIED_PIN = -1; private const int SPICLOCKFREQUENCY = 24_000_000; private const UInt16 SPIBLOCKLENGTH = 0x1000; private enum SSD1351_CMD : byte { SetColumn = 0x15, WriteRam = 0x5C, SetRow = 0x75, SetRemap = 0xA0, SetStartLine = 0xA1, SetDisplayOffset = 0xA2, SetNormalDisplay = 0xA6, SelectFunction = 0xAB, SleepIn = 0xAE, SleepOut = 0xAF, SetPrecharge = 0xB1, SetClockDiv = 0xB3, SetVSL = 0xB4, SetGPIO = 0xB5, SetPrecharge2 = 0xB6, SetDeselectVoltageLevel = 0xBE, SetContrastABC = 0xC1, SetContrastMasterCurrent = 0xC7, SetMultiplexorRatio = 0xCA, SetCommandLocks = 0xFD } private static readonly int[] rpiDataCmdOledDcPin = new int[] { 16, 15 }; // low command, high data private static readonly int[] rpiResetOledRstPin = new int[] { 18, 13 }; // low active, keep high during normal operation private byte _remap = 0b01100100; // 565, C->B->A, Solomon SSD1351 manual private byte _startLine; private byte[] _spiBlock = new byte[SPIBLOCKLENGTH]; private SKBitmap _bitmap = new SKBitmap(128, 128, SKColorType.Rgb565, SKAlphaType.Premul); private SKCanvas _canvas; private int _spiClockFrequency = SPICLOCKFREQUENCY; private int _dataPin = UNMODIFIED_PIN; private int _resetPin = UNMODIFIED_PIN; private GpioController _gpioController; private SpiConnectionSettings _spiConnectionSettings; // ls /dev/ -l | grep spi private SpiDevice _spiDevice; private bool _initialized = false; /// /// Constructor /// /// SPI bus number (0 or 1) /// initial rotation (0,..,3) public Ssd1351(byte spiBus, byte rotation) { switch (spiBus) { case 0: // spidev0.0 case 1: // spidev1.0 _spiConnectionSettings = new SpiConnectionSettings(spiBus, 0) { ClockFrequency = _spiClockFrequency }; if (_dataPin == UNMODIFIED_PIN) { _dataPin = rpiDataCmdOledDcPin[spiBus]; } if (_resetPin == UNMODIFIED_PIN) { _resetPin = rpiResetOledRstPin[spiBus]; } break; default: throw new ArgumentException("0 and 1 allowed (spidev0.0 and spidev1.0)", "spiBus"); } switch (rotation % 4) { case 0: _remap |= 0b00010000; _startLine = 128; break; case 1: _remap |= 0b00010011; _startLine = 128; break; case 2: _remap |= 0b00000010; _startLine = 0; break; case 3: _remap |= 0b00000001; _startLine = 0; break; } _gpioController = new GpioController(PinNumberingScheme.Board); _spiDevice = SpiDevice.Create(_spiConnectionSettings); _canvas = new SKCanvas(_bitmap); } /// /// Constructor /// /// SPI bus number (0 or 1) /// initial rotation (0,..,3) /// spi clock frequency /// custom data pin /// custom reset pin public Ssd1351(byte spiBus, byte rotation, int spiClockFrequency, int dataPin, int resetPin) : this(spiBus, rotation) { _spiClockFrequency = spiClockFrequency; _dataPin = dataPin; _resetPin = resetPin; } /// /// SKCanvas to draw on /// public SKCanvas SKCanvas { get { return _canvas; } } private bool _isOn; /// /// Display status /// public bool IsOn { get { return _isOn; } } private void SendCommand(SSD1351_CMD cmd, byte[]? data) { _gpioController.Write(_dataPin, PinValue.Low); _spiDevice.WriteByte((byte)cmd); if (data == null) { return; } _gpioController.Write(_dataPin, PinValue.High); if (data.Length <= SPIBLOCKLENGTH) { _spiDevice.Write(data); } else { int blocks = data.Length / SPIBLOCKLENGTH; int remain = data.Length - blocks * SPIBLOCKLENGTH; byte[]? spiBlockRemain = remain > 0 ? new byte[remain] : null; for (int i = 0; i < blocks; i++) { Buffer.BlockCopy(data, i * SPIBLOCKLENGTH, _spiBlock, 0, SPIBLOCKLENGTH); _spiDevice.Write(_spiBlock); } if (spiBlockRemain != null) { Buffer.BlockCopy(data, blocks * SPIBLOCKLENGTH, spiBlockRemain, 0, spiBlockRemain.Length); _spiDevice.Write(spiBlockRemain); } } } private void Initialize() { _gpioController.OpenPin(_dataPin, PinMode.Output); _gpioController.OpenPin(_resetPin, PinMode.Output); _gpioController.Write(_resetPin, PinValue.Low); Thread.Sleep(200); _gpioController.Write(_resetPin, PinValue.High); Thread.Sleep(20); SendCommand(SSD1351_CMD.SetCommandLocks, new byte[] { 0x12 }); SendCommand(SSD1351_CMD.SetCommandLocks, new byte[] { 0xB1 }); SendCommand(SSD1351_CMD.SleepIn, null); SendCommand(SSD1351_CMD.SetClockDiv, new byte[] { 0xFF }); SendCommand(SSD1351_CMD.SetMultiplexorRatio, new byte[] { 127 }); SendCommand(SSD1351_CMD.SetDisplayOffset, new byte[] { 0x00 }); SendCommand(SSD1351_CMD.SelectFunction, new byte[] { 0x01 }); SendCommand(SSD1351_CMD.SetPrecharge, new byte[] { 0x32 }); SendCommand(SSD1351_CMD.SetDeselectVoltageLevel, new byte[] { 0x05 }); SendCommand(SSD1351_CMD.SetNormalDisplay, null); SendCommand(SSD1351_CMD.SetContrastABC, new byte[] { 0xD0, 0xD0, 0xD0 }); SendCommand(SSD1351_CMD.SetContrastMasterCurrent, new byte[] { 0b1111 }); SendCommand(SSD1351_CMD.SetPrecharge2, new byte[] { 0x01 }); SendCommand(SSD1351_CMD.SleepOut, null); SendCommand(SSD1351_CMD.WriteRam, new byte[128 * 128 * 2]); SendCommand(SSD1351_CMD.SetColumn, new byte[] { 0, 127 }); SendCommand(SSD1351_CMD.SetRow, new byte[] { 0, 127 }); SendCommand(SSD1351_CMD.SetRemap, new byte[] { _remap }); SendCommand(SSD1351_CMD.SetStartLine, new byte[] { _startLine }); _isOn = true; _initialized = true; } /// /// Turn the display on /// (enabled the internal Vdd regulator - more than 30mA current draw) /// public void PowerOn() { if (!_initialized) { Initialize(); } else if (!_isOn) { SendCommand(SSD1351_CMD.SelectFunction, new byte[] { 0x01 }); Thread.Sleep(10); SendCommand(SSD1351_CMD.SleepOut, null); _isOn = true; } } /// /// Turn the display off /// (disable internal Vdd regulator - less than 1mA current draw) /// public void PowerOff() { if (_isOn) { SendCommand(SSD1351_CMD.SleepIn, null); SendCommand(SSD1351_CMD.SelectFunction, new byte[] { 0x00 }); _isOn = false; } } /// /// Send the underlying bitmap of the canvas to the device. /// public void Update() { byte[] pixels = _bitmap.GetPixelSpan().ToArray(); // low byte byte of 565 16-bit expected first byte bucket; for (int i = 0; i < pixels.Length / 2; i++) { bucket = pixels[i * 2]; pixels[i * 2] = (byte)(pixels[i * 2 + 1]); pixels[i * 2 + 1] = bucket; } SendCommand(SSD1351_CMD.WriteRam, pixels); } /// /// Send the underlying bitmap of the canvas to the device. /// Same as the synchronous Update function making it possible to run two Update functions (two displays) in parallel with await Task.WhenAll. /// /// public async Task UpdateAsync() { await Task.Run(() => Update()); } /// /// Power off and clean up /// public void Dispose() { PowerOff(); _bitmap.Dispose(); _canvas.Dispose(); _spiDevice.Dispose(); _gpioController.Dispose(); } } } ```
Two displays on the RpiZero2 ![RpiZero2](https://user-images.githubusercontent.com/20378441/150108057-5f65e26d-9600-49c1-8c2d-7408dd457234.png)

https://dev.to/mindplay/comment/aka9 https://swharden.com/CsharpDataVis/alt/drawing-with-ImageSharp.md.html

raffaeler commented 2 years ago

@A-J-Bauer I am not sure to understand your point on HDMI. You can (and I am doing it via a library I don't remember now) use the Rpi graphics/gpu capabilities to output as you would do on a normal pc. The transfer rate is higher of course than spi/i2c devices and GPU-driven manipulation could make a huge difference. Anyway, this is less relevant in conjuction with the iot library. My point was about adopting the same abstraction regardless using the GPU or simpler spi/i2c devices. I have no strong opinion on that anyway.

A-J-Bauer commented 2 years ago

@raffaeler no offense, not saying you can't, just thinking people that are using (usually bigger) displays > 1080x720 pixels probably want to have widgets, textboxes, input a.s.o. and doing all this from scratch is just not justified, but for custom visualization or a game there is spezial builds of SKIA for OpenGL, not entirely sure but for iot devices this might or might not be useful.

raffaeler commented 2 years ago

@A-J-Bauer It is very much use-case dependent. You do not always need a controls library. For example, if you are using the display to just show some data being acquired (graphs are the typical example), you may want to avoid the browser and html at all (which requires the whole desktop environment and running the browser in kiosk mode, sacrificing a lot of memory). Instead it would be far easier using a standard drawing library. I am stressing that it would be useful to provide the same abstraction for both i2c/spi devices and the normal GPU/HDMI displays. For example I am using 3.5" and 7" displays that are both attached to the RPi via HDMI.

Ideally, the best library from this perspective would be the one providing an accelerated "driver" (OpenGL?) and also a simple way to create "custom drivers" for i2c/spi devices. I am not proficient with Skia to say if it fits in this scenario.

A-J-Bauer commented 2 years ago

@raffaeler "It is very much use-case dependent" Totally agree, "For example I am using 3.5" and 7" displays that are both attached to the RPi via HDMI" Do you know which library you are using for this, raylib? SKIA might have the potential for future utilization of the GPU if the host supports this,

I have the feeling that SkiaSharp would be the better choice for making the necessary move away from System.Drawing, that's all.

A-J-Bauer commented 2 years ago

@raffaeler ok you convinced me, so I found some promising links to dig thru regarding directly drawing over HDMI (which I now know on Linux is into a frame buffer device), the last two look especially interesting.

https://medium.com/@avik.das/writing-gui-applications-on-the-raspberry-pi-without-a-desktop-environment-8f8f840d9867 https://github.com/mono/SkiaSharp/issues/492 https://stackoverflow.com/questions/53360854/skiasharp-drawing-image-with-gpu-acceleration

krwq commented 2 years ago

I don't want to get into a very lengthy thread on this issue, can you guys create a new thread and try to compile some table (or some other format) where we can easily compare benefits of any proposed library? I'd suggest at minimum include:

Alternatively if compiling table is hard, once there is enough data we can perhaps schedule some short meeting with everyone on the thread + everyone attending triage meeting where we can discuss this subject.

krwq commented 2 years ago

or you can hijack #displays channel on discord

Ellerbach commented 2 years ago

@A-J-Bauer we can use this Discord channel to discuss and then do a summary here (link here: https://discord.gg/GyEfttux)

raffaeler commented 2 years ago

@A-J-Bauer ok, just to wrap-up the discussion for future readers:

I will open a different thread for the requirements so that it gets indexed. For any other discussion, let's do it on discord (my handle is raf).

joperezr commented 2 years ago

Hey @michaelgsharp some time ago in an offline conversation I believe you were considering creating a drawing abstraction layer for ML.NET, and at that time I told you that if you were to do this we (dotnet/iot) would be interested in potentially use that abstraction as well. Are we still planning on doing this?

michaelgsharp commented 2 years ago

We are still planning on doing it, though we haven't figured out exactly what we will do or when we will do it. Worst case scenario is that it needs to be done before the release of .NET 7 in November (since .NET 7 no longer supports System.Drawing on Unix). My guess is that it will be much sooner than that, but I can't guarantee that.

Do you have any thoughts on what you would like it to look like?

joperezr commented 2 years ago

That's great to hear! We haven't really thought about it too much, but we are currently using ImageSharp so the closer it is to that the easier it would be for us to adapt our code. That said, I don't really think it would not be too bad to adapt to the design that you guys decide on. I'd be interested to be on the loop for API reviews where this gets proposed just to make sure that the design would accommodate our needs.

joperezr commented 2 years ago

Closing this in favor of https://github.com/dotnet/iot/issues/1767