Closed krwq closed 2 years ago
[Triage] We need to list all the bindings that uses System.Drawing and identify groups which need to be adjusted together. We also need to make sure that the person who'll do some of the work has the proper hardware to test it.
Here is the list of references to System.Drawing in the repo:
Those only using Color are quite straight forward to migrate. They require always the same pattern to transform the color space to the right one. A limited hardware can be needed to test but the pattern will always be the same. So that can limit the errors when the hardware is not available.
The problematic ones will be VideoDevice, RGBLedMAtrix and Ssd1351. They will for sure require the hardware to test and make sure all is working perfectly.
VideoDevice doesn't work for me properly. I get an error in this part of code video = device.CaptureContinuous(); Bitmap myBitmap = new Bitmap(video);
System.ArgumentException: Parameter is not valid. at System.Drawing.Image.InitializeFromStream(Stream stream) at System.Drawing.Bitmap..ctor(Stream stream)
@MaxPrimeAERY can you please create a separate issue for this problem? This is not related to this specific issue.
We're gonna need help implementing Font Hinting to be able to render test properly on low resolution devices. Any assistance there would be most welcome.
@JimBobSquarePants notice we currently use BDF fonts for very low resolution (couple pixels by couple pixels) but I agree it would be good to have that support
I understand the reason for moving away from System.Drawing. System.Drawing.Common only supported on Windows
Is there a particular reason for picking ImageSharp over e.g. SkiaSharp?
@A-J-Bauer I guess at this point we've already started using it here in some bindings, I can't remember exactly now but I think it might have been also the only alternative in the article at the time this issue got created (now I can see it mentions other alternatives, either overlook or it wasn't there before). I think this is still not set in stone but at this point folks got most familiar with it so might be the least work. I'm happy to reiterate on this decision, I personally don't have strong feelings to any library but we will need to test it on all devices if we plan to change this again. I'll be happy to hear any good comparison between couple of alternatives. In most of the devices I doubt library choice will be a bottleneck - I suspect SPI/I2C transmission will be much slower than any library choice we have.
I am not very familiar with SkiaSharp, but the two APIs are quite different and porting code could take a while.
Another reason to stay with System.Drawing is the amount of examples/code available in the wild.
I agree with @krwq that any library would be faster than I2C/SPI devices. But since this is an abstraction, it could make sense to use a library that can also be adopted when drawing on a regular display over HDMI. Anyway, I am not aware of any perf gain with SkiaSharp on the RPi for HDMI. Any comment on this is also welcome.
my 2 cents
@krwq The bottleneck analogy suggests that drawing functions are done in parallel and the program flow is constantly waiting for the bus to deliver bytes to the display's chip. In practice the drawing functions add significantly to the time needed to show a frame on the display since it is usually done sequentially (game loop). Even if using back buffers, doing things like 'Graphics DrawImage' from GDI+ or BitBlt from GDI in pure managed code without an underlying unmanaged custom memcpy (not Buffer.Blockcopy) somwhere is just very slow.
@raffaelerI think trying to draw directly over HDMI is not applicable. For larger screens people use a browser (possibly in kiosk mode if on the same device) as frontend and a web server on the device for dealing the HTML and Script (interestingly most browsers use Skia for the rendering). The question should probably be what is the best option for a graphics engine/library for small I2C/SPI displays with a dedicated chip when using C# for multiple plattforms.
SkiaSharp (backed by Microsoft Xamarin), Skia (backed by Google Chromium), ImageSharp is not nearly finished it seems..
SkiaSharp is more like System.Drawing than ImageSharp actually and provides Alias fonts without fiddeling around on the target plattform font (via configs) which is especially helpful for readability on black/white displays and small displays in general.
The Ssd1351 implementation below would be used like this:
using System.Diagnostics;
using System.Threading.Tasks;
using Iot.Device.Ssd1351;
using SkiaSharp;
// wait for debugger
// while (!Debugger.IsAttached) { await Task.Delay(1000).ContinueWith(dummy => { }); }
// create a paint object
SKPaint paint = new SKPaint()
{
Color = SKColors.White,
IsStroke = true,
StrokeWidth = 5,
IsAntialias = true
};
// create a display using SPI 0 and an initial rotation of 3 * 90°
Ssd1351 display = new Ssd1351(0, 3);
// turn the display on
display.PowerOn();
// get the 128x128 canvas from the display
SKCanvas canvas = display.SKCanvas;
// draw something on the canvas
canvas.Draw...
// update the display
display.Update();
await Task.Delay(6000);
display.Dispose();
https://dev.to/mindplay/comment/aka9 https://swharden.com/CsharpDataVis/alt/drawing-with-ImageSharp.md.html
@A-J-Bauer I am not sure to understand your point on HDMI. You can (and I am doing it via a library I don't remember now) use the Rpi graphics/gpu capabilities to output as you would do on a normal pc. The transfer rate is higher of course than spi/i2c devices and GPU-driven manipulation could make a huge difference. Anyway, this is less relevant in conjuction with the iot library. My point was about adopting the same abstraction regardless using the GPU or simpler spi/i2c devices. I have no strong opinion on that anyway.
@raffaeler no offense, not saying you can't, just thinking people that are using (usually bigger) displays > 1080x720 pixels probably want to have widgets, textboxes, input a.s.o. and doing all this from scratch is just not justified, but for custom visualization or a game there is spezial builds of SKIA for OpenGL, not entirely sure but for iot devices this might or might not be useful.
@A-J-Bauer It is very much use-case dependent. You do not always need a controls library. For example, if you are using the display to just show some data being acquired (graphs are the typical example), you may want to avoid the browser and html at all (which requires the whole desktop environment and running the browser in kiosk mode, sacrificing a lot of memory). Instead it would be far easier using a standard drawing library. I am stressing that it would be useful to provide the same abstraction for both i2c/spi devices and the normal GPU/HDMI displays. For example I am using 3.5" and 7" displays that are both attached to the RPi via HDMI.
Ideally, the best library from this perspective would be the one providing an accelerated "driver" (OpenGL?) and also a simple way to create "custom drivers" for i2c/spi devices. I am not proficient with Skia to say if it fits in this scenario.
@raffaeler "It is very much use-case dependent" Totally agree, "For example I am using 3.5" and 7" displays that are both attached to the RPi via HDMI" Do you know which library you are using for this, raylib? SKIA might have the potential for future utilization of the GPU if the host supports this,
I have the feeling that SkiaSharp would be the better choice for making the necessary move away from System.Drawing, that's all.
@raffaeler ok you convinced me, so I found some promising links to dig thru regarding directly drawing over HDMI (which I now know on Linux is into a frame buffer device), the last two look especially interesting.
https://medium.com/@avik.das/writing-gui-applications-on-the-raspberry-pi-without-a-desktop-environment-8f8f840d9867 https://github.com/mono/SkiaSharp/issues/492 https://stackoverflow.com/questions/53360854/skiasharp-drawing-image-with-gpu-acceleration
I don't want to get into a very lengthy thread on this issue, can you guys create a new thread and try to compile some table (or some other format) where we can easily compare benefits of any proposed library? I'd suggest at minimum include:
Alternatively if compiling table is hard, once there is enough data we can perhaps schedule some short meeting with everyone on the thread + everyone attending triage meeting where we can discuss this subject.
or you can hijack #displays channel on discord
@A-J-Bauer we can use this Discord channel to discuss and then do a summary here (link here: https://discord.gg/GyEfttux)
@A-J-Bauer ok, just to wrap-up the discussion for future readers:
I will open a different thread for the requirements so that it gets indexed. For any other discussion, let's do it on discord (my handle is raf).
Hey @michaelgsharp some time ago in an offline conversation I believe you were considering creating a drawing abstraction layer for ML.NET, and at that time I told you that if you were to do this we (dotnet/iot) would be interested in potentially use that abstraction as well. Are we still planning on doing this?
We are still planning on doing it, though we haven't figured out exactly what we will do or when we will do it. Worst case scenario is that it needs to be done before the release of .NET 7 in November (since .NET 7 no longer supports System.Drawing on Unix). My guess is that it will be much sooner than that, but I can't guarantee that.
Do you have any thoughts on what you would like it to look like?
That's great to hear! We haven't really thought about it too much, but we are currently using ImageSharp so the closer it is to that the easier it would be for us to adapt our code. That said, I don't really think it would not be too bad to adapt to the design that you guys decide on. I'd be interested to be on the loop for API reviews where this gets proposed just to make sure that the design would accommodate our needs.
Closing this in favor of https://github.com/dotnet/iot/issues/1767
System.Drawing is officially deprecated, we should use ImageSharp instead. See: https://docs.microsoft.com/en-us/dotnet/api/system.drawing?view=dotnet-plat-ext-5.0#remarks