SerialPort Support on Windows

[shaggygi]

Just wondering if SerialPort namespace will be included at some point?

---

Execution plan

• [x] 1. @JeremyKuhne to seed the sources in CoreFX (from full .NET Framework) - add source code (sanitized, with licenses), the source won't build
• [x] 2. Make the code build
  • CC: @JeremyKuhne @karelz on PRs
  • Note: We will not take any functional changes to the implementation, architecture, or API surface at this point unless they are absolutely necessary to make the code compile on .NET Core. Please give heads up on this issue as soon as you discover cases like that.
• [x] 3. Add tests - tracking progress in dotnet/corefx#15752
  • [x] @JeremyKuhne to find out if we can leverage full .NET Framework tests
  • We need to design the test environment - we need ability to test the code in CI (ideally without heavy modification of CI machines)
  • We need to set goal for code coverage

If anyone is working on any step, please mention it & coordinate here to avoid duplicated effort. @karelz will co-assign the issue to you as well.

[gusmanb]

@shaggygi Once libusb support is done, then a class whith suport for USB serial devices can be written, as libusb allows to send any report to any device is a matter of porting the report data from some USB drivers like the ones from FTDI.

Of course there's another route to bring serial comms easier, and is using file descriptors. On Linux any device will appear as an entry on the file system, on Windows you can use CreateFile to also access them, opening these files should allow to read/write to the serial port but not change it's speed or porperties it must be done on the device settings or accesing it through ioctl. The biggest problem I see is to have to maintain different codes for different OS'es, that's why I suggested the use of libusb, just one API for all OS'es, only different PInvoke definitions.

[gusmanb]

@mobidev111 I would be glad to help on my spare time as supporting USB is a very necessary functionality for .net core.

[shaggygi]

The work @davidfowl is doing with channels might help for file streams, but not sure when. https://github.com/davidfowl/Channels/issues/81

And don't get me wrong... i would definitely use libusb as it would serve some scenarios for me if it works with ftdi cables. Thx for trying to drum support as this is a feature that needs to move forward with .net core. 👍

[shaggygi]

There is also a little bit of work going on here for .NET Standard. https://github.com/dotnet/standard/tree/master/extensions/serialport

[danmoseley]

The label edit above was by me, I reported the glitch to github.

[mobidev111]

libusb support could be here: "LibUsbDotNet for .NET Core 2.2.8-r101" https://www.nuget.org/packages/CoreCompat.LibUsbDotNet/ can't find the sources though - will contact the authors

[mobidev111]

sources are here: https://github.com/CoreCompat/LibUsbDotNet/tree/fixes/upgrade-msbuild @qmfrederik in what stage is your LibUsbDotNet modification - does it fully work on .Net Core? what are known limitations? @gusmanb could this be working to add support for USB serial devices?

[danmoseley]

@mobidev111 bear in mind @stephentoub 's note above that while .NET Core can certainly depend on 3rd party libraries, we cannot generally ship them or link with them. On Windows at least we should be able to just use the .NET Framework implementation (or the Mono one) without extra dependencies. On OSX/Unix ideally the dependencies would already be present.

Of course if this is a 3rd party standalone NuGet package it can do any of those things, but it would be great to be part of .NET Core itself.

@shaggygi the work in standard that you noted is only to define the surface area that such an extension to .NET Standard should have -- naturally this is essentially the .NET Framework surface area.

If it helps note that generally we can use and ship Mono sources. Whether or not we do, we can have have reused Mono tests.

[n8allan]

Just a note from experience, however, for any considering using mono source, that the SerialPort implementation in Mono has serious problems (on Linux). It has been months since I endured them so I don't remember all the particulars. Sporadic errors opening the port, especially after having it open then closing it, is one problem I recall, and I wasn't doing anything exotic. The same code on Windows & .NET works reliably.

[ItayGal2]

I'm working with mono's serial port on Raspbian and it works fine. tried it on mono v3 and v4

On Fri, Nov 11, 2016 at 7:57 PM, Nathan Allan notifications@github.com wrote:

Just a note from experience, however, for any considering using mono source, that the SerialPort implementation in Mono has serious problems (on Linux). It has been months since I endured them so I don't remember all the particulars. Sporadic errors opening the port, especially after having it open then closing it, is one problem I recall, and I wasn't doing anything exotic. The same code on Windows & .NET works reliably.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dotnet/corefx/issues/984#issuecomment-260015960, or mute the thread https://github.com/notifications/unsubscribe-auth/AJifNGRqy5b0GdYDaIiOFsE5X7PRUnBAks5q9KyHgaJpZM4Dj7O4 .

[danmoseley]

I would suggest the approach proposed by @stephentoub above

The first step is getting the Windows implementation brought into the corefx repo, getting it running with .NET Core on Windows, and putting tests in place. The second step is refactoring the library to introduce a managed PAL layer as we have in other libraries, with stubbed out Unix entry points to be implemented. Then comes the task of seeing what those entry points are and implementing them.

This way you get progressive value.

[qmfrederik]

@mobidev111 I consider the .NET Core port of LibUsbDotNet stable - we use it in production accross Windows, macOS & Linux.

Note that this is about USB devices and not the traditional ports. That said, if there is a plan to add USB support to .NET core, it's definitively worth considering libusb - it's essentially a PAL with Win, macOS & Linux implementations.

It's LGPL licensed, not sure wheter that would cause any issues.

So if you're considering libusb, I guess these two questions need to be answered first:

• Will USB support APIs be accepted into .NET Core?
• Is depending on LGPL libraries (via P/Invoke or linking) a possibility?

[stephentoub]

@qmfrederik, can you help me better understand what you're proposing? It sounds like you're talking about adding new surface area / library to corefx, rather than providing an implementation of the existing SerialPort surface area, is that correct? If so. I don't think that's a direction we want to head. We don't need every .NET library to come from the corefx repo, in fact we expect only a very small minority to come from the corefx repo. Other libs, like one focused on USB devices, would come from other repos, be published as NuGet packages, etc. If you haven't already, it'd be great to see your library published on NuGet and made available for others to consume with .NET Core.

If you're actually talking about implementing the SerialPort surface area on top of another library, that's something we could discuss, but whatever it P/Invokes into would need to be available everywhere. We don't want to include LGPL source in corefx, nor do we want to statically link to LGPL binaries, so for the implementation to be meaningful, we'd want the target library to be available on pretty much every platform we target... this is the bar we've tried to hold for other dependencies, like zlib, libcurl, etc.

[shaggygi]

@stephentoub I would definitely be happy if what is provided by current SerialPort be added. However, it would be even better to update it to include Async. For example ReadByte seems to hang until something is received on the port and would be nice to have a cancel token.

[qmfrederik]

@stephentoub I'm not suggesting anything in particular, just responding to @mobidev111 who added me to this thread :)

I think you answered both questions - LGPL code isn't a good candidate and USB support doesnt't fit CoreFX, so I guess that kind of settles the questions raised w.r.t. the USB side-track.

[stephentoub]

I would definitely be happy if what is provided by current SerialPort be added

@shaggygi, yup, understood, and as @danmosemsft noted, we're certainly open to it being added to corefx, we just haven't dedicated time yet to doing it. If you or someone else motivated to see it happen wanted to work on such a port, we'd be happy to see that happen, help code review, etc.

I'm not suggesting anything in particular

@qmfrederik, ah, ok, thanks for sharing about your lib, regardless.

[jcoder76]

+1 It is difficult for my work to transition to .NET standard/core without this. However, I would propose NOT to copy the failings of System.IO.Ports verbatim. Its modularity and compatibility is horrendous.

We have the opportunity to improve the serial port here since this is a break away from the original code base. Something more like: RJCP.SerialPortStream 2.0 would be better although it still targets .NET Framework 4.0 and depends on a lot of primitives, so it is hard to say how difficult it would be to port it to .NET Core. In the least we need to resolve many of the same problems if we start from the existing System.IO.Ports.

[viniciusvillas]

Quick technical question: should the SerialPort implementation in .NET Core for Linux be based on file operations over the device? I mean, should, for instance, SerialPort.Write(...) call a write on /dev/ttyUSB0? In this case, how can we deal with the different serial port configurations that can be set by termios (tcsetattr/tcgetattr)? Should it always reset it to a raw serial port state?

[jcoder76]

@viniciusvillas if there is a way the feature can be abstracted for the advanced serial port use cases and still be compatible across all platforms I don't see why not, otherwise it belongs in a platform specific library.

[shaggygi]

Just a side note on future possibility. I came across this link how to access serial device within Docker container.

http://stackoverflow.com/questions/24225647/docker-any-way-to-give-access-to-host-usb-or-serial-device

[DirkRoelandt]

Please note UWP's implementation is not really useful implementation as it simply does not support system-internal or on-chassis serial ports (read plain RS232).

[AlexeiScherbakov]

+1 Also implementation must be overridable. For example if you need write serial port over socket without writing driver, or other serial port implementation.

Current .NET Framework/Mono implementation for example don't allow to override Open method, and it is a problem on Linux (http://antanas.veiverys.com/mono-serialport-datareceived-event-workaround-using-a-derived-class/). When it overriden with new keyword existing libraries don't call it :-(

[danmoseley]

@karelz can you remind me the next step here, do we have a volunteers we can assign to to start the implementation?

this assumes it follows the existing API, otherwise it would need API proposal/review.

[karelz]

I would like to wait with SerialPort until we see reasonable progress on SignedXml and DirectoryServices where we primed the source code and asked community to help us to finish it off. If those efforts go well, we can kick off something similar with SerialPort in January. Assuming we will have volunteers to help with the work :)

[Wigmund]

+1 I am another which requires (on board) serial port access for UWP apps - its very restrictive for us without :/ we need to make UWP more comparable with standard .Net!

[karelz]

@Wigmund I agree that we need .NET Standard in UWP, that is on our radar already. However, given that SerialPort is not part of .NET Standard, it won't help with your problem. Moreover, UWP has restricted set of allowed Win32 APIs. If the SerialPort implementation from Desktop code (which we plan to eventually port over) uses banned APIs, then UWP will require special implementation.

You might want to check out WinRT SerialPort API. cc: @alecont

[shaggygi]

@karelz Is it the final decision to port the desktop version? If so, will it only be limited to Windows? I'm hoping the API will work on multiple platforms (Windows, Linux, Android, etc.) and devices (RPi, BeagleBone, PC, mobile, etc.). I completely understand it gets complex as there are many implementations/limitations... iOS for example.

[Wigmund]

@karelz I have a UWP app which needs to access a serial port for data (maybe UWP was a bad choice from the start and I need to port it to desktop .Net), however, am I correct in understanding that if I buy a USB to Serial adapter (for Windows) I will have more success in accessing the RS232 port under UWP?

[shaggygi]

@Wigmund I've used the SerialDevice API for UWP before and seems to do the job. I've used different dongles/chipsets (FTDI, Silicon Labs) and will work. The problem is 1) it is not the same simple API as System.IO.SerialPort in full .NET and 2) you have to configure more in the UWP app manifest to allow access similar to what you do with cameras and such.

[DirkRoelandt]

@karelz (and @shaggygi),

Thank for the note and reference to WinRT Serial API. The text suggests under point 1 of Create a SerialDevice object that serial port such "COM1" can be used under WinRT/UWP. Do you agree this is not the case when this actually a UART/Onboard serial port and thus only relevant with the Serial-to-USB convertor ? Without any extra convertor it simply does not work.

[karelz]

@shaggygi Is it the final decision to port the desktop version?

No it is not final. It is just IMO natural start to create a compatible API surface with Desktop. For such efforts we usually start with the Desktop implementation and then decide if we create another one for Linux/Mac, or if we can adjust the Windows one to be more x-plat. Components which heavily rely on OS specific APIs or libraries are obviously more tricky. Components which are pure IL are usually easily x-plat by their own nature. Creation of brand new API surface for SerialPort which is x-plat friendly is NOT tracked by this issue.

@shaggygi ... I'm hoping the API will work on multiple platforms (Windows, Linux, Android, etc.) and devices (RPi, BeagleBone, PC, mobile, etc.).

The .NET Core implementation(s) should ideally work in all places where .NET Core is supported (Windows / Windows UWP, Linux, Mac), unless there are technical difficulties preventing us to implement it on some of the platforms.

@Wigmund ... am I correct in understanding that if I buy a USB to Serial adapter (for Windows) I will have more success in accessing the RS232 port under UWP? @DirkRoelandt ... Do you agree this is not the case when this actually a UART/Onboard serial port and thus only relevant with the Serial-to-USB convertor ? Without any extra convertor is simply does not work.

You have reached the limits of my knowledge in the space - @alecont is the best person to chime in here (his team owns the WinRT APIs).

[shaggygi]

@karelz thanks for details. And regarding volunteers... I don't think I would be much help on dev, but will definitely be in line for testing and trying the bits out.

[willdean]

I would be happy to help in an effort to get SerialPort (for all its faults) ported to Core, at least on Windows platforms. Probably 75% of our projects use SerialPort, so if we're ever going to move anything to Core we will either have to write or find a serial port library anyway. I have a fair bit of experience of the Windows serial port stuff all the way down into the kernel.

SerialPort doesn't actually expose very many of the features of the underlying Win32 serial API, so I wouldn't imagine it's too awful a starting point for a x-plat library either.

My thoughts would be that a direct port of the existing ref source would be a good start - improvements like async, etc would be obvious additions but could come later.

Is there any assistance that MS can offer in terms of making this an 'official' effort, even if there aren't actually MS programmers working on it?

[karelz]

Yes, we can offer assistance to community efforts. We kicked off 2 similar efforts in December - System.DirectoryServices dotnet/corefx#2089 (sadly, it didn't catch on and the promised community contributions didn't start, yet) and SignedXml dotnet/corefx#4278 (with significant progress by community). Note that SerialPort was the next on the list -- we just didn't want to kick off too many such efforts in parallel, without any previous experience. And we felt the interest was a bit lower for SerialPort than for the first two. I believe we are now in position to try out couple of more such efforts, if there is interest from the community :).

In both cases we (MS employees) primed the source code (sanitized, with licenses), and then provided plan with steps (see the 2 examples above - build, add tests, etc.). The high-level idea is to make first only straightforward port, without any modifications, improvements and refactorings which are not necessary. Of course, incl. tests with reasonable test coverage. Compare the implementation between Desktop and Core. Then the next steps may be x-plat ports, and general evolution of the API surface and its implementation (e.g. new APIs, refactorings, perf improvements, etc.) You can expect MS employees to do code reviews (as we will have to own/shepherd the code base long-term), provide guidance on infra and chime in on testing strategy and test coverage goals (which may be particularly 'interesting' for SerialPort - I have no idea how testable it is).

If you're still interested, please let me know and I will find out who can prime the sources. Maybe more community members will stand up to help :). cc: @ianhays @JeremyKuhne @danmosemsft

[SWY1985]

Hi, I'm currently working on porting one of our .NET libraries to .NET Standard so that we can use the same library on Windows and Linux. Serial Ports are an important part of that library. I would be happy to assist in porting and testing where I can.

[JeremyKuhne]

As there is clearly interest, I'll start taking a look at moving the code. :)

At first glance it doesn't look too crazy.

[abdullah2993]

@JeremyKuhne that did be really great.

[willdean]

@karelz I like the process set out at the top of those two other efforts you linked to - both from the pov of absolutely not trying to improve the architecture / API on a first pass, and also seeing if there's any existing test code which can come across from outside the refsource but within MS.

Would porting to .NET Standard 2.0 be the current aim, or are we supposed to go to something lower at the moment?

[shaggygi]

I agree with @karelz and not quite sure on testing without physical hardware. Maybe the first go would be to use hardware and then add better test support when x-plat and API improvements round occurs. I did a quick search and came across these links that might give some insight.

PySerial Unit Tests - This has some example tests, but uses a loop back connector (again, physical hardware).

PySerial Stub for Testing - This mentions something about creating a "virtual port".

[willdean]

One thing we need to be really clear about with using hardware for test is what that hardware actually is, and whether we end up testing the hardware's drivers more than we do the SerialPort implementation. The same applies to the various 'virtual' ports systems.

In a perfect world I guess we would test against a 'real' (8250/16550-family) serial port as a kind of gold-standard, which used the native Windows serial.sys driver, but I suspect a lot of people would find this quite difficult to organise on modern computers.

Failing that, something FTDI-based (USB->Serial) would probably be the next best option - they're probably the most widely used and highest-quality driver implementation of USB serial ports (they're in-box since Win7), and you can easily buy a cable with the electronics moulded in that could be made into a loop-back using just a paperclip.

I am looking forward to finding out how the original .NET framework was tested in this regard.

[AlexeiScherbakov]

Serial Port can be emulated on linux (real machine, virtual or docker) via "socat". So VM can be used as test agent for linux based OSes. Is Serial Port loopback avaliable on Windows? May be Windows Server can make virtual serial port on VM that is linked not to real serial port, but to other virtual port VM?

30 Янв 2017 г. 15:04 пользователь "Greg Ingram" notifications@github.com написал:

I agree with @karelz https://github.com/karelz and not quite sure on testing without physical hardware. Maybe the first go would be to use hardware and then add better test support when x-plat and API improvements round occurs. I did a quick search and came across these links that might give some insight.

PySerial Unit Tests http://pyserial.readthedocs.io/en/latest/examples.html#unit-tests - This has some example tests, but uses a loop back connector (again, physical hardware).

PySerial Stub for Testing https://github.com/pyserial/pyserial/issues/120 - This mentions something about creating a "virtual port".

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dotnet/corefx/issues/984#issuecomment-276046171, or mute the thread https://github.com/notifications/unsubscribe-auth/AIQ7O7yZ2I9mPGJ-xOvXcvPFKa1Vqk3lks5rXdFsgaJpZM4Dj7O4 .

[SWY1985]

I am able to test with physical hardware on Linux and Windows using an USB RS485 converter (FTDI) and a generic USB RS232 port.

[karelz]

@willdean Reusing existing tests from Desktop .NET Framework is definitely and option. I will let @JeremyKuhne check how feasible it is (probably during his source code porting exercise). Maybe we use some SerialPort emulator there ... @ianhays might have also insight on SerialPort tests in .NET Framework - I think he made some fixes there recently-ish.

@willdean Would porting to .NET Standard 2.0 be the current aim, or are we supposed to go to something lower at the moment?

I would suggest to build the library on top of .NET Standard 2.0. That way, it can be reused on any platform which supports .NET Standard 2.0 (incl. Xamarin).

I hope we will be able to find some solution which does not need HW. In the worst case we can fall back to mocking OS behavior (esp. for CI), while adding ability to run the tests against real HW on demand.

[karelz]

I have added the execution plan to the top-most post. Let me know if there is anything missing, or warrants clarification.

I plan to assign the issue to folks who participates in the effort - for that reason I sent collaboration requests to @willdean and @SWY1985 (GitHub limitation - please ping me here when you accept). If others are serious about contributing to the effort, please say so explicitly here and I'll add you as well (I was intentionally rather conservative in interpreting the replies above as "I will help with code" - I prefer not to "force" anyone into contributing :)).

[shaggygi]

@karelz Once code is in a state we can run in our own dev environment, how do you want issues added (if some are found, that is :smile:) to this repo? Add in this issue thread? Create new issues labeled [SerialPort] XXX? Other?

I guess it would be more appropriate to create separate issues and reference back to this thread. That would allow for more direct focus on respective issue discussion.

[DirkRoelandt]

Hi Karelz,

Great to see some movement this topic. We are also candidate to do testing within our software package providing this core framework implementation will support UWP & onboard UART hardware.

[karelz]

@shaggygi I assume you are referring to steps [4] and [5] in the execution plan. I would use common sense :). Personally I think [5] should be filed separately (I included it in the plan for completeness) and [4] can be decided later ... if it starts right away as part of this effort, we may reuse this issue. If it doesn't start right away, it does not make sense to keep this issue alive just because it does not support Linux/Mac yet - it may be also easier to track Linux/Mac specific discussion in separate issue, etc. ... let's do what makes most sense when we get there.

@DirkRoelandt if SerialPort source code requires .NET Standard 2.0 APIs, then it will light up when UWP supports .NET Standard 2.0 (which is in planning & funding phase now, no ETAs yet). If there is strong desire to have SerialPort for UWP earlier than that, we could coordinate rewrite of SerialPort to target lower netstandard to unblock it earlier. I would consider it as part of [5] in the execution plan - i.e. AFTER we finish steps [1]-[3]. Warning: UWP implementation might depend on Windows APIs which may not be available in UWP process. In such case we would have to ask Windows to white-list them (.NET Standard 2.0 timeframe), or rewrite the UWP implementation on top of UWP-allowed APIs (e.g. WinRT). @alecont do you have any early insights here? Let's deal with the UWP problem as next step, after [1]-[3] are fully done.

[DirkRoelandt]

Fair enough.
Although I can not emphasize enough that the only the true added value of UWP (on desktop, kiosk, embedded, .. obviously) can be unleashed by having a proper serial interface to communicating with peripherals (still present for next years to come in a large majority of specialized hardware). This is currently huge inconvenience on the UWP platform.

[karelz]

@DirkRoelandt thanks for feedback on the scenario value! We are passing it to UWP platform owners to make sure we are on the same page regarding importance of the scenario.

[jcoder76]

I do not understand why we want to repeat all the work done by the RJCP serial port already. Could we not request to assume that implementation. It works cross platform already and closely follows the form of the System.IO.Ports.