Serial Port support on Unix/OSX

[danmoseley]

Support the existing Windows API on OSX/Linux machines.

Original Windows thread was https://github.com/dotnet/corefx/issues/984

Any volunteers interested in starting to scope this out? Please thumbs up if you would use this library.

@willdean

[shaggygi]

This is somewhat related, but is more to do with WSL. I thought I would add as a link if it helps...

https://blogs.msdn.microsoft.com/wsl/2017/04/14/serial-support-on-the-windows-subsystem-for-linux/

Also, Rich Turner mentions SerialPort as one of the currently most features asked as it relates to IoT and such. You can watch here (around 1:55:30 mark) Windows Weekly 514: Bash on Windows

[daniel-mcassey-tmsuk]

Has there been any further progress on this?

[willdean]

@daniel-mcassey-tmsuk No, not yet. Somebody needs to do a big refactor on SerialPort to get it into the proper multi-platform shape, then things will be a little more straightforward for someone to do the Unix port.

I think the MS guys are very busy with the 2.0 release right at the moment, and I'm very busy with commercial stuff, so after the big push to port the Windows libraries and tests earlier in the year progress has rather slowed.

[obizues]

Also currently busy with commercial stuff, but if anything is learnt in that area we can pass it on.

[JeremyKuhne]

When we do the refactoring to enable this we should make sure we allow for fault testing, notably that we surface errors from the background thread. dotnet/corefx#19146

[dima117]

Mono's Posix Library now ported to .NET Core is available. https://www.nuget.org/packages/Mono.Posix.NETStandard It seems, now there is an opportunity to port code from the Mono project to .NET Core https://github.com/mono/mono/tree/master/mcs/class/System/System.IO.Ports

[dima117]

I tried to compile the source code from Mono for .NET Standard 2.0 using the Mono.Posix.NETStandard library. The compilation was successful and I was testing COM port reading and writing on the Mac OS.

Is there any way which I can help in the implementation of the serial port API on Unix/Mac OS? I can provide the working source code. Also, I'm ready to participate in the development, but I do not know about the development process.

[jamespettigrew]

@dima117, I'd love to see how you got it working. I've got a library with the Mono serial port implementation building just fine but I think I have a packaging issue as the DllImports fail to find MonoPosixHelper.

[dima117]

@jamespettigrew, you can find MonoPosixHelper library in the Mono.Posix.NETStandard package in NuGet.

It works fine, but i found the issue - mono serial port doesn't implement serial port events, like SerialPort.DataReceived

[jamespettigrew]

@dima117 It appears to be working now that I've referenced Mono.Posix.NETStandard in my CLI project. For some reason it's not being included in my library project's build output despite referencing the Posix NuGet package.

[danmoseley]

@dima117 we would love to include Unix/Mac support. One way to do this might be to follow the pattern of System.Drawing, which is to move the Mono code into CoreFX, and Mono pulls it from CoreFX via submodule. @mellinoe the motivation for doing that (rather than just pulling from Mono repo) was to include the Windows implementation in the same package, right?

I assume it is fine for CoreFX code to depend on Mono.Posix.NETStandard package although I don't know of precedent for an outgoing Nuget dependency. @weshaggard would there be concerns about that?

Is there any anecdata to help gauge the level of interest in Unix/Mac support?

/cc @marek-safar @mellinoe

[weshaggard]

I assume it is fine for CoreFX code to depend on Mono.Posix.NETStandard package although I don't know of precedent for an outgoing Nuget dependency. @weshaggard would there be concerns about that?

We cannot depend on that package unless we pull the building of it into our closure for our build from source efforts. That is certainly possible but it will require work and we will need to think twice before taking that dependency.

[danmoseley]

Interesting. @eerhardt are you aware of other cases where we might want to take a dependency from CoreFX onto Mono.Posix? I believe you had ideas for other Unix libraries we could offer, and those presumably would sit on Mono.Posix. So you will need to solve this problem.

[danmoseley]

@dima117 @jamespettigrew I'm also curious what your code uses serial ports for? For newly written code, we'd probably direct them to a different library, since the System.IO.Ports API's are not necessarily satisfactory. We've ported them mainly for preexisting code.

[jamespettigrew]

@danmosemsft I'm using it to communicate over RS-485 for a Linux based IoT type HVAC application. I'm not aware of any other suitable libraries, have I missed some?

[danmoseley]

@jamespettigrew ha, no I just assumed they existed. @willdean do you know of any?

[nicolasr75]

@jamespettigrew @danmosemsft I did a few tests with https://github.com/jcurl/SerialPortStream and was able to connect to my z-wave usb stick from an rpi with Ubuntu. That will hopefully allow me to talk to z-wave compatible sensors and actors for home automation.

[dima117]

@danmosemsft I'm using it to communicate with FT232R device over USB on MacOS and Linux.

@nicolasr75 I have tested SerialPortStream library too, but it not works for me. SerialPortStream depends on unmanaged libnserial library, which I can't compile (on MacOS). When I try to compile it, I see uninformative error message. I also asked for help to SerialPortStream developer, but he didn't help me.

[willdean]

@jamespettigrew ha, no I just assumed they existed. @willdean do you know of any?

Alternatives to SerialPort on unix? I don't know much about it, but SerialPortStream is the main alternative to SerialPort. I don't personally have as much of a problem with SerialPort as some of the world seems to, so I'm not really up on the alternatives.

[nicolasr75]

@dima117 Can't say much about MacOS. I had to compile libnserial myself too to get it working on ARM but that worked without any issues. You may want to check the tips I got here https://github.com/jcurl/SerialPortStream/issues/29

[dima117]

@nicolasr75 Yes, libnserial compiles fine on Linux, but it not compiles on MacOS out of box. Now I see this error message

CMake Error at libnserial/CMakeLists.txt:85 (install):
  install TARGETS given no FRAMEWORK DESTINATION for shared library FRAMEWORK
  target "nserial".

... and I don't know where find the solution.

[eerhardt]

I assume it is fine for CoreFX code to depend on Mono.Posix.NETStandard package although I don't know of precedent for an outgoing Nuget dependency. @weshaggard would there be concerns about that?

We cannot depend on that package unless we pull the building of it into our closure for our build from source efforts. That is certainly possible but it will require work and we will need to think twice before taking that dependency.

Interesting. @eerhardt are you aware of other cases where we might want to take a dependency from CoreFX onto Mono.Posix? I believe you had ideas for other Unix libraries we could offer, and those presumably would sit on Mono.Posix.

I don't think it is the plan to have CoreFX depend on Mono.Posix.NETStandard, at least it was never my intention. Mono.Posix.NETStandard is intended for code outside of the BCL that needs to call POSIX APIs. For code inside of the BCL, we have our own set of "shim" libraries (System.Native, System.Net.Http.Native, System.IO.Compression.Native, etc.) when we need to call into native libraries. I think we would follow the same pattern here.

I think we should port the System.IO.Ports native shim code from MonoPosixHelper into either a new shim library, or an existing shim library in dotnet/corefx.

[JohnRusk]

I got caught out by this. I ran the .NET Portability Analyzer over an assembly, and it said that everything used by the assembly was supported on Core. But it used System.IO.Ports.Serial, which doesn't work on *nix. Isn't that some kind of documentation, or metadata bug, somewhere, if we are reporting the API as supported, but it just doesn't work on some platforms? Shouldn't we be reporting it as not yet supported?

[danmoseley]

@terrajobst @dsplaisted (not sure who owns portability analyzer right now) ... can .NET Portability Analyzer tell you anything about Linux portability and if not is that planned?

[JairoMarques]

Any news?

[danmoseley]

@JairoMarques we do not have it on our schedule. As mentioned above - we would welcome a community effort if folks are interested along the lines of @eerhardt's suggestion above.

[KylePreuss]

This would be a wonderful thing for .NET Core apps running in Docker. If one could deploy services to a remote system and allow for serial communication with devices like GPS receivers, cellular/satellite modems, avionics, sensors etc. without having to write hardware-specific wrappers in another language/framework...

[ghost]

The significance of SerialPort has been established, there are some serious (even industrial) workloads which require serial communication and consider it as a main factor in choosing platform.

Should we ask contributors of https://github.com/mono/mono/blob/3b0c0bd1/support/serial.c and/or someone familiar with CoreFX shims can help moving the implementation forward.

@danmosemsft, @eerhardt, any suggestions on "where to start"? Does it require all-in-one-go kind of scary PR, or else what would be the best logical division of this work into smaller chunks implementable one at a time (like port discovery, port opening, port closing, sending message etc.)?

If we can lay down a checklist-y plan, based on your previous shim-driven API implementation experiences, that may lead to a successful implementation of Serial Port on Unix in near future. To me at least, it's quite overwhelming to understand https://github.com/mono/mono/blob/3b0c0bd1/support/serial.c and it's includes, then where these are being P/Invoked in mono, then translate it into CoreFX style shims and filter out unsupported platforms before begin to implement the C# mechanics.

[danmoseley]

How important is it to folks that the Unix API March the existing Windows API? Same API would mean code could be ported, but I understand the API is not that well designed for Unix semantics (I don't have details handy)

Maybe we can do a quick survey, vote this comment up if you prefer the same API AS windows , down if it doesn't matter for your scenario.

[ghost]

Doesn't matter from my point of view because existing (Windows-only) code can be ported as is, for Unix support developer can perform:

• runtime detection: switch/case over platform detection, and implement using Unix-y APIs
• compile-time detection: #if UNIX && !WIN32 kind of semantics to reduce the code size (and to make it slightly performant), if they don't necessarily need to share the assembly between windows and unix and know the target platform in advance (which is mostly the case). nuget provides platform detection first hand as well, so if dev needs productizing, they can create MyAssembly.Windows.nupkg and one for Unix (like native package creation, distribution and installation)

[wfurt]

I look at System.IO.Ports.Serial and it seems reasonable. I think ability to configure Serial port e.g. speed, parity and flow control and ability to read and write data would be good start @kasper3. After all, on Unix this is just a device file you an open. If we use tcgetattr/tcsetattr I would put them to Unix specific PAL. That should also work for macOS and BSD.

Aside from minimal implementation it would be good to find anything from the API which would not make sense on Unix or would be hart to impossible implement.

[danmoseley]

Someone want to take a shot at throwing up a minimal PR?

[wfurt]

part of Microsoft.windows.compatibility

[wfurt]

I've got basic prototype working on Ubuntu using existing System.IO.Port synchronous API. I can get/set signals, do basic configuration and send/receive data via RS-232. So far I have NOT done any testing with OSX, USB devices, Raspberry or RS-423. I think most of it should work on any Unix with exception of GetPortNames(). That currently use sysfs (/sys) on Linux. I can probably get it in in next few days if there is interest.

[JensNordenbro]

@wfurt, will this work using docker Ubuntu Images?

[wfurt]

Yes, it should as long as you can access the device files. (you can always create them with mknod) I somewhat lost momentum on this distracted with other work. I can try to bump it up on my TODO list if there would be some interest in beta testing.

related to dotnet/corefx#29033

[karbeny]

I work with serialPort in Linux like fileStream and it's working realy great.

[shaggygi]

@karbeny you have any code examples to share?

[karbeny]

Here is example of my using.... It's not complete code.

ExecuteCommand("stty 115200 raw -echo -F /dev/ttyUSB0");
FileStream fs = new FileStream("/dev/ttyUSB0",FileMode.Open);

        public static void ExecuteCommand(string command)
        {
            System.Diagnostics.Process proc = new System.Diagnostics.Process ();
            proc.StartInfo.FileName = "/bin/bash";
            proc.StartInfo.Arguments = "-c \" " + command + " \"";
            proc.StartInfo.UseShellExecute = false; 
            proc.StartInfo.RedirectStandardOutput = true;
            proc.Start ();

        }

[karbeny]

Here is implementation inside c# code.

using System;
using System.IO;
using SimpleWebServer;
using System.Net;
using System.Threading;
using System.Timers;
using System.Text;

namespace Test
{
    class Program
    {
        static string serialString = "";
        public delegate void ZobrazZpravu(byte zprava);
        public static ZobrazZpravu myDelegate;
        static Thread vlakno;
        public static FileStream fs;
        static byte[] komunikace = new byte[1024];
        static byte[] serout_buffer = new byte[64];
        static int pozice_komunikace = 0;
        static bool active_serial = true;
        static void Main(string[] args)
        {
            ExecuteCommand("stty 115200 raw -echo -F /dev/ttyUSB0");
            System.Timers.Timer aTimer = new System.Timers.Timer();
            aTimer.Elapsed += new ElapsedEventHandler(OnTimedEvent);
            aTimer.Interval = 3000;
            aTimer.Enabled = true;

            try{
            fs = new FileStream("/dev/ttyUSB0",FileMode.Open);

            myDelegate = new ZobrazZpravu(UkazKomunikaci);
            vlakno = new Thread(new ThreadStart(ReceiveMessage));
            vlakno.IsBackground = true;
            vlakno.Start();
            }
            catch {}
            WebServer ws = new WebServer(SendResponse, new string[] { "http://192.168.0.173:80/", "http://localhost:80/" });
            ws.Run();
            Console.WriteLine("Easy WebServer. Press any key for exit");
            Console.ReadKey();
            ws.Stop();
            fs.Close();
        }

        public static byte[] SendResponse(HttpListenerRequest request)
        {
            Console.WriteLine(request.RawUrl);
            Console.WriteLine(request.Url.ToString());
            Console.WriteLine(request.HttpMethod);
            if (request.RawUrl.Length > 1)
            {
                StreamReader sr = new StreamReader("web_data" + request.RawUrl);
                byte[] ret = new byte[sr.BaseStream.Length];
                for (int i = 0; i < sr.BaseStream.Length; i++)
                    ret[i] = (byte)sr.BaseStream.ReadByte();
                return ret;

            }
            return new byte[0];
        }

        private static void UkazKomunikaci(byte zprava)
        {
            serialString += (char)zprava;
            komunikace[pozice_komunikace] = zprava;
            pozice_komunikace++;
        }

        private static void ReceiveMessage()
        {

            while (active_serial)
            {
                while (active_serial)
                {
                    byte serbuff = (byte)fs.ReadByte();
                    if (serbuff > 0) UkazKomunikaci(serbuff);

                }

            }

        }
        private static void OnTimedEvent(object source, ElapsedEventArgs e)
        {
            for(int i = 0;i < pozice_komunikace;i++)
            Console.Write((char)komunikace[i]);
            fs.Write(komunikace,0,pozice_komunikace-1);
            serialString = "";
            pozice_komunikace = 0;
        }

        public static void ExecuteCommand(string command)
        {
            System.Diagnostics.Process proc = new System.Diagnostics.Process ();
            proc.StartInfo.FileName = "/bin/bash";
            proc.StartInfo.Arguments = "-c \" " + command + " \"";
            proc.StartInfo.UseShellExecute = false; 
            proc.StartInfo.RedirectStandardOutput = true;
            proc.Start ();

        }
    }
}

[haiduc32]

macOS user here. I managed to port the previous example to macOS but it's really unstable. At least with my Arduino project. I had to put some Thread.Sleep in place to make it work, which means there is a problem somewhere. I really need it for my project so I'll keep digging and post when I have something stable.

[karbeny]

you must use FileStream.flush() to stable using. In upper example fs.flush() after fs.write(.....) to send data immediately.

[haiduc32]

That's the bit of the code that writes the last byte, sleeps and flushes. Won't work without sleep (at least in my case):

            _fs.WriteByte((byte)'\n');
            Thread.Sleep(1500);
            _fs.Flush(true);

As I write that, I realised that I should try to move the sleep just before writing the first byte, and see if that still works. Maybe there is some kind of delay before the serial port can actually receive data..

p.s. all writes after this bit, are working fine without any sleep. So there seems to be some race problem when opening the serial port and writing the first data.

p.s.2 I'll build an echo sketch on my arduino, and write bytes one by one and flush immediately with a small delay between these sequences (like 100ms), and have a constant reading to see when the board actually starts receives the data (after opening the port) and can send it back.

[karbeny]

Arduino serial buffer size is 64 bytes. If you write more that 64 bytes you must optimize read function in arduino.

[haiduc32]

I send: "hello\n" And the board replies with a short greeting.

[krwq]

Hey @haiduc32 and @karbeny, could you try using the latest SerialPort bits from corefx repo?

Here is how you can consume them (this assumes you build for arm, change RuntimeIdentifier accordingly):

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
    <RootNamespace>SerialPortsApp</RootNamespace>
    <!-- this can also be passed from the command line (i.e. dotnet publish -r linux-arm) but currently it is also used -->
    <RuntimeIdentifier>linux-arm</RuntimeIdentifier>
    <!--
        To use the latest version of core-setup we can pass the version manually:
        <RuntimeFrameworkVersion>4.6.0-preview.18559.1</RuntimeFrameworkVersion>
        Latest versions of all packages used here can be found here
           (RuntimeFrameworkVersion is related to Microsoft.NETCore.App package):
            https://github.com/dotnet/versions/blob/master/build-info/dotnet/core-setup/master/Latest_Packages.txt
        Currently this is not needed because we consume private corefx.
        Once this gets public the following line and package reference to Microsoft.Private.CoreFx.NETCoreApp
        can be removed.
    -->
    <PackageConflictPreferredPackages>Microsoft.Private.CoreFx.NETCoreApp;runtime.$(RuntimeIdentifier).Microsoft.Private.CoreFx.NETCoreApp;$(PackageConflictPreferredPackages)</PackageConflictPreferredPackages>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Private.CoreFx.NETCoreApp" Version="4.6.0-preview.18559.1" />
    <PackageReference Include="System.IO.Ports" Version="4.6.0-preview.18558.3" />
  </ItemGroup>

</Project>

You will also need nuget.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="dotnet-core" value="https://dotnet.myget.org/F/dotnet-core/api/v3/index.json" />
  </packageSources>
</configuration>

and latest dotnet sdk which you can download from here: https://github.com/dotnet/core-sdk

Soon doing it should be slightly simpler as all packages get published.

I've tried talking with Arduino Circuit Playground Express from Rapsberry Pi 3 Model B and everything was working.

[haiduc32]

@krwq my pain is on macOS. I'll try it on a Raspberry Pi if I'm not lazy, but I can't promise.

Ok, so, on macOS there is indeed a delay between when I open the FileStream till I can safely send and receive the data. I've measured it at around 600ms with my Arduino. How I tested:

            _fs = new FileStream(comPortName,FileMode.Open);

            Task t = Task.Factory.StartNew(() =>
            {
                int b = 0;
                while ((b = _fs.ReadByte()) >=0)
                {
                    Console.Write((char)b);
                }
            });

            for (int i = 0; i < 20; i++)
            {
                foreach (char c in i.ToString())
                {
                    _fs.WriteByte((byte)c);
                }
                _fs.WriteByte((byte)'\n');
                _fs.Flush(true);
                Thread.Sleep(100);
            }

And what I typically get(meaning I can get even faster results sometimes):

R6
7
8
9
10
11
12
13
14
15
16
17
18
19

R - is because of timing, some bits get messed up and I get one more broken character (at random).

Now that I know that, I'll just write my code to ping the board till it gets data that makes sense. After that initial delay, everything works ok.

[wfurt]

What hardware do you have @haiduc32 ? I assume USB<->Serial adapter? There is HW complexity right there. However I did some testing on OSX and code we have should be portable. Adding OSX support would be mostly build changes.

[haiduc32]

I'm using a macbook pro connected to an Arduino UNO compatbile board, which uses a FT231X chip for serial adapter. My guess is that there is something in the way macOS handles these serial file streams. Previously, I did write a Swift app that was communicating with an arduino using a serial driver and there were no such problems, so when .net core gets proper support for serial, it should also work ok.

[shaggygi]

See https://github.com/dotnet/corefx/issues/33374