MacOS/Linux Support - Githubissues

DarthRamone commented 7 years ago

As far as I understand, it only builds on windows? Are there any plans to support other OS?

NicolasDorier commented 7 years ago

hey, sorry did not see the issue as I was not watching the project. I would like to make it run on other OS (.NET Core). But it requires me to gather the native libraries hidapi on all plateform I plan to support.

And compiling this library on all the plateform is very hard.

MelbourneDeveloper commented 6 years ago

@NicolasDorier which platforms does this library already support?

I need Android, and UWP as a starting point. Has this library been tested on those platforms?

MelbourneDeveloper commented 6 years ago

Oh. I see that it is using HidLibrary . That library only supports Windows classic from what I can tell. My Hid library is https://bitbucket.org/MelbourneDeveloper/hid.net/src/master/ (Hid.Net). It's got support for classic windows, Windows 10, and Android. I'm now in the process of refactoring this library to use Hid.Net . If anyone wants to write adapters for Linux or MacOS, it would save me the time because I have to do that anyway.

MelbourneDeveloper commented 6 years ago

@DarthRamone , as of right now, I have the library working with my library Hid.Net here: https://github.com/MelbourneDeveloper/ledger-dotnet-api

I'm now doing some more refactoring for dependency injection and it will run on Android, and UWP. All that's needed is Hid adapters for MacOS and Linux. I will keep you posted with my progress.

I've added samples for Android, and UWP, but still no luck in getting them to work. I have to go to do some non programming stuff unfortunately, but hope to be back at this soon. Meanwhile, if anyone knows anything about UWP: https://stackoverflow.com/questions/51219062/uwp-usb-hid-device-wont-connect

This is the branch of Hid.Net I'm working on to get Ledger working: https://bitbucket.org/MelbourneDeveloper/hid.net/branch/LedgerSupport

NicolasDorier commented 6 years ago

@MelbourneDeveloper does not work on mobile. HID library supports linux as well. You should give it a try.

This week I found someone who wants to spend time making it works on linux (I think it was @lontivero). Normally, it should work quite easily as far as we have the hid lib deployed on linux though.

MelbourneDeveloper commented 6 years ago

@NicolasDorier , Hi. I didn't realise that. I have tried that library. I originally wanted to use it for my project Hardfolio, but it didn't support UWP and Android, so I ended up having to write my own library. Do you know what other platforms it supports?

Would you mind taking a look at the version I've made? I spend most of this weekend porting the code to use my library Hid.Net. https://github.com/MelbourneDeveloper/ledger-dotnet-api

I haven't tried all the unit tests but CanGetWalletPubKey definitely works. The main thing you will notice is that there is an Android, and a UWP sample there. There is a permissions issue on UWP that I'm trying to get sorted out. https://stackoverflow.com/questions/51219062/uwp-usb-hid-device-wont-connect . Android is pretty close. I think there is just some quirks in the way Android deals with incoming/outgoing data. I will be able to get that fixed soon.

As for Linux, I haven't done much work with Linux, but it would be very easy to get Hid.Net working on Linux because it uses dependency injection for the platforms. Someone would just need to write the adapter. If Hidlibrary works on Linux, it would be easy enough to bring that code in to Hid.Net.

@lontivero , this is Hid.Net https://bitbucket.org/MelbourneDeveloper/hid.net/src/master/ . I'd be stoked if you'd have a look at the library and see what's involved with getting it to work on Linux.

MelbourneDeveloper commented 6 years ago

Just to give you a bit of background, Hidlibrary was the first library to come up when I first started investigating Trezor integration for my app. I would have liked to have used it, but the code doesn't have dependency injection so there is no way to differentiate between the way the different platforms connect to USB devices. Android, for example, is very different to UWP in the way you need to connect. This is my Trezor library: https://github.com/MelbourneDeveloper/Trezor.Net .

This is how device connection works on Android. It's just very different architecturally to UWP

    [Activity(Label = "@string/app_name", Theme = "@style/AppTheme.NoActionBar", MainLauncher = true)]
    [IntentFilter(new[] { UsbManager.ActionUsbDeviceAttached })]
    [MetaData(UsbManager.ActionUsbDeviceAttached, Resource = "@xml/device_filter")]
    public class MainActivity : AppCompatActivity
    {

        protected override void OnCreate(Bundle savedInstanceState)
        {
            base.OnCreate(savedInstanceState);

            _LedgerHidDevice = new AndroidHidDevice(GetSystemService(UsbService) as UsbManager, ApplicationContext, 3000, 64, 11415, 1);

            _LedgerHidDevice.Connected += _LedgerHidDevice_Connected;

            RegisterReceiver();

            SetContentView(Resource.Layout.activity_main);

            Android.Support.V7.Widget.Toolbar toolbar = FindViewById<Android.Support.V7.Widget.Toolbar>(Resource.Id.toolbar);
            SetSupportActionBar(toolbar);

            FloatingActionButton fab = FindViewById<FloatingActionButton>(Resource.Id.fab);
            fab.Click += FabOnClick;
        }

        private async void _LedgerHidDevice_Connected(object sender, EventArgs e)
        {
            try
            {
                await Task.Delay(1000);
                var ledgerTransport = new HIDLedgerTransport(_LedgerHidDevice);
                var ledgerClient = new LedgerClient(ledgerTransport);
                var asdasd = await ledgerClient.GetFirmwareVersionAsync();
                Toast.MakeText(ApplicationContext, asdasd.Major, ToastLength.Long).Show();
            }
            catch (Exception ex)
            {
                Toast.MakeText(ApplicationContext, ex.ToString(), ToastLength.Long).Show();
            }
        }
        private void RegisterReceiver()
        {
            try
            {
                lock (_ReceiverLock)
                {
                    if (_UsbDeviceAttachedReceiver != null)
                    {
                        UnregisterReceiver(_UsbDeviceAttachedReceiver);
                        _UsbDeviceAttachedReceiver.Dispose();
                    }

                    _UsbDeviceAttachedReceiver = new UsbDeviceAttachedReceiver(_LedgerHidDevice);
                    RegisterReceiver(_UsbDeviceAttachedReceiver, new IntentFilter(UsbManager.ActionUsbDeviceAttached));

                    if (_UsbDeviceDetachedReceiver != null)
                    {
                        UnregisterReceiver(_UsbDeviceDetachedReceiver);
                        _UsbDeviceDetachedReceiver.Dispose();
                    }

                    _UsbDeviceDetachedReceiver = new UsbDeviceDetachedReceiver(_LedgerHidDevice);
                    RegisterReceiver(_UsbDeviceDetachedReceiver, new IntentFilter(UsbManager.ActionUsbDeviceDetached));
                }
            }
            catch (Exception ex)
            {
            }
        }

    }

UWP has a poller that waits for a device to be connected:

    public sealed partial class MainPage : Page
    {
        public MainPage()
        {
            this.InitializeComponent();
        }

        UWPHidDevice uwpHidDevice;
        LedgerClient ledgerClient;

        private void button_Click(object sender, RoutedEventArgs e)
        {

            var uwpHidDevice = new UWPHidDevice();
            new UWPHidDevicePoller(1, 11415, uwpHidDevice);
            uwpHidDevice.Connected += UwpHidDevice_Connected;

        }

        private async void UwpHidDevice_Connected(object sender, EventArgs e)
        {
            await new MessageDialog("Connected").ShowAsync();
        }

        private async void button1_Click(object sender, RoutedEventArgs e)
        {
            await Task.Delay(1000);
            await uwpHidDevice.InitializeAsync();
            var ledgerTransport = new HIDLedgerTransport(uwpHidDevice);
            ledgerClient = new LedgerClient(ledgerTransport);
            var asdasd = await ledgerClient.GetFirmwareVersionAsync();
        }
    }

MelbourneDeveloper commented 6 years ago

YES! @NicolasDorier @DarthRamone , I have an Android sample working. To see the sample:

You will need an OTG cable for your Android.

Clone https://github.com/MelbourneDeveloper/ledger-dotnet-api.git
Plug in the Ledger (I'm using Nano with vendor 11415, pid 1). You might need to change the code for a different Ledger device.
Enter your pin and open the Bitcoin app
Compile and run the Android sample
It should show the firmware version on start up

Or

Clone https://github.com/MelbourneDeveloper/ledger-dotnet-api.git
Compile and run the Android sample
Plug in the Ledger (I'm using Nano with vendor 11415, pid 1). You might need to change the code for a different Ledger device.
Enter your pin and open the Bitcoin app
It should show the firmware version

Note: The vid and pid are specified in this code in the MainActivity _LedgerHidDevice = new AndroidHidDevice(GetSystemService(UsbService) as UsbManager, ApplicationContext, 3000, 64, 11415, 1);

MelbourneDeveloper commented 6 years ago

The UWP problem actually looks like a problem with the Windows 10 SDK. The Ledger team is probably going to hit up against the same problem if they are trying to build for UWP. I've logged an issue here:

https://github.com/tpn/winsdk-10/issues/2

MelbourneDeveloper commented 6 years ago

Can anyone point me to a good Linux distro that I can use to start testing on Linux? Will try to do this next.

NicolasDorier commented 6 years ago

Wow this is so awesome! I should include that to the library. Well go on ubuntu. Normally all is needed is recompiling the hid lib for linux.

MelbourneDeveloper commented 6 years ago

@NicolasDorier lets work together on this. I'm hellbent on getting Hardfolio (https://play.google.com/store/apps/details?id=com.Hardfolio) to work on all platforms, with all hardware wallets. So, I'll do whatever is necessary to make sure the library works with Linux and MacOS as well as the other platforms.

NicolasDorier commented 6 years ago

I can't really help because I don't have the platforms where I can test. :(

Do you want me to merge some stuff you already did though?

MelbourneDeveloper commented 6 years ago

@NicolasDorier that would be great, but I don't want you to break any existing stuff. You might want to do a code review of what I've done? I will need to go over the unit tests to make sure they pass.

NicolasDorier commented 6 years ago

You are doing awesome stuff, I also needed a lib for Trezor. I'll review, see what you did and check if I can just publish a nuget package which target all the plateforms you want.

MelbourneDeveloper commented 6 years ago

I will also spend some time fixing the unit tests that I have broken as the next step.

MelbourneDeveloper commented 6 years ago

A few updates.

Apparently HidLibrary doesn't support Linux. https://github.com/mikeobrien/HidLibrary/issues/103 . I did find another library that does support Linux, but I can't vouch for how easy it is to get working: https://github.com/LibUsbDotNet/LibUsbDotNet.

This is another project for Linux but looks older: http://libusbdotnet.sourceforge.net/V2/Index.html

This library seems to have MacOS support: https://github.com/treehopper-electronics/HIDSharp but I have not dug in to it yet.

Hid.Net was designed with Task based async in mind, so I realised that the synchronous code won't work correctly. I'm wondering if I need to refactor Hid.Net to support synchronous code... Not sure. But, for now I have changed the unit tests so that they use task based async. I'm working on getting all the unit tests to pass. I had success with signing transactions today.

A-Manning commented 6 years ago

@MelbourneDeveloper Please make a PR as soon as possible! Linux support would be really helpful

NicolasDorier commented 6 years ago

@MelbourneDeveloper HidLibrary is small enough to consider copy/pasting it instead of depending on their package. Don't let it stop you.

That said, you don't normally need this. HidLibrary is normally nothing more than a wrapper around hidlib. If you run ledger-dotnet-api on linux, you should have an error about hidlib.so not being found.

I think if you configure linux to find the hid.so with ld, it should just work out of the box.

MelbourneDeveloper commented 6 years ago

@A-Manning , I'm working on this. Unfortunately, there's no clear path to getting Linux working. As @NicolasDorier points out, HidLibrary is a wrapper for hidlib, but it is mixed in with very Windows specific code like this:

    [DllImport("kernel32.dll")]
    static internal extern bool WriteFile(IntPtr hFile, byte[] lpBuffer, uint nNumberOfBytesToWrite, out uint lpNumberOfBytesWritten, [In] ref System.Threading.NativeOverlapped lpOverlapped);

It would probably be possible to get HidLibrary working on Linux, but I'd need to do the research for finding the replacement calls to do IO on Linux. It's something I can and will do in time, but Android, and UWP support were challenging enough by themselves.

HidLibrary is small enough to consider copy/pasting it instead of depending on their package. Don't let it stop you.

Yep. You are right. That's how I traced the transfer between the Ledger and replicated it with Hid.Net. Don't worry, this won't stop me.

I think if you configure linux to find the hid.so with ld, it should just work out of the box.

That's the long term plan. Meanwhile, I have to set up a virtual machine, and sift through working C# samples and so on. Linux and MacOS are not my development area of expertise. Android, and UWP were fairly straight forward to implement because there are clear, documented C# libraries with working samples.

Android (Xamarin) https://developer.xamarin.com/api/field/Android.Hardware.Usb.UsbClass.Hid/

UWP https://docs.microsoft.com/en-us/uwp/api/windows.devices.humaninterfacedevice

You will notice that these libraries are extremely different in there architecture and approach. Hid.Net is an attempt to put a layer over all three platforms to allow for dependency injection. It's the only way I could get Hardfolio to have exactly the same code between Android, and UWP. Here are some snippets from Hardfolio.

    /// <summary>
    /// Adds an address to the collection from the Trezor
    /// </summary>
    private async Task AddTrezorAddress(CoinInfo coinInfo, CurrencyHolding currencyHolding, Collection<AddressAndPathInfo> addresses, TrezorManager.AddressType trezorAddressType, uint i, bool isChange)
    {
        var blockChainAddressInformation = currencyHolding?.BlockChainAddresses.FirstOrDefault(bca => bca.Index.HasValue && bca.Index.Value == i && bca.IsChange == isChange);
        if (blockChainAddressInformation != null)
        {
            addresses.Add(new AddressAndPathInfo(i, isChange, blockChainAddressInformation.Address));
            return;
        }

        ccn.Logger.Log("Trezor required...", null, GetHoldingLogSection);

        if (!await GetIsTrezorConnected())
        {
            ccn.Logger.Log("Trezor manager says Trezor is not connnected", null, GetHoldingLogSection);
            _IsWaitingForTrezor = true;
            TrezorConnectionRequired?.Invoke(this, new EventArgs());
            return;
        }

        if (!_IsTrezorInitialized)
        {
            await _TrezorManager.InitializeAsync();
        }

        _IsTrezorInitialized = true;
        var address = await _TrezorManager.GetAddressAsync(coinInfo.Symbol, coinInfo.CoinNumber.Value, isChange, i, false, trezorAddressType);
        addresses.Add(new AddressAndPathInfo(i, isChange, address));
    }

The parts that are different are here which is the connection part which is achieved with dependency injection:

UWP:

            var uwpHidDevice = new UWPHidDevice();
            new UWPHidDevicePoller(TrezorManager.TrezorProductId, TrezorManager.TrezorVendorId, uwpHidDevice);

            LoadApplication(new p.App(new CrossPlatformUtilities(new IsolatedStoragePersister(), new UWPRESTClientFactory(new NewtonsoftSerializationAdapter())), uwpHidDevice, GetPin));

Android:

   _TrezorHidDevice = new AndroidHidDevice(GetSystemService(UsbService) as UsbManager, ApplicationContext, 3000, 64, TrezorManager.TrezorVendorId, TrezorManager.TrezorProductId);

                 RegisterReceiver();

                 var application = new App(new CrossPlatformUtilities(new IsolatedStoragePersister(), new AndroidRESTClientFactory(new NewtonsoftSerializationAdapter())), _TrezorHidDevice, GetPin);

                 LoadApplication(application);

Hid.Net was designed with the assumption that it would only ever support Task based async. The reasoning for this was that UWP does not allow for synchronous IO calls, and therefore, issues would arise if I tried to bend the library to force UWP to work in synchronous mode. Saying that though, I do not wish to impinge on the integrity of your original library @NicolasDorier , so no matter what I do, I will be mindful to work toward and end result that is acceptable for you.

Unfortunately @A-Manning and @NicolasDorier , I will not be submitting a pull request any time soon. I will continue to expand on what I've got, and continue researching until I can get Hid.Net to work on Linux and MacOS. However, getting Ledger support in Hardfolio on Android, and UWP is my primary goal right now, so it may be some time before I'm able to dust something enough that is deserving of a pull request. Help would be much appreciated if anyone is willing to give it.

My only question @NicolasDorier is: given that UWP does not really support synchronous operations easily, does Ledger Wallet Client API need to support synchronous operations? Hid.Net was built for UI based applications only. Will Ledger Wallet Client API be used for back end operations? If so, I will need to refactor Hid.Net to include synchronous operations.

NicolasDorier commented 6 years ago

How do you manage to try on Linux via Virtual Machine? I never managed to access my ledger from VMs.

does Ledger Wallet Client API need to support synchronous operations

Normally the synchronous calls are just wrapper on top of async one just for convenience. Everything should be async based, if that's not the case I must refactor it.

NicolasDorier commented 6 years ago

Another approach I am using for ledger support on cross platform, but it might not work for your case:

Ledger expose a U2F interface that browser can use. I have a project somewhere (I can give it to you if interested), which is a bridge Websocket <-> U2F. Then server side, I use a custom Transport which send the APDU of ledger via the Websocket.

If your app is Electron based, this might do the trick.

MelbourneDeveloper commented 6 years ago

How do you manage to try on Linux via Virtual Machine?

@NicolasDorier , actually I have never tried it. I assumed it worked, but maybe not. :(

Normally the synchronous calls are just wrapper on top of async one just for convenience. Everything should be async based, if that's not the case I must refactor it.

Perfect. It's OK. I already refactored it. Because you said that, I changed my mind, and I have submitted the pull request: https://github.com/LedgerHQ/ledger-dotnet-api/pull/11

Ledger expose a U2F interface that browser can use. I have a project somewhere (I can give it to you if interested), which is a bridge Websocket <-> U2F. Then server side, I use a custom Transport which send the APDU of ledger via the Websocket.

This sounds very promising. My app is purely C# based. But, I am considering building a https transport for Trezor and Ledger so that iPhone apps can use these devices. So, the user would plug in the device in to a desktop, and they could use the Hardfolio app on their phones. This is a long way away but I'd be interested to see what other people are doing.

MelbourneDeveloper commented 6 years ago

@NicolasDorier , what do you think about this problem?

It's pretty much impossible to write a platform agnostic method to enumerate Hid devices. If it is possible, it's not really the recommended approach.

For example, this is the normal approach with Android. You have to set up a "Receiver", and then check to see if the device is the one you are looking for when it is attached, as well as detach it later.

UWP is completely different again. Hid.Net doesn't try to put a layer over this on each platform. It just gives you helpers for each platform so you can do it like this:

Windows

            var vid = (ushort)11415;
            var devices = WindowsHidDevice.GetConnectedDeviceInformations();
            var newDevice = devices.Where(d => d.VendorId == vid).ToList()[0];
            var windowsHidDevice = new WindowsHidDevice(newDevice);
            windowsHidDevice.DataHasExtraByte = true;
            await windowsHidDevice.InitializeAsync();
            var ledgerTransport = new HIDLedgerTransport(windowsHidDevice);

UWP

        private void button_Click(object sender, RoutedEventArgs e)
        {
            var uwpHidDevice = new UWPHidDevice();
            new UWPHidDevicePoller(1, 11415, uwpHidDevice);
            uwpHidDevice.Connected += UwpHidDevice_Connected;
        }

        private async void button1_Click(object sender, RoutedEventArgs e)
        {
            await Task.Delay(1000);
            await uwpHidDevice.InitializeAsync();
            var ledgerTransport = new HIDLedgerTransport(uwpHidDevice);
            ledgerClient = new LedgerClient(ledgerTransport);
            var asdasd = await ledgerClient.GetFirmwareVersionAsync();
        }

Android

    private void RegisterReceiver()
        {
            try
            {
                lock (_ReceiverLock)
                {
                    if (_UsbDeviceAttachedReceiver != null)
                    {
                        UnregisterReceiver(_UsbDeviceAttachedReceiver);
                        _UsbDeviceAttachedReceiver.Dispose();
                    }

                    _UsbDeviceAttachedReceiver = new UsbDeviceAttachedReceiver(_LedgerHidDevice);
                    RegisterReceiver(_UsbDeviceAttachedReceiver, new IntentFilter(UsbManager.ActionUsbDeviceAttached));

                    if (_UsbDeviceDetachedReceiver != null)
                    {
                        UnregisterReceiver(_UsbDeviceDetachedReceiver);
                        _UsbDeviceDetachedReceiver.Dispose();
                    }

                    _UsbDeviceDetachedReceiver = new UsbDeviceDetachedReceiver(_LedgerHidDevice);
                    RegisterReceiver(_UsbDeviceDetachedReceiver, new IntentFilter(UsbManager.ActionUsbDeviceDetached));
                }
            }
            catch (Exception ex)
            {
            }
        }

MelbourneDeveloper commented 6 years ago

What I am saying is: HidLibrary gives you the tools to enumerate devices and then connect to them, but it only works on one platform: Windows. Hid.Net gives you the tools to connect on each platform, but the code is different for each platform because the connection mechanism is very different.

MelbourneDeveloper commented 5 years ago

@NicolasDorier just an update on this. There's a lot of activity going on at the moment. Trezor have switched from Hid to vanilla USB. Hid is going to be broken up in to three parts:

Device.Net Hid.Net Usb.Net

I am also working hard on getting device enumeration easier across platforms (https://github.com/MelbourneDeveloper/Hid.Net/issues/9). I'll be in touch about all this when all the changes are settled down. Basically, I think I will be implementing your suggestions here.

NicolasDorier commented 5 years ago

@MelbourneDeveloper the way I intend to use your lib is by routing HID packets to websocket, then the websocket will just use the trezor bridge. So it is my own transport.

MelbourneDeveloper commented 5 years ago

@DarthRamone , @NicolasDorier I'd like to upgrade the Hid Nuget package to the latest version. There has been a lot of work on this framework here: https://github.com/MelbourneDeveloper/Device.Net

I now have a library which wraps LibUsbDotNet for MacOS and Linux. It's always been possible to use that, but I've written framework so that the connection code is the same across all platforms. This is some example code for UWP to get connected devices:

        //Register the factory for creating Usb devices. This only needs to be done once.
        UWPUsbDeviceFactory.Register();

        //Register the factory for creating Usb devices. This only needs to be done once.
        UWPHidDeviceFactory.Register();

        //Define the types of devices to search for. This particular device can be connected to via USB, or Hid
        var deviceDefinitions = new List<FilterDeviceDefinition>
        {
            new FilterDeviceDefinition{ DeviceType= DeviceType.Hid, VendorId= 0x534C, ProductId=0x0001, Label="Trezor One Firmware 1.6.x" },
            new FilterDeviceDefinition{ DeviceType= DeviceType.Usb, VendorId= 0x1209, ProductId=0x53C1, Label="Trezor One Firmware 1.7.x" },
            new FilterDeviceDefinition{ DeviceType= DeviceType.Usb, VendorId= 0x1209, ProductId=0x53C0, Label="Model T" }
        };

        //Get the first available device and connect to it
        var devices = await DeviceManager.Current.GetDevices(deviceDefinitions);

As you can see though, you have to call Register on the platform's factory. That's how the latest version works. This would be a breaking change for existing code because the factories won't enumerate the devices if they are not registered. The reason you have to call this is because you only want to register the factory that is useful for your platform. There's no sense in registering the Windows factories on Android, and it would crash anyway.

If want me to go ahead with this upgrade I can start putting the pull request together. It will basically mean UWP, Windows, Android, MacOS, and Linux are all covered and are easy to get connected. Also, there will probably be a library to slot in when Ledger Nano X comes out. Device.Net is abstract so if we need a Bluetooth library, it can be added.

NicolasDorier commented 5 years ago

@MelbourneDeveloper this kind of global registration is an anti pattern. Easy to shoot yourself in the foot.

Better way would be if on UWP someone could just do UWPHidDeviceFactory.CreateHIDDeviceAsync()/EnumerateAsync() on an instance of such factory.

MelbourneDeveloper commented 5 years ago

@NicolasDorier, that's no problem. But, either way, it requires a platform specific call which is not how the Ledger Dot Net API works. It's not possible to avoid making a platform specific call, unless the Ledger Dot Net API is split in to separate assemblies for each platform. The main question is whether or not you are OK with that?

NicolasDorier commented 5 years ago

Does it make sense to update at all this library? Your libraries seems more complete, and you provide a common abstraction layer right?

NicolasDorier commented 5 years ago

Do you have twitter or somewhere we can chat? Considering using your project even for BTCPay instead of my lib.

MelbourneDeveloper commented 5 years ago

@NicolasDorier yes, please hit me up on Twitter and I will invite you to the slack channel.

https://twitter.com/cfdevelop

We are both working toward the same goals, so if we communicate when get their faster together.

LedgerHQ / ledger-dotnet-api

MacOS/Linux Support #2