Linux: Internal process error (null)

[suvayu]

Ledger Live Version and Operating System

• Ledger Live 1.0.1
• Platform: Linux

Expected behavior

I want to add my existing account for my Ledger Nano S. Following which, I should be able to sync and access my wallet.

Actual behavior

When I follow the steps to add an account, I get the following error.

Something went wrong during synchronisation, please try again.
Internal process error (null)

There is no backtrace in the terminal.

Steps to reproduce the behavior

1. Click add an account
2. Select the blockchain (Bitcoin, Litecoin, etc)
3. Unlock your Ledger device with your pin
4. Open the corresponding app on the Ledger device
5. Press continue

Screenshots

after step (4)

after step (5)

[meriadec]

can you provide the output of

uname -a

[suvayu]

$ uname -a
Linux hostname 4.17.2-200.fc28.x86_64 #1 SMP Mon Jun 18 20:09:31 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[bocekm]

Duplicate to https://github.com/LedgerHQ/ledger-live-desktop/issues/950

[gre]

are you able to import your accounts on the same computer but with the Ledger Bitcoin Chrome app ?

[suvayu]

I have not tried importing my accounts with the chrome app (as I have never wiped my wallet), but I have been successfully using the Chrome app and Electrum ever since I initialised it after my purchase.

[snaggen]

I have this exact issue on Ledger Live 1.0.1 on Fedora Linux 28. However, the chrome app works fine.

[drizzt]

Hi, after a poor-man debugging I found that the problem is fixable by using the following libraries from Ubuntu 18.04: libgssapi_krb5.so.2, libk5crypto.so.3, libkrb5.so.3 and libkrb5support.so.0 I attached a script that downloads the last (1.0.2) AppImage and repack it by including the 4 needed shared libraries (taked from Ubuntu 18.04): ledger-live-desktop-fedora-28-workaround.tar.gz

To use it just run:

curl -fOL https://github.com/LedgerHQ/ledger-live-desktop/files/2193653/ledger-live-desktop-fedora-28-workaround.tar.gz
tar xf ledger-live-desktop-fedora-28-workaround.tar.gz
cd ledger-live-desktop-fedora-28-workaround
./runme.sh
./ledger-live-desktop-1.0.2-patched-linux-x86_64.AppImage

And you'll found the "patches" appimage as ledger-live-desktop-1.0.2-patched-linux-x86_64.AppImage

[bocekm]

The same issue is still happening even with v1.0.2.

[daktak]

drizzt's script worked for me on Qubes-OS 3.2, Fedora 28 appvm.

[suvayu]

@daktak, I don't know if you realise, but @drizzt's workaround replaces the libraries on your system with his version of these libraries. This might put your funds at risk unless you trust @drizzt completely. If you want to use his workaround, I suggest you look at the runme.sh script in his tarball, and replace the libraries with your own version (essentially a version that you can trust).

[meriadec]

@drizzt thx for investigating. We'll either integrate something similar to our build chain (pathing the .AppImage), or build different releases by distribution. The optimal flow will be to directly push build scripts into distribution repositories (binaries or not) and let users compile the app via their package manager.

[drizzt]

@suvayu I changed my script to take the libraries from Ubuntu directly, so you shouldn't trust me: https://gist.githubusercontent.com/drizzt/f95444586650f07444b03058a96d2de9/raw/ledger-live-desktop-fedora-28-workaround.sh

[suvayu]

@drizzt: do you have an idea what is the incompatibility between the libraries from Ubuntu 18.04 vs Fedora 28?

[meriadec]

@bocekm can you try with this version, compiled on an Ubuntu 16.04 virtual machine? https://drive.google.com/file/d/1S3PCV36Th6BMHtzXrxhOkTbeaqomYo15

[suvayu]

I tried, same issue.

[eric-vader]

Same issue on Linux desktop 4.17.7-200.fc28.x86_64 #1 SMP Tue Jul 17 16:28:31 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[snaggen]

One solution to this issue is to distribute it using flatpak, since that would make sure to use the exact same environment.

[drizzt]

@snaggen currently electron-builder doesn't support flatpak, moreover I don't think flatpak cannot access hidraw so easly. I think using snap can be a better solution since it supports hidraw and it's supported by electron-builder natively

[snaggen]

Not sure about hidraw, but I think flatpak supports direct access to devices... Isn't that enough? The good thing with using flatpaks for this kind of project is that you have the runtimes to build on. That gives you a clear sdk to target and still a relatively small package to distribute. I know the endless guys have packaged a few electron apps like signal and slack, so it is doable and there are examples. But of course it would be better with direct electron builder support. I haven't looked that deep into snaps, so I can't really comment on the advantage of using that.

[snaggen]

Electron doesn't seem very difficult with flatpak either, there are some docs describing the process http://docs.flatpak.org/en/latest/electron.html

[meriadec]

Successfully installed app on Fedora 28, fresh install (adding udev rule) successfully synchronized LTC(and other altcoins)/ETH/XRP + LTC transaction. Closing for now, if the pb persists we may need additional informations to reproduce from a fresh install.

[suvayu]

Hi @meriadec,

Well, it's still unresolved here (I'm the original reporter).

if the pb persists we may need additional informations to reproduce from a fresh install.

It's quite frustrating as there is no clear way to provide more useful debug info. In all associated bugs (#933, #950, etc), the messages are all along the lines of "Something went wrong ... Internal process error" (even in the log files). There's also a clear lack of instructions or documentation on how to produce effective debug information.

For example, @drizzt's workaround points to some kind of library incompatibility, but there has been no explanation from any of the devs as to what that might be. There's a lot the dev team can do here.

1. Produce debug builds with actual useful error messages that the reporter (myself) can test and provide feedback.
2. Provide instructions how to produce more verbose debug information to augment the bug report, e.g. I see there's a host of DEBUG* environment variables mentioned in the README, but without any guidance the reporter (me) is none-the-wiser.
3. The least you can do is to recommend a workaround (like the one above), with some explanations so that the reporter can choose what's best for them.

FWIW, just making the codebase FOSS isn't enough, you have to engage with the community to make your product better. Anyway, I hope you heed this message, otherwise I'll have to look for a different hardware wallet in the future.

[meriadec]

Hey @suvayu, you're totally right, and apologizes if in any way it looked like ignoring the problem (our biggest problem here is time, because we have many things to focus on, and have to prioritize stuff).

Anyway, your message has been received.

The actual problem with "internal process error" is...we don't have much additional information on our side either, because the C++ libcore (which run in separate process) isn't really providing stacktraces, so it's quite hard to debug (and hardest if we don't reproduce).

To iterate faster, if it's possible on your side to use the app in dev mode (cf README yarn start), activating any DEBUG_* env vars it will be very appreciated.

Let's re-open that thing. The goal wasn't to create any frustration or delegate the pb :)

[suvayu]

Hi @meriadec, thank you for receiving my comments constructively.

our biggest problem here is time, because we have many things to focus on, and have to prioritize stuff

Since I can still use Electrum (for BTC and LTC), and the old Chrome apps for everything, I am perfectly fine with a longer time line.

if it's possible on your side to use the app in dev mode ...

I will try build my own, run in dev mode, and report back soon.

Thanks again for listening

[suvayu]

I tried the following to no avail:

1. Running with DEBUG_* environment variables set didn't yield any new information (my logs)
2. Since you mentioned the error is happening in the C++ layer, I ran it under strace to identify the library (I was expecting open calls would help me identify that), didn't help. Looking in node_modules I guess it's libledger-core.so.
3. I then tried launching Ledger Live under GDB, but that prooved troublesome. If I knew the PID of the "internal process", I could attach to it using gdb -p <pid>, but the pid doesn't show up in the logs until I close the app (see logs in 1). Even if I knew the process command name, I could use pgrep. I tried using htop, but I couldn't quite figure it out. I guess it's one of the electron processes?

Any further guidance would be helpful.

[bocekm]

I also yarn started it with the DEBUG envvars and I'm getting the same output in terminal as @suvayu. I'm adding a screenshot of an error from the developer console but it probably won't help either.

[meriadec]

The internal process is created here: https://github.com/LedgerHQ/ledger-live-desktop/blob/0ff2aa618f5c06d9e34d52460d42ebb93e168b45/src/main/bridge.js#L54-L65

(we definitely should add its pid to logs.. but you can log in from here as quick workaround :smile: thanks for taking your time for that! :+1: )

@bocekm the difficulty is that the stacktrace isn't going further on JS side when we deserialize it https://github.com/LedgerHQ/ledger-live-desktop/blob/0ff2aa618f5c06d9e34d52460d42ebb93e168b45/src/helpers/ipc.js#L64-L69

[meriadec]

I confirm that the libcore for Linux is libledger-core.so, located in the @ledgerhq/ledger-core node module

[suvayu]

Hi @meriadec, I think I made some headway.

Here are my findings (tested with the bitcoin wallet):

1. It's a segfault happening inside libcrypto (from openssl-libs).
2. My guess is it's happening when secp256k1 makes a call to libcrypto, probably in the file bench_verify.c (because that's where I found #include <openssl/..>). I have no clue what it does, but makes sense since I was testing with the bitcoin wallet. I didn't proceed further as I wasn't sure how I can put breakpoints inside C-source files in node modules.

gdb caught the segfault here:

Thread 1 "Ledger Live Int" received signal SIGSEGV, Segmentation fault.
0x00007fa645d33b89 in EVP_MD_CTX_clear_flags (ctx=ctx@entry=0x0, flags=flags@entry=2) at crypto/evp/evp_lib.c:479
479         ctx->flags &= ~flags;

As I don't know any javascript, nor am I familiar with either the design of Ledger Live, or any of the dependent libraries like secp256k1 and openssl-libs, I cannot hazard a guess how libledger-core.so might be triggering the segfault when calling these. I would backtrack the segfault by placing breakpoints first in the relevant points in secp256k1, and then follow it up the stack to libledger-core.so.

A few notes for whoever wants to attach gdb to the internal process.

1. you need not edit the source to find out the pid of the internal process, you may find it with pgrep 'Ledger Live Int'. That said, the logging does need this information, and probably others.
2. you would also need debug symbols for the C/C++ libraries. It's easy for system libraries (dnf debuginfo-install <package> on Fedora), but I'm not sure what's the process for libraries in node modules (secp256k1, libledger-core.so, etc).

I hope this helps. Please let me know if I can provide any other information.

[snaggen]

I think Fedora might compile openssl without some ecliptic curve implementations due to patent concerns or something. So, you might need to provide your own openssl libs if that is the case.

[suvayu]

@snaggen do you know how I could check this?

[drizzt]

@snaggen it shouldn't be a problem of openssl (since I tried to use the openssl ubuntu shared library). the libraries that "workaround" the problem are the kerberos libraries (libgssapi_krb5.so.2, libk5crypto.so.3, libkrb5.so.3 and libkrb5support.so.0). I also found that the AppImage tries to load the libraries also for the local directory and so if you download the 4 kerberos libraries from Ubuntu 18.04 and you put them inside the same directory of the AppImage it works. I already tried to re-build libledger-core.so on Fedora, but it crashed with the same error

To download the libraries you may use (console) something like:

for package in libgssapi-krb5-2 libk5crypto3 libkrb5-3 libkrb5support0; do curl -s "https://packages.ubuntu.com/bionic/amd64/$package/download" | sed -n 's:\s*<li><a href="\([^"]*\)">.*:\1:p' | head -n1 | xargs -n1 curl -fOL; ar p "$package"_*_amd64.deb data.tar.xz | tar xvJf - --strip-components 4 ./usr/lib/x86_64-linux-gnu/; done

[snaggen]

Ok, my guess is then that Fedora have the same kind of restrictions on the kerberos package. I can't verify this myself right now, but install the kerberos src package and check the spec file for the build parameters they use.

Den lör 28 juli 2018 17:37Timothy Redaelli notifications@github.com skrev:

@snaggen https://github.com/snaggen it shouldn't be a problem of openssl (since I tried to use the openssl ubuntu shared library). the libraries that "workaround" the problem are the kerberos libraries (libgssapi_krb5.so.2, libk5crypto.so.3, libkrb5.so.3 and libkrb5support.so.0). I also found that the AppImage tries to load the libraries also for the local directory and so if you download the 4 kerberos libraries from Ubuntu 18.04 and you put them inside the same directory of the AppImage it works. I already tried to re-build libledger-core.so on Fedora, but it crashed with the same error

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LedgerHQ/ledger-live-desktop/issues/1010#issuecomment-408615992, or mute the thread https://github.com/notifications/unsubscribe-auth/AGH4nzAxFhSQBWv06KqqL2PgitFpdUJJks5uLIUlgaJpZM4VJGlV .

[suvayu]

@snaggen okay, here's definitive proof that that's not the case:

$ openssl ecparam -list_curves
secp224r1 : NIST/SECG curve over a 224 bit prime field
secp256k1 : SECG curve over a 256 bit prime field        # <- the relavant one
secp384r1 : NIST/SECG curve over a 384 bit prime field
secp521r1 : NIST/SECG curve over a 521 bit prime field
prime256v1: X9.62/SECG curve over a 256 bit prime field

[suvayu]

Hi @meriadec, after having a closer look at the source I would like to withdraw my point (2) about secp256k1.

Here's a more complete backtrace from the segfault:

(gdb) bt 6
#0  0x00007f8d27f46b89 in EVP_MD_CTX_clear_flags (ctx=ctx@entry=0x0, flags=flags@entry=2) at crypto/evp/evp_lib.c:479
#1  0x00007f8d27f36d21 in EVP_DigestInit_ex (ctx=0x0, type=0x7f8d2825eb40 <sha512_md>, impl=0x0) at crypto/evp/digest.c:66
#2  0x00007f8d27f54c3a in HMAC_Init_ex (ctx=0x7ffd0a9792c0, key=<optimized out>, len=<optimized out>, md=<optimized out>, impl=0x0) at crypto/hmac/hmac.c:70
#3  0x00007f8d1e4b0a48 in ledger::core::HMAC::sha512(std::vector<unsigned char, std::allocator<unsigned char> > const&, std::vector<unsigned char, std::allocator<unsigned char> > const&) ()
    at /scratch/build/bitcoin/ledger-live-desktop/node_modules/@ledgerhq/ledger-core/build/Release/libledger-core.so
#4  0x00007f8d1e4ac43a in ledger::core::DeterministicPublicKey::derive(unsigned int) const ()
    at /scratch/build/bitcoin/ledger-live-desktop/node_modules/@ledgerhq/ledger-core/build/Release/libledger-core.so
#5  0x00007f8d1e4614e2 in ledger::core::_derive(int, std::vector<unsigned int, std::allocator<unsigned int> > const&, ledger::core::DeterministicPublicKey const&) ()
    at /scratch/build/bitcoin/ledger-live-desktop/node_modules/@ledgerhq/ledger-core/build/Release/libledger-core.so
#6  0x00007f8d1e4617de in ledger::core::BitcoinLikeExtendedPublicKey::derive(ledger::core::DerivationPath const&) ()                                        
    at /scratch/build/bitcoin/ledger-live-desktop/node_modules/@ledgerhq/ledger-core/build/Release/libledger-core.so

It's a bit hard to go through computer-generated code (I presume that's what ledger-core is); I couldn't quite find the relevant lines in the source. That said, if you look at stack #2, you will see a valid context is passed on to libcrypto, however further down the stack it becomes a null_ptr. My first thought is that the context goes out of scope or is garbage collected. Without more familiarity with the code, I'm afraid this is as far as I can go.

[meriadec]

@suvayu thx for the backtrace, can you give your installed versions of openssl and libcrypto ? so we install the same on our side.

[suvayu]

$ rpm -qa openssl{,-libs}
openssl-libs-1.1.0h-3.fc28.i686
openssl-1.1.0h-3.fc28.x86_64
openssl-libs-1.1.0h-3.fc28.x86_64
$ rpm -ql openssl-libs | grep libcrypto
/usr/lib64/.libcrypto.so.1.1.0h.hmac
/usr/lib64/.libcrypto.so.1.1.hmac
/usr/lib64/libcrypto.so.1.1
/usr/lib64/libcrypto.so.1.1.0h
/usr/lib/.libcrypto.so.1.1.0h.hmac
/usr/lib/.libcrypto.so.1.1.hmac
/usr/lib/libcrypto.so.1.1
/usr/lib/libcrypto.so.1.1.0h

[meriadec]

Here is output on my side

$ rpm -qa openssl{,-libs}
openssl-libs-1.1.0h-3.fc28.x86_64
openssl-1.1.0h-3.fc28.x86_64
$ rpm -ql openssl-libs | grep libcrypto
/usr/lib64/.libcrypto.so.1.1.0h.hmac
/usr/lib64/.libcrypto.so.1.1.hmac
/usr/lib64/libcrypto.so.1.1
/usr/lib64/libcrypto.so.1.1.0h

Only thing I notice is on my side I only have the x64 versions, I wonder if there is a chance that ledger-core is not using them in favor of i686 ones, and that may cause the trouble (I'm not used to this kind of pb so I'm not sure it would makes sense..)

[suvayu]

Hmm, I removed the 32 bit openssl-libs and tried, no luck :-|. This is baffling.

[fansari]

I can confirm that Ledger Live 1.0.7 is not working with Fedora 28 for Bitcoin.

[fansari@bat ledger]$ openssl ecparam -list_curves secp224r1 : NIST/SECG curve over a 224 bit prime field secp256k1 : SECG curve over a 256 bit prime field secp384r1 : NIST/SECG curve over a 384 bit prime field secp521r1 : NIST/SECG curve over a 521 bit prime field prime256v1: X9.62/SECG curve over a 256 bit prime field

[fansari@bat ledger]$ rpm -ql openssl-libs | grep libcrypto /usr/lib64/.libcrypto.so.1.1.0h.hmac /usr/lib64/.libcrypto.so.1.1.hmac /usr/lib64/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1.0h /usr/lib/.libcrypto.so.1.1.0h.hmac /usr/lib/.libcrypto.so.1.1.hmac /usr/lib/libcrypto.so.1.1 /usr/lib/libcrypto.so.1.1.0h

[bocekm]

Because the Ledger Live works with the Ubuntu's krb5 libraries as @drizzt mentions above, I've checked the difference between the Ubuntu and Fedora krb5 packages. On Ubuntu the version is 1.16-2build1 and on Fedora it's 1.16.1-17. From the Ubuntu's changelog it states "No change rebuild against openssl1.1." What I get from that is that the Fedora's krb5 package is not built against openssl-1.1.

[richardsaloga]

v1.1.0-rc.3

[suvayu]

@bocekm that doesn't make sense, installing multiple versions of openssl isn't possible. In any case, the transcript below definitively proves that's not the case.

$ rpm -q krb5-libs
krb5-libs-1.16.1-13.fc28.x86_64
krb5-libs-1.16.1-13.fc28.i686
$ rpm -q openssl-libs
openssl-libs-1.1.0h-3.fc28.x86_64
openssl-libs-1.1.0h-3.fc28.i686
$ rpm -ql krb5-libs | grep crypto
/etc/krb5.conf.d/crypto-policies
/usr/lib64/libk5crypto.so.3
/usr/lib64/libk5crypto.so.3.1
/etc/krb5.conf.d/crypto-policies
/usr/lib/libk5crypto.so.3
/usr/lib/libk5crypto.so.3.1
$ ldd /usr/lib64/libk5crypto.so.3 | grep crypto
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007faed00d4000)

[richardsaloga]

Just built v1.1.0 from source on Fedora 28 and the issue persists.

[OrvilleRed]

Same error for me, reproduced under v 1.1.0 on Fedora 28 Also reproduces after adding udev rules with OWNER=(myuser) under /etc/udev/rules.d/20-hw1.rules

Good news -- I can access and sync my accounts with the Chrome app. So I have a workaround, and this appears to be a problem only with the Ledger Live software.

I do see this error in the logs when running Ledger Live: {"type":"cmd.ERROR","data":{"message":"Ledger device: Security not satisfied (dongle locked or have invalid access rights) (0x6982)","name":"TransportStatusError","stack":"Error\n at TransportStatusError (/tmp/.mount_ledgern5oiRG/app/resources/app.asar/node_modules/@ledgerhq/hw-transport/lib/Transport.js:152:16)\n at TransportNodeHid._callee$ (/tmp/.mount_ledgern5oiRG/app/resources/app.asar/node_modules/@ledgerhq/hw-transport/lib/Transport.js:205:23)\n at tryCatch (/tmp/.mount_ledgern5oiRG/app/resources/app.asar/node_modules/regenerator-runtime/runtime.js:62:40)\n at Generator.invoke [as _invoke] (/tmp/.mount_ledgern5oiRG/app/resources/app.asar/node_modules/regenerator-runtime/runtime.js:296:22)\n at Generator.prototype.(anonymous function) [as next] (/tmp/.mount_ledgern5oiRG/app/resources/app.asar/node_modules/regenerator-runtime/runtime.js:114:21)\n at step (/tmp/.mount_ledgern5oiRG/app/resources/app.asar/node_modules/babel-runtime/helpers/asyncToGenerator.js:17:30)\n at /tmp/.mount_ledgern5oiRG/app/resources/app.asar/node_modules/babel-runtime/helpers/asyncToGenerator.js:28:13\n at <anonymous>","statusCode":27010,"statusText":"SECURITY_STATUS_NOT_SATISFIED"},"level":"warn","message":"✖ CMD getAddress error","pname":"renderer","timestamp":"2018-08-04T20:51:48.532Z"}

[fansari]

Did I miss something? The Chrome apps were working all the time.

[infestdead]

I also don't understand, the chrome app works fine, also I thought the idea of AppImage format is to be distribution agnostic, everything statically linked and included - that's why it's 76MB or am I missing something?

[meriadec]

The Ledger Live app is using local C++ library for derivations etc. while the chrome apps are only using APIs.

Size is pretty huge because of electron, which is embedding its own V8 + chromium :p

[infestdead]

Can you package everything? I don't think people will mind if it's 200MB even, as long as it works on all distributions and it's really identical as long as the host can run AppImage binaries. It will also solve a lot of issues for you.

[eric-vader]

Yea I totally agree with infestdead ! I don’t mind the large size after all now disk space is not an issue. Being able to run without problems on all distros is important to me, if not I wouldn’t be able to make it work at all on Fedora. Then I will either stop using Ledger or switch distro!