Linux: Internal process error (null)

[suvayu]

Ledger Live Version and Operating System

• Ledger Live 1.0.1
• Platform: Linux

Expected behavior

I want to add my existing account for my Ledger Nano S. Following which, I should be able to sync and access my wallet.

Actual behavior

When I follow the steps to add an account, I get the following error.

Something went wrong during synchronisation, please try again.
Internal process error (null)

There is no backtrace in the terminal.

Steps to reproduce the behavior

1. Click add an account
2. Select the blockchain (Bitcoin, Litecoin, etc)
3. Unlock your Ledger device with your pin
4. Open the corresponding app on the Ledger device
5. Press continue

Screenshots

after step (4)

after step (5)

[meriadec]

@KhalilBellakrid what would be the cost / difficulty to statically link everything inside lib-ledger-core ?

[richardsaloga]

I am by no means an App-Image-ologist but their docs do seem pretty clear: Do not depend on system-provided resources.

[suvayu]

I'm a bit disappointed with the conclusions here. Ledger Live is open source, so I don't understand this Windows/Mac mentality of "let's bundle everything". Disk space might be cheap for some, does not mean everyone can afford to waste hundreds of MBs for one single purpose application. This is beside the argument, from a security standpoint external libraries aren't to be trusted implicitly.

FWIW, here are some disk usage stats for applications in the cryptocurrency space (excluding user data):

$ du -hs build/bitcoin-0.16.2/
62M     build/bitcoin-0.16.2/
$ du -hs .local/lib/python3.6/site-packages/electrum{,_{gui,plugins}}
3.0M    .local/lib/python3.6/site-packages/electrum
4.7M    .local/lib/python3.6/site-packages/electrum_gui
1.6M    .local/lib/python3.6/site-packages/electrum_plugins
$ du -hs .local/lib/python3.6/site-packages/electrum_ltc*
2.9M    .local/lib/python3.6/site-packages/electrum_ltc
1.8M    .local/lib/python3.6/site-packages/electrum_ltc_gui
760K    .local/lib/python3.6/site-packages/electrum_ltc_plugins
$  du -hs .local/lib/python3.6/site-packages/eps  # Electrum Personal Server
376K    .local/lib/python3.6/site-packages/eps

The only large ones are applications that I compile myself because either they are unreleased or for other reasons I prefer a source build, e.g. c-lightning, WalletWasabi, bisq, etc.

[eric-vader]

Well, I think if Ledger wants to distribute via AppImage, then it should be self-contained. It is meant to be packaged in this manner.

Also, I think it would be much better for Ledger to be available in each of the distro's software repos. That way, we do not need to bundle everything and its also better from a security standpoint.

[suvayu]

Well, I think if Ledger wants to distribute via AppImage, then it should be self-contained. It is meant to be packaged in this manner.

Yes, and that's why formats like these are preferred by vendors of proprietary software.

Also, I think it would be much better for Ledger to be available in each of the distro's software repos. That way, we do not need to bundle everything and its also better from a security standpoint.

That would be ideal, and I'm sure if they worked with the community, there would be plenty of help. I guess most companies shy away from it fearing the initial effort, without realising going forward that's the path with least friction (and probably the widest base to receive bug reports).

[meriadec]

I agree that the best solution is to provide independent packages for each distro (electron packager has multiple target format, including .deb, .rpm). For the moment we stick to .AppImage because it's the only one that supports self-update, but in Linux world it makes more sense to update the system than letting apps updating theirselves (also: this is a backdoor risk).

If community want to help it's super appreciated, because project management makes focus on mobile app for now.

[snaggen]

Using flatpak and publish it on flathub.org would give a clear sdk for the developers, with a guarantee that all users have the same runtime libraries. Flathub.org provides a good app store with easy upgrade. So, it gives a good balance between size, security and user-friendliness. Things to investigate is if device access works well enough, and if electron is overly complicated to build as flatpaks (but flathub.org contains multiple electron apps, so it probably isn't).

But the whole user running different libraries issues should be gone, since that is one of the things flatpak was designed to fix.

Den tis 7 aug. 2018 13:14Meriadec Pillet notifications@github.com skrev:

I agree that the best solution is to provide independent packages for each distro (electron packager has multiple target format, including .deb, .rpm). For the moment we stick to .AppImage because it's the only one that supports self-update, but in Linux world it makes more sense to update the system than letting apps updating theirselves (also: this is a backdoor risk).

If community want to help it's super appreciated, because project management makes focus on mobile app for now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LedgerHQ/ledger-live-desktop/issues/1010#issuecomment-411021613, or mute the thread https://github.com/notifications/unsubscribe-auth/AGH4n7i5z6nrlFon1HF2xuV1B96ZxWprks5uOXaygaJpZM4VJGlV .

[richardsaloga]

Just built v1.1.5 from source on Fedora 28 and the issue persists.

[ugilio]

After a bit of debugging, I think I know what is going on.

TL;DR: lib-ledger-core embeds its own version of openssl (1.0.1v-dev), but because of dependency chains ledger-live-desktop loads the system-installed openssl (in case of Fedora 28, 1.1.0h) before. As a result, very bad things happen.

In more detail:

• lib-ledger-core depends on libcups.so.2, and on Fedora 28 loading libcups.so.2 causes the following libraries to be loaded: libcups.so.2 -> libgssapi_krb5.so.2 -> libk5crypto.so.3 -> libcrypto.so.1.1
• When lib-ledger-core is loaded, the openssl symbols that the library needs are resolved against the already-loaded libcrypto.so.1.1, and not against the "internal openssl".
• So, in this case lib-ledger-core is calling 1.1 functions using 1.0 definitions and macros, which is dangerous

About this specific crash, consider this method from lib-ledger-core in core/src/crypto/HMAC.cpp

std::vector<uint8_t> ledger::core::HMAC::sha512(const std::vector<uint8_t>& key,
                                                    const std::vector<uint8_t>& data) {
    auto len = SHA512_DIGEST_LENGTH;
    uint8_t hash[len];
    HMAC_CTX hmac;
    HMAC_CTX_init(&hmac);
    HMAC_Init_ex(&hmac, key.data(), key.size(), EVP_sha512(), NULL);
    HMAC_Update(&hmac, data.data(), data.size());
    HMAC_Final(&hmac, hash, (unsigned int *)(&len));
    HMAC_cleanup(&hmac);
    return std::vector<uint8_t>(hash, hash + SHA512_DIGEST_LENGTH);
}

Now. HMAC_CTX_init was removed in openssl 1.1, so it is resolved to the internal (1.0.1) function. , while the other functions are resolved to the system-provided 1.1 functions:

(gdb) info sym HMAC_CTX_init
HMAC_CTX_init in section .text of /tmp/.mount_ledgerOK3AAL/app/resources/app.asar.unpacked/node_modules/@ledgerhq/ledger-core/build/Release/libledger-core.so
(gdb) info sym EVP_sha512
EVP_sha512 in section .text of /lib64/libcrypto.so.1.1
(gdb) info sym HMAC_Init_ex
HMAC_Init_ex in section .text of /lib64/libcrypto.so.1.1

Also, the HMAC_CTX struct is the one defined in 1.0.1.

• In 1.0.1 it is

  typedef struct hmac_ctx_st {
  const EVP_MD *md;
  EVP_MD_CTX md_ctx;
  EVP_MD_CTX i_ctx;
  EVP_MD_CTX o_ctx;
  unsigned int key_length;
  unsigned char key[HMAC_MAX_MD_CBLOCK];
  } HMAC_CTX;

• while in 1.1 it is opaque, but internally it is

  type = struct hmac_ctx_st {
  const EVP_MD *md;
  EVP_MD_CTX *md_ctx;
  EVP_MD_CTX *i_ctx;
  EVP_MD_CTX *o_ctx;
  unsigned int key_length;
  unsigned char key[128];
  }

  So, the HMAC_CTX allocated is the 1.0 one, and is initialized by the 1.0 HMAC_CTX_init, but then is passed to the 1.1 HMAC_Init_ex, and we have a SIGSEGV when it tries to dereference i_ctx, which is a pointer in 1.1, but in 1.0 that memory location is probably a 0 inside md_ctx;

What can be done?

1. Make sure lib-ledger-core doesn't try to resolve openssl symbols at runtime, but the references are resolved at compile time. Probably there is some configuration option to tell the linker to do so. This way ledger uses its own internal version of openssl, regardless of the operating-system supplied version which might not include all the needed algorithms. OR
2. Don't embed openssl on linux, but use the system-provided one. However, some changes are needed to be compatible with newer openssls (e.g allocate the HMAC_CTX on the heap with HMAC_CTX_new instead of having the variable living on the stack, and so on). Fedora still patches openssl with regards to Elliptic Cryptography, and I am not sure if the provided version is enough for Ledger Live to work

I think that currently the application works on some systems because for some reason the libcrypto 1.1 library is not loaded due to different installed packages or configuration, so the internal openssl version is used, or on older systems a 1.0 version of openssl is loaded and by chance things work, but it's of course not the most robust thing.

[meriadec]

@ugilio thanks a lot, this is very detailed & useful debugging :heart_eyes: @KhalilBellakrid it's a good hint to possibly solve the issue. WDYT

[KhalilBellakrid]

@ugilio Thank you for your analysis ! Indeed, I was wondering why the context was lost because I've seen it in previously posted logs but I couldn't figure out why because we didn't manage to reproduce the bug from our side ! I agree on the first solution that normally should already be the case since we build a static dependency of libcrypto in lib-ledger-core, we just need to add it during build phase of our node bindings (used by Ledger Live). @meriadec we can make a test where I upload the static libcrypto, then we add it through the preinstall script of our bindings (same as for windows) that could solve the issue !

[richardsaloga]

The issue persists with v.1.1.9.

[emx0r]

I wonder if there is any hope to get this fixed, it's really annoying

[rzeidler]

This issues appears to persist with v.1.1.11 on Fedora 28.

[infestdead]

@rzeidler I wouldn't bet on Ledger fixing this anytime soon, I brought again the issue to them and got told that it probably doesn't affect many people and if I'm still not happy - I should send a patch myself :) So yeah, don't get your hopes up.

[meriadec]

@infestdead please don't bring false claims and emotions to the discussion, it is irrelevant. (relatable)

As said earlier, this problem is affecting some Fedora installations, is due to incompatibilities with dynamically linked openssl system library, and is not present when using a fresh install of Fedora 28 ISO.

There is a bunch of workarounds to access and use your funds for now (use other distro / OS, use chrome apps), keep in mind this is only a matter of stats and prioritization and there is currently no bandwidth to address the problem :confused:

[infestdead]

@meriadec sorry if I sound salty, frustrated and disappointed, but my claims are not false.

About the fresh install, I saw you said you synced LTC and ETH/XRP and other altcoins, I also managed to sync ETH, I don't have LTC so can't test, but BTC/BCH/DASH don't work, that's my main issue. Can you confirm those work on a fresh install? I can spin a virtual machine, but I don't have a full spare physical machine around to test with.

I know about the workarounds, I use the chrome app right now. It's just that when I saw you using an app format like AppImage I was finally hoping not to be treating like a second class citizen and expected the app to work from day one just like it works on Windows and Ubuntu and OSX, well.. not really.. not even for BTC, not even after a few months :)

[gre]

@meriadec is actually using Linux 100% of the time & was during the Ledger Live mvp development. @btchip also is, so we do not consider Linux as secondary. We'll try to experience this again and will test on BTC / BCH / DASH.

[gre]

can you tell us your Linux distribution / version ? thanks

[suvayu]

@gre, @meriadec, sorry to interject, but as has been illustrated earlier: https://github.com/LedgerHQ/ledger-live-desktop/issues/1010#issuecomment-412111461, it's actually rather coincidental that this even works on some platforms, to repeatedly blame a platform is quite frustrating. Here is the conclusion from that comment:

I think that currently the application works on some systems because for some reason the libcrypto 1.1 library is not loaded due to different installed packages or configuration, so the internal openssl version is used, or on older systems a 1.0 version of openssl is loaded and by chance things work, but it's of course not the most robust thing.

The irony is, this packaging bug ends up defeating the very goal of using the AppImage packaging format, platform independence! There is nothing that prevents this from reappearing again for a different build on a different platform. Anyway, I have had enough of this issue, and I'll vote with my wallet by moving to other hardware wallets.

[emx0r]

@gre I have same problem on Fedora 28 4.18.8-200.fc28.x86_64 #1 SMP Sun Sep 16 18:15:41 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

It does not work for me since first release of Ledger Live - only BTC and "BTC related" coins are having issue

[infestdead]

@gre I'm on latest Fedora 28, everything is kept updated almost daily, worst weekly. I have two PCs - work and home - same results. I'm also the OP in #950 which got closed I guess because this got more traction, but I also spent a lot of my time there testing different releases, building the app from source etc, no success.

[meriadec]

Ok thanks for clarifying and I realize how disappointing this whole thing could be..

@gre +1 for allocating budget on this and trying again on our side

[tom0105]

I am using Fedora 28 and I am also affected by this issue. I tried the script @drizzt provided with the newest Leger Live and it is working. With the patched Appimage I can sync BTC, LTC and all other coins. Until an "official" fix is provided, is this a safe method to use? I know the packages are taken from the official Ubuntu repository, but I just would like to make sure the replacement of the libraries does not cause any other issues.

[fansari]

Currently this page https://www.ledger.com/pages/ledger-live gives HTTP error 404 so I tried the latest version I had on my PC - 1.0.7. I can confirm that this @drizzt fix is working. So far I did not try transaction but I could open my Bitcoin app which did not work without this patch.

[ronmckown]

I am also having issues on Fedora 28 with latest package updates installed and I'm also getting a 404 on the following webpage: https://www.ledger.com/pages/ledger-live

I opened up a support ticket for the site 404:

211096

Hopefully this will all be resolved soon.

[meriadec]

For the 404 bug, there is an issue on user-agent detection on marketing website (see https://github.com/LedgerHQ/ledger-live-desktop/issues/1460) which has been raised and (hopefully) the page will get updated soon.

For the crash issue, the bug has been reproduced on our side this week and we are working on a fix. Expect it to be delivered in the 1.2.1 version (1.2.0 coming this week).

[Foxh0und]

ETH, ETC and XRP all fine for me, but getting the internal process issue(null) with LTC. Any ideas? On Fedora 28.

[meriadec]

relatable:

• https://github.com/signalapp/Signal-Desktop/issues/2610
• https://github.com/signalapp/Signal-Desktop/issues/2634

[PatDubois]

Do we have an approximate release date for 1.2.1 ?

[gre]

Should be out the 2018-10-22. Prereleased a bit sooner hopefully

[PatDubois]

I can confirm the 1.2.1 release works for me on a fully patched F28 workstation.

Much appreciated !

[emx0r]

It works for me as well. Fedora 28. Thx

[rzeidler]

For me too, thanks!

[infestdead]

Thanks, works for me as well on latest fedora 28, thanks guys!

[fansari]

I can give confirmation too. Works with Fedora 28.

[o-Epictetus-o]

The issue is now back. Pop! 19.10 eoan, fresh ledger live install and directly went to add bitcoin account

[carleryd]

Same issue as @Kartereo on Ubuntu 18.04.2 LTS on Ledger Nano S using firmware 1.6. I'm able to view my Ada through the chrome extension but neither Bitcoin nor Ethereum passes the "Accounts" step and I receive the error above "Internal process error (null)"

EDIT: I managed to resolve this by resetting my Ledger Live in settings. Before it actually said my device was defective but I continued anyway. Now I can view my Bitcoin and Ethereum wallets :tada: