ghmgit commented 3 years ago

Describe the bug

In RHEL-8.4, everything works fine including joining a pool. But farming into the pool fails with:

629 2021-07-11T19:10:04.807 farmer chia.farmer.farmer : ERROR Exception in GET /pool_info https://pool.letsfarm.io, Cannot connect to host pool.letsfarm.io:443 ssl:True [SS 629 LCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')] 630 2021-07-11T19:10:04.808 farmer chia.farmer.farmer : WARNING No pool specific authentication_token_timeout has been set for 2576c1609d2f5171b994b5800f4afd742f3df34565d0026 630 6febfbee9ab3059b5, check communication with the pool.

SIDENOTE1: If you get this error: 23 2021-07-11T17:24:24.291 farmer chia.farmer.farmer : ERROR Exception in GET /pool_info https://eu1.pool.space, Cannot connect to host eu1.pool.space:443 ssl:default [Net 23 work is unreachable] That is caused by IPv6 not working. In my case I had to enable IPv6 in my Fritzbox DSL router.

To Reproduce

1) Install RHEL-8.4 2) Install chia 1.2.0 rpm 3) Start your farm/ create plots 4) Join a Pool 5) Look into "debug.log" to find out it is not working (despite the GUI does not show any error message).

SIDENOTE2: You need to replace "chiavdf.cpython-38-x86_64-linux-gnu.so" with the file from the deb package, but that is another problem: chia42@m620a daemon]$ ldd chiavdf.cpython-38-x86_64-linux-gnu.so chiapos.cpython-38-x86_64-linux-gnu.so |cat -n 1 chiavdf.cpython-38-x86_64-linux-gnu.so: 2 not a dynamic executable 3 chiapos.cpython-38-x86_64-linux-gnu.so: 4 linux-vdso.so.1 (0x00007ffda9ab6000) 5 libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0e1421c000) 6 libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f0e13e87000) 7 libm.so.6 => /lib64/libm.so.6 (0x00007f0e13b05000) 8 libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f0e138ed000) 9 libc.so.6 => /lib64/libc.so.6 (0x00007f0e13528000) 10 /lib64/ld-linux-x86-64.so.2 (0x00007f0e14aaf000)

Expected behavior

Farming with a pool should work.

Screenshots See above for cut-n-paste of error message from "debug.log"

Desktop

OS: Linux
OS Version/Flavor: RHEL-8.4
CPU: does not matter

Additional context

Root-Cause-Analysis

1) Connecting with OpenSSL on command line works fine:

[chia42@m620a lib]$ echo -n |openssl s_client -connect pool.letsfarm.io:443 |cat -n depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = .letsfarm.io verify return:1 DONE 1 CONNECTED(00000003) 2 --- 3 Certificate chain 4 0 s:CN = .letsfarm.io 5 i:C = US, O = Let's Encrypt, CN = R3 6 1 s:C = US, O = Let's Encrypt, CN = R3 7 i:O = Digital Signature Trust Co., CN = DST Root CA X3 8 --- 9 Server certificate ... ... 34 ZjHBm5+4hXhn+eJXB6sUbjJH30I= 35 -----END CERTIFICATE----- 36 subject=CN = *.letsfarm.io 37
38 issuer=C = US, O = Let's Encrypt, CN = R3 39
40 --- 41 No client certificate CA names sent 42 Peer signing digest: SHA256 43 Peer signature type: ECDSA 44 Server Temp Key: X25519, 253 bits 45 --- 46 SSL handshake has read 2567 bytes and written 394 bytes 47 Verification: OK 48 --- 49 New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 50 Server public key is 256 bit 51 Secure Renegotiation IS NOT supported 52 Compression: NONE 53 Expansion: NONE 54 No ALPN negotiated 55 Early data was not sent 56 Verify return code: 0 (ok) 57 ---

2) Conclusion: The problem is not caused by the pooling server. The problem might be caused by chia software.

3) Often such problems can be analyzed by looking at which files (configs, certificate bundles) are used. Unfortunately the situation in Linux variants can be a mess.

The "strace" command shows what files are opened:

chia42@m620a lib]$ echo -n |strace openssl s_client -connect pool.letsfarm.io:443 |& grep open |cat -n 1 execve("/usr/bin/openssl", ["openssl", "s_client", "-connect", "pool.letsfarm.io:443"], 0x7ffd8e148eb8 / 20 vars /) = 0 2 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 3 openat(AT_FDCWD, "/lib64/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = 3 4 openat(AT_FDCWD, "/lib64/libcrypto.so.1.1", O_RDONLY|O_CLOEXEC) = 3 5 openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3 6 openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3 7 openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3 8 openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 9 openat(AT_FDCWD, "/proc/sys/crypto/fips_enabled", O_RDONLY) = 3 10 openat(AT_FDCWD, "/etc/pki/tls/openssl.cnf", O_RDONLY) = 3 11 stat("/etc/crypto-policies/back-ends/opensslcnf.config", {st_mode=S_IFREG|0644, st_size=583, ...}) = 0 12 openat(AT_FDCWD, "/etc/crypto-policies/back-ends/opensslcnf.config", O_RDONLY) = 4 13 openat(AT_FDCWD, "/etc/crypto-policies/back-ends/openssl.config", O_RDONLY) = 3 14 openat(AT_FDCWD, "/etc/pki/tls/ct_log_list.cnf", O_RDONLY) = 3 15 openat(AT_FDCWD, "/etc/pki/tls/cert.pem", O_RDONLY) = 3 16 openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3 17 openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3 18 openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3 19 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 20 openat(AT_FDCWD, "/lib64/libnss_files.so.2", O_RDONLY|O_CLOEXEC) = 3 21 openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3 22 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 23 openat(AT_FDCWD, "/lib64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = 3 24 openat(AT_FDCWD, "/lib64/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3 25 openat(AT_FDCWD, "/etc/gai.conf", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) 26 openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 4

4) Now we know what files are read when using "openssl" on command line.

5) Now we need to find out what chia_farmer is doing:

6) The chia_farmer is started by chia_daemon, so "strace -f -o chiatrace.out -p PID-of-Chia_Daemon", "chia start -r farmer-only", wait for the error message

7) Now look at "ssl" related lines: cat chiatrace.out |grep ssl

948107 20:56:39 openat(AT_FDCWD, "/usr/lib/chia-blockchain/resources/app.asar.unpacked/daemon/cryptography/hazmat/bindings/_openssl.abi3.so", O_RDONLY|O_CLOEXEC) = 7 <0.000012> 2948107 20:56:39 openat(AT_FDCWD, "/opt/pyca/cryptography/openssl/openssl.cnf", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000010> 2948107 20:56:39 openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000010> 2948107 20:56:39 openat(AT_FDCWD, "/home/chia42/.chia/mainnet/config/ssl/ca/private_ca.crt", O_RDONLY) = 10 <0.000011> 2948107 20:56:39 openat(AT_FDCWD, "/home/chia42/.chia/mainnet/config/ssl/farmer/private_farmer.crt", O_RDONLY) = 10 <0.000011> 2948107 20:56:39 openat(AT_FDCWD, "/home/chia42/.chia/mainnet/config/ssl/farmer/private_farmer.key", O_RDONLY) = 10 <0.000009>

8) The root cause is in this line:"openssl.cnf" is not found ! 2948107 20:56:39 openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000010>

And also later the "cert.pem" bundle is missing also: 2948107 20:56:40 openat(AT_FDCWD, "/usr/lib/ssl/cert.pem", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000011>

9) FINAL ROOT CAUSE: Chia brings its own openssl library which uses "/usr/lib/ssl/openssl.cnf". This is in conflict with RHEL-8.4 system provided openssl library which uses "/etc/pki/tls/openssl.cnf".

10) WORKAROUND Make a symbolic link, so chia can find the config file: ln -s /etc/plit/tls /usr/lib/ssl chia start -r farmer-only BINGO (sic!) 021-07-11T21:32:27.400 farmer chia.farmer.farmer : INFO GET /pool_info response: {'authentication_token_timeout': 5, 'description': 'Our Chia pool is stable, secure and reliable with zero-downtime. Payout is done using Pay-Per-Last N Shares (PPLNS). First 3 months without fee!', 'fee': '0.0', 'logo_url': 'https://letsfarm.io/images/pool.letsfarm.io.svg', 'minimum_difficulty': 1, 'name': 'letsfarm.io - Chia Network Farming Pool', 'protocol_version': 1, 'relative_lock_height': 100, 'target_puzzle_hash': '0x5ea0ba51c67a4830d9c4f70b491a417448b76f88343abba7282068cf36c1b880'}

11) RESOLUTION: Chia does not use the OS provided openssl library but brings its own: 2948107 20:56:37 openat(AT_FDCWD, "/usr/lib/chia-blockchain/resources/app.asar.unpacked/daemon/libssl.so.1.1", O_RDONLY|O_CLOEXEC) = 3 <0.000010>

So either

teach chia to also use /etc/pki/tls in addition to /usr/lib/ssl
or chia should bring its own config / not rely on OS config (it already brings its own cert bundle: -rw-r--r--. 1 root root 208075 Jul 8 06:12 ./resources/app.asar.unpacked/daemon/mozilla-ca/cacert.pem)

Regards, gm

P.S. If this was helpful to you or you appreciate this for other reasons you can send some mojo as feedback to xch1hetrkzn507tlregyc8msruskzr8j435vf0axmrw9wug5gem55akqcesu4d Note: sending mojo is completely voluntary.

extrobe commented 3 years ago

Getting the same issue in MacOS 11.4 on M1 chip.

The suggested workaround here didn't work in my case (Operation not permitted), but will watch this ticket to see if any other suggestions come up.

lordkiwiz commented 3 years ago

Excellent workaround. The openssl config directory on manjaro is /etc/ssl, and not /etc/pki/tls. Linking /etc/ssl to /usr/lib/ssl solved it for me.

sgmihai commented 3 years ago

Thanks, also got this problem on an arch based distro, trying to figure it out for 2 days. /etc/ssl to /usr/lib/ssl worked, I'm getting points at pool now, although I still get ssl errors in log for some reason.