search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.83k stars 896 forks source link

Gossip Store malformed on OSX #3041

Closed willcl-ark closed 5 years ago

willcl-ark commented 5 years ago

Channels created to/from an OSX-built node can be opened and closed but not used.

Opening a channel to and from this peer seems to work fine, however regardless of which direction the channel was opened, invoices generated cannot be paid: "Could not find a route", Error 205.

Closing the channel results in a mutually closed channel.

Here is my gossip_store saved with a .txt extension. gossip_store.txt

Hope it's not user error, but it reads to me like it can't create the gossip_store properly...

willcl-ark commented 5 years ago

OK I have tried rebuilding this with gcc-9 from homebrew, but I still get the same error messages about the gossip store, and channels, whilst they can be opened and closed, still cannot be used...

willcl-ark commented 5 years ago

Seems like it could be related to https://github.com/ElementsProject/lightning/issues/2011 ?

Also, since re-building with gcc-9, although the gossip store is still malformed, the remote node who opened a channel to local node can now pay invoices to local node, hooray!

When I create a new invoice I see in the logs: UNUSUAL lightningd(23824): invoice: insufficient incoming capacity for 1000msat, but I think this might be because the two nodes are segregated from the main testnet network (they are only connected to each other) and so the peer is considered a "dead end" by lightningd? "warning_capacity": "No channel with a peer that is not a dead end, has sufficient incoming capacity"

darosior commented 5 years ago

although the gossip store is still malformed

Are you still using the same gossip_store ? Just to be sure, have you tried deleting it ?

"warning_capacity": "No channel with a peer that is not a dead end, has sufficient incoming capacity"

Without gossip you won't be able to find a route at all.

willcl-ark commented 5 years ago

Yep I’ve tried deleting it. I have also tried deleting everything except config.

The only slightly abnormal settings I’m using are testnet and port 9733, but that shouldn’t affect it...

darosior commented 5 years ago

The only slightly abnormal settings I’m using are testnet and port 9733, but that shouldn’t affect it...

Indeed it should not, but I'm not sure about gossip_store stuff and I'll let @rustyrussell or @m-schmoock answer you..

rustyrussell commented 5 years ago

Wow. Seems like pwritev isn't working on MacOS. Or at least, it doesn't return the right value!

Can you compile and run this program, and return the output? And maybe the contents of /tmp/testfile?

#include <err.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/uio.h>
#include <unistd.h>

int main(void)
{
    int fd = open("/tmp/testfile", O_RDWR|O_TRUNC|O_CREAT, 0600);
    int r;
    struct iovec iov[2];

    if (fd < 0)
        err(1, "opening /tmp/testfile");
    r = 0;
    iov[0].iov_base = &r;
    iov[0].iov_len = sizeof(r);
    iov[1].iov_base = &fd;
    iov[1].iov_len = sizeof(fd);
    r = pwritev(fd, iov, 2, 3);
    printf("pwritev gave %i, errno = %i\n", r, errno);
}
willcl-ark commented 5 years ago

here is the output using clang:

~/Desktop
➜ gcc -v -o output pwrite_test.c
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.14.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name pwrite_test.c -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fno-strict-return -masm-verbose -munwind-tables -target-cpu penryn -dwarf-column-info -debugger-tuning=lldb -target-linker-version 450.3 -v -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1 -Wno-atomic-implicit-seq-cst -Wno-framework-include-private-from-public -Wno-atimport-in-framework-header -Wno-quoted-include-in-framework-header -fdebug-compilation-dir /Users/will/Desktop -ferror-limit 19 -fmessage-length 112 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fobjc-runtime=macosx-10.14.0 -fmax-type-align=16 -fdiagnostics-show-option -fcolor-diagnostics -o /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T/pwrite_test-51e598.o -x c pwrite_test.c
clang -cc1 version 10.0.1 (clang-1001.0.46.4) default target x86_64-apple-darwin18.7.0
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
pwrite_test.c:21:6: warning: implicit declaration of function 'pwritev' is invalid in C99
      [-Wimplicit-function-declaration]
        r = pwritev(fd, iov, 2, 3);
            ^
1 warning generated.
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -lto_library /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch x86_64 -macosx_version_min 10.14.0 -o output /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T/pwrite_test-51e598.o -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/10.0.1/lib/darwin/libclang_rt.osx.a
Undefined symbols for architecture x86_64:
  "_pwritev", referenced from:
      _main in pwrite_test-51e598.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
willcl-ark commented 5 years ago

and with gcc-9:

~/Desktop
➜ gcc-9 -v -o output pwrite_test.c
Using built-in specs.
COLLECT_GCC=gcc-9
COLLECT_LTO_WRAPPER=/usr/local/Cellar/gcc/9.2.0/libexec/gcc/x86_64-apple-darwin18/9.2.0/lto-wrapper
Target: x86_64-apple-darwin18
Configured with: ../configure --build=x86_64-apple-darwin18 --prefix=/usr/local/Cellar/gcc/9.2.0 --libdir=/usr/local/Cellar/gcc/9.2.0/lib/gcc/9 --disable-nls --enable-checking=release --enable-languages=c,c++,objc,obj-c++,fortran --program-suffix=-9 --with-gmp=/usr/local/opt/gmp --with-mpfr=/usr/local/opt/mpfr --with-mpc=/usr/local/opt/libmpc --with-isl=/usr/local/opt/isl --with-system-zlib --with-pkgversion='Homebrew GCC 9.2.0' --with-bugurl=https://github.com/Homebrew/homebrew-core/issues --disable-multilib --with-native-system-header-dir=/usr/include --with-sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk
Thread model: posix
gcc version 9.2.0 (Homebrew GCC 9.2.0)
COLLECT_GCC_OPTIONS='-v' '-o' 'output' '-mmacosx-version-min=10.14.0' '-asm_macosx_version_min=10.14' '-mtune=core2'
 /usr/local/Cellar/gcc/9.2.0/libexec/gcc/x86_64-apple-darwin18/9.2.0/cc1 -quiet -v -D__DYNAMIC__ pwrite_test.c -fPIC -quiet -dumpbase pwrite_test.c -mmacosx-version-min=10.14.0 -mtune=core2 -auxbase pwrite_test -version -o /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T//ccgkrrVP.s
GNU C17 (Homebrew GCC 9.2.0) version 9.2.0 (x86_64-apple-darwin18)
    compiled by GNU C version 9.2.0, GMP version 6.1.2, MPFR version 4.0.2, MPC version 1.1.0, isl version isl-0.21-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/../../../../../../x86_64-apple-darwin18/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/include
 /usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/include-fixed
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks
End of search list.
GNU C17 (Homebrew GCC 9.2.0) version 9.2.0 (x86_64-apple-darwin18)
    compiled by GNU C version 9.2.0, GMP version 6.1.2, MPFR version 4.0.2, MPC version 1.1.0, isl version isl-0.21-GMP

GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: 299875c881ffa94e36973da59ff78aeb
pwrite_test.c: In function 'main':
pwrite_test.c:21:6: warning: implicit declaration of function 'pwritev'; did you mean 'pwrite'? [-Wimplicit-function-declaration]
   21 |  r = pwritev(fd, iov, 2, 3);
      |      ^~~~~~~
      |      pwrite
COLLECT_GCC_OPTIONS='-v' '-o' 'output' '-mmacosx-version-min=10.14.0'  '-mtune=core2'
 as -arch x86_64 -v -force_cpusubtype_ALL -mmacosx-version-min=10.14 -o /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T//ccgMN2rE.o /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T//ccgkrrVP.s
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1as -triple x86_64-apple-macosx10.14.0 -filetype obj -main-file-name ccgkrrVP.s -target-cpu penryn -fdebug-compilation-dir /Users/will/Desktop -dwarf-debug-producer Apple LLVM version 10.0.1 (clang-1001.0.46.4) -dwarf-version=4 -mrelocation-model pic -o /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T//ccgMN2rE.o /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T//ccgkrrVP.s
COMPILER_PATH=/usr/local/Cellar/gcc/9.2.0/libexec/gcc/x86_64-apple-darwin18/9.2.0/:/usr/local/Cellar/gcc/9.2.0/libexec/gcc/x86_64-apple-darwin18/9.2.0/:/usr/local/Cellar/gcc/9.2.0/libexec/gcc/x86_64-apple-darwin18/:/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/:/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/
LIBRARY_PATH=/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/:/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/../../../
COLLECT_GCC_OPTIONS='-v' '-o' 'output' '-mmacosx-version-min=10.14.0'  '-mtune=core2'
 /usr/local/Cellar/gcc/9.2.0/libexec/gcc/x86_64-apple-darwin18/9.2.0/collect2 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/ -dynamic -arch x86_64 -macosx_version_min 10.14.0 -weak_reference_mismatches non-weak -o output -L/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0 -L/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/../../.. /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T//ccgMN2rE.o -no_compact_unwind -lSystem -lgcc_ext.10.5 -lgcc -lSystem -v
collect2 version 9.2.0
/usr/bin/ld -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/ -dynamic -arch x86_64 -macosx_version_min 10.14.0 -weak_reference_mismatches non-weak -o output -L/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0 -L/usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/../../.. /var/folders/0d/tgscg8cj0rx035gf2lvl15fh0000gn/T//ccgMN2rE.o -no_compact_unwind -lSystem -lgcc_ext.10.5 -lgcc -lSystem -v
@(#)PROGRAM:ld  PROJECT:ld64-450.3
BUILD 18:16:53 Apr  5 2019
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
Library search paths:
    /usr/local/Cellar/gcc/9.2.0/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0
    /usr/local/Cellar/gcc/9.2.0/lib/gcc/9
    /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib
Framework search paths:
    /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/
Undefined symbols for architecture x86_64:
  "_pwritev", referenced from:
      _main in ccgMN2rE.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
willcl-ark commented 5 years ago

@rustyrussell I wonder if it can simply be routed around as per: https://salsa.debian.org/debian/libuv1/commit/af96f458e2583a9cb2e7f2f20b3b3ff04bca6c24

I don’t know enough about it and C-Lightning to know, but this was the closest I could find to a solution when I saw C-Lightning (master) compilation errors earlier this week....

willcl-ark commented 5 years ago

I assume you might have known this might happen, but v0.7.0 works just fine with my system. I can't see the references to pwritev and preadv in the gossip source for that (or v 0.7.1) so I guess it was introduced for 0.7.2...

Anyway, let me know if you want any more tests or logs done. I have enough to continue with my own work now so probably won't update this any more unless requested :)

m-schmoock commented 5 years ago

@willcl-ark you should not be using v0.7.0 any more. There is a critical security issue with versions <v0.7.1

willcl-ark commented 5 years ago

Thanks. I heard the announcement and still forgot! I’m only testing now so will try building 0.7.1 but otherwise stick with 0.7.0 while this hopefully 😋 gets resolved

darosior commented 5 years ago

I heard the announcement and still forgot! I’m only testing now so will try building 0.7.1 but otherwise stick with 0.7.0

You should try v0.7.1 quickly..

willcl-ark commented 5 years ago

@darosior can confirm that 0.7.1 builds OK on OSX and seems to work as well as 0.7.0

saubyk commented 5 years ago

Hi, any update on this issue? I just pulled from the master and getting the below error on make:

cc -DBINTOPKGLIBEXECDIR="\"../libexec/c-lightning\"" -Wall -Wundef -Wmissing-prototypes -Wmissing-declarations -Wstrict-prototypes -Wold-style-definition -Werror -std=gnu11 -g -fstack-protector -Og -I ccan -I external/libwally-core/include/ -I external/libwally-core/src/secp256k1/include/ -I external/jsmn/ -I external/libbacktrace/ -I external/libbacktrace-build -I . -I/usr/local/include    -DSHACHAIN_BITS=48 -DJSMN_PARENT_LINKS  -DCOMPAT_V052=1 -DCOMPAT_V060=1 -DCOMPAT_V061=1 -DCOMPAT_V062=1 -DCOMPAT_V070=1 -DBINTOPKGLIBEXECDIR="\"../libexec/c-lightning\""  -c -o gossipd/gossip_store.o gossipd/gossip_store.c
gossipd/gossip_store.c:74:6: error: implicit declaration of function 'pwritev' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        if (pwritev(fd, iov, ARRAY_SIZE(iov), *len) != sizeof(hdr) + msglen)
            ^
gossipd/gossip_store.c:74:6: error: this function declaration is not a prototype [-Werror,-Wstrict-prototypes]
rustyrussell commented 5 years ago

Ok, this is good! It means I can detect a missing pwritev/preadv, and work around it. One moment...