ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.84k stars 901 forks source link

Install takes forever(?) #6849

Closed Sjors closed 4 months ago

Sjors commented 12 months ago

I'm trying to install v23.11rc1 on Ubuntu 23.04, but the installer seems to get stuck.

git checkout v23.11rc1
git clean -dfx
./configure --prefix=/home/lightning
make -j7
make -j7 install # this seems to repeat all of make ???

Log for the latter command:

mkdir -p /home/lightning/bin
mkdir -p /home/lightning/libexec/c-lightning
mkdir -p /home/lightning/libexec/c-lightning/plugins
mkdir -p /home/lightning/share/man/man1
mkdir -p /home/lightning/share/man/man5
mkdir -p /home/lightning/share/man/man7
mkdir -p /home/lightning/share/man/man8
mkdir -p /home/lightning/share/doc/c-lightning
cargo build --quiet --bin cln-grpc
xgettext plugins/bkpr/statements_gettextgen.po
cc plugins/bkpr/account.c
cc plugins/bkpr/account_entry.c
cc plugins/bkpr/bookkeeper.c
cc plugins/bkpr/chain_event.c
cc plugins/bkpr/channel_event.c
cc plugins/bkpr/channelsapy.c
cc plugins/bkpr/db.c
cc plugins/bkpr/incomestmt.c
cc plugins/bkpr/onchain_fee.c
cc plugins/bkpr/recorder.c
sql-rewrite plugins/bkpr/db_sqlite3_sqlgen.c
sql-rewrite plugins/bkpr/db_postgres_sqlgen.c
cd external/lowdown && CC="cc" CFLAGS="" ./configure LDFLAGS="" PREFIX=`pwd`/../../external/build-x86_64-linux-gnu/lowdown-build/
config.log: writing...
GNU submake detected: using --no-print-directory
make[1]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
make[1]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
cc plugins/bkpr/db_sqlite3_sqlgen.c
configure.local: no (fully automatic configuration)
cc plugins/bkpr/db_postgres_sqlgen.c
arc4random: yes 
b64_ntop: yes (with -lresolv)
capsicum: no
ld plugins/bookkeeper
crypt: yes (with -lcrypt)
endian_h: yes 
err: no
explicit_bzero: yes 
fts: yes 
getexecname: no
getprogname: no
INFTIM: no
lib_socket: yes 
md5: yes (with -lmd)
memmem: yes 
memrchr: yes 
memset_s: no
mkfifoat: yes 
mknodat: yes 
osbyteorder_h: no
PATH_MAX: yes 
pledge: no
program_invocation_short_name: yes 
readpassphrase: no
reallocarray: yes 
recallocarray: no
sandbox_init: no
seccomp-filter: yes 
setresgid: yes 
setresuid: yes 
sha2: yes (with -lmd)
SOCK_NONBLOCK: yes 
static: yes 
strlcat: no
strlcpy: no
strndup: yes 
strnlen: yes 
strtonum: no
sys_byteorder_h: no
sys_endian_h: no
sys_mkdev_h: no
sys_sysmacros_h: yes 
sys_queue: no
sys_tree: no
unveil: no
WAIT_ANY: yes 
__progname: yes 
config.h: written
Makefile.configure: written
make -C external/lowdown install
make[1]: Entering directory '/home/lightning/src/lightning/external/lowdown'
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o autolink.o autolink.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o buffer.o buffer.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o diff.o diff.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o document.o document.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o entity.o entity.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o gemini.o gemini.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o html.o html.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o html_escape.o html_escape.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o latex.o latex.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o library.o library.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o libdiff.o libdiff.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o nroff.o nroff.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o odt.o odt.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o smartypants.o smartypants.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o term.o term.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o tree.o tree.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o util.o util.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o compats.o compats.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter -fPIC   -c -o main.o main.c
main.c:138:2: warning: #warning Compiling without sandbox support. [-Wcpp]
  138 | #warning Compiling without sandbox support.
      |  ^~~~~~~
ar rs liblowdown.a autolink.o buffer.o diff.o document.o entity.o gemini.o html.o html_escape.o latex.o library.o libdiff.o nroff.o odt.o smartypants.o term.o tree.o util.o compats.o
cc -o lowdown main.o liblowdown.a  -lmd -lm
ln -f lowdown lowdown-diff
mkdir -p /home/lightning/src/lightning/external/lowdown/../../external/build-x86_64-linux-gnu/lowdown-build//bin
mkdir -p /home/lightning/src/lightning/external/lowdown/../../external/build-x86_64-linux-gnu/lowdown-build//man/man1
mkdir -p /home/lightning/src/lightning/external/lowdown/../../external/build-x86_64-linux-gnu/lowdown-build//man/man5
mkdir -p /home/lightning/src/lightning/external/lowdown/../../external/build-x86_64-linux-gnu/lowdown-build//share/lowdown/odt
install -m 0444 share/odt/styles.xml /home/lightning/src/lightning/external/lowdown/../../external/build-x86_64-linux-gnu/lowdown-build//share/lowdown/odt
install -m 0555 lowdown /home/lightning/src/lightning/external/lowdown/../../external/build-x86_64-linux-gnu/lowdown-build//bin
install -m 0555 lowdown-diff /home/lightning/src/lightning/external/lowdown/../../external/build-x86_64-linux-gnu/lowdown-build//bin
for f in man/lowdown.1.html man/lowdown-diff.1.html man/lowdown.5.html ; do \
    name=`basename $f .html` ; \
    section=${name##*.} ; \
    install -m 0444 man/$name /home/lightning/src/lightning/external/lowdown/../../external/build-x86_64-linux-gnu/lowdown-build//man/man$section ; \
done
make[1]: Leaving directory '/home/lightning/src/lightning/external/lowdown'
touch tools/md2man.sh
md2man doc/lightning-cli.1.md
md2man doc/lightningd-config.5.md
md2man doc/lightningd-rpc.7.md
md2man doc/lightning-addgossip.7.md
md2man doc/lightning-autoclean-once.7.md
md2man doc/lightning-autoclean-status.7.md
md2man doc/lightning-batching.7.md
md2man doc/lightning-bkpr-channelsapy.7.md
md2man doc/lightning-bkpr-dumpincomecsv.7.md
md2man doc/lightning-bkpr-inspect.7.md
md2man doc/lightning-bkpr-listaccountevents.7.md
md2man doc/lightning-bkpr-listbalances.7.md
md2man doc/lightning-bkpr-listincome.7.md
md2man doc/lightning-blacklistrune.7.md
md2man doc/lightning-check.7.md
md2man doc/lightning-checkmessage.7.md
md2man doc/lightning-checkrune.7.md
md2man doc/lightning-close.7.md
md2man doc/lightning-connect.7.md
md2man doc/lightning-commando.7.md
md2man doc/lightning-commando-blacklist.7.md
md2man doc/lightning-commando-listrunes.7.md
md2man doc/lightning-commando-rune.7.md
md2man doc/lightning-createonion.7.md
md2man doc/lightning-createinvoice.7.md
md2man doc/lightning-createrune.7.md
md2man doc/lightning-datastore.7.md
md2man doc/lightning-datastoreusage.7.md
md2man doc/lightning-decodepay.7.md
md2man doc/lightning-decode.7.md
md2man doc/lightning-deldatastore.7.md
md2man doc/lightning-delexpiredinvoice.7.md
md2man doc/lightning-delforward.7.md
md2man doc/lightning-delinvoice.7.md
md2man doc/lightning-delpay.7.md
md2man doc/lightning-disableinvoicerequest.7.md
md2man doc/lightning-disableoffer.7.md
md2man doc/lightning-disconnect.7.md
md2man doc/lightning-emergencyrecover.7.md
md2man doc/lightning-feerates.7.md
md2man doc/lightning-fetchinvoice.7.md
md2man doc/lightning-fundchannel.7.md
md2man doc/lightning-fundchannel_start.7.md
md2man doc/lightning-fundchannel_complete.7.md
md2man doc/lightning-fundchannel_cancel.7.md
md2man doc/lightning-funderupdate.7.md
md2man doc/lightning-addpsbtoutput.7.md
md2man doc/lightning-fundpsbt.7.md
md2man doc/lightning-getroute.7.md
md2man doc/lightning-invoice.7.md
md2man doc/lightning-invoicerequest.7.md
md2man doc/lightning-keysend.7.md
md2man doc/lightning-listchannels.7.md
md2man doc/lightning-listclosedchannels.7.md
md2man doc/lightning-listdatastore.7.md
md2man doc/lightning-listforwards.7.md
md2man doc/lightning-listfunds.7.md
md2man doc/lightning-listhtlcs.7.md
md2man doc/lightning-listinvoices.7.md
md2man doc/lightning-listinvoicerequests.7.md
md2man doc/lightning-listoffers.7.md
md2man doc/lightning-listpays.7.md
md2man doc/lightning-listpeers.7.md
md2man doc/lightning-listpeerchannels.7.md
md2man doc/lightning-showrunes.7.md
md2man doc/lightning-listsendpays.7.md
md2man doc/lightning-makesecret.7.md
md2man doc/lightning-multifundchannel.7.md
md2man doc/lightning-multiwithdraw.7.md
md2man doc/lightning-newaddr.7.md
md2man doc/lightning-notifications.7.md
md2man doc/lightning-offer.7.md
md2man doc/lightning-openchannel_abort.7.md
md2man doc/lightning-openchannel_bump.7.md
md2man doc/lightning-openchannel_init.7.md
md2man doc/lightning-openchannel_signed.7.md
md2man doc/lightning-openchannel_update.7.md
md2man doc/lightning-pay.7.md
md2man doc/lightning-parsefeerate.7.md
md2man doc/lightning-plugin.7.md
md2man doc/lightning-preapproveinvoice.7.md
md2man doc/lightning-preapprovekeysend.7.md
md2man doc/lightning-recover.7.md
md2man doc/lightning-recoverchannel.7.md
md2man doc/lightning-renepay.7.md
md2man doc/lightning-renepaystatus.7.md
md2man doc/lightning-reserveinputs.7.md
md2man doc/lightning-sendinvoice.7.md
md2man doc/lightning-sendonion.7.md
md2man doc/lightning-sendonionmessage.7.md
md2man doc/lightning-sendpay.7.md
md2man doc/lightning-setchannel.7.md
md2man doc/lightning-setconfig.7.md
md2man doc/lightning-setpsbtversion.7.md
md2man doc/lightning-sendcustommsg.7.md
md2man doc/lightning-signinvoice.7.md
md2man doc/lightning-signmessage.7.md
md2man doc/lightning-splice_init.7.md
md2man doc/lightning-splice_update.7.md
md2man doc/lightning-splice_signed.7.md
md2man doc/lightning-staticbackup.7.md
md2man doc/lightning-txprepare.7.md
md2man doc/lightning-txdiscard.7.md
md2man doc/lightning-txsend.7.md
md2man doc/lightning-unreserveinputs.7.md
md2man doc/lightning-utxopsbt.7.md
md2man doc/lightning-wait.7.md
md2man doc/lightning-waitinvoice.7.md
md2man doc/lightning-waitanyinvoice.7.md
md2man doc/lightning-waitblockheight.7.md
md2man doc/lightning-waitsendpay.7.md
md2man doc/lightning-withdraw.7.md
md2man doc/lightning-ping.7.md
md2man doc/lightning-stop.7.md
md2man doc/lightning-signpsbt.7.md
md2man doc/lightning-sendpsbt.7.md
md2man doc/lightning-getinfo.7.md
md2man doc/lightning-listtransactions.7.md
md2man doc/lightning-listnodes.7.md
md2man doc/lightning-listconfigs.7.md
md2man doc/lightning-help.7.md
md2man doc/lightning-getlog.7.md
md2man doc/reckless.7.md
md2man doc/lightning-listsqlschemas.7.md
md2man doc/lightning-sql.7.md
md2man doc/lightningd.8.md
md2man doc/lightning-hsmtool.8.md
install -m 644 doc/lightning-cli.1 /home/lightning/share/man/man1
install -m 644 doc/lightningd-config.5 /home/lightning/share/man/man5
install -m 644 doc/lightningd-rpc.7 doc/lightning-addgossip.7 doc/lightning-autoclean-once.7 doc/lightning-autoclean-status.7 doc/lightning-batching.7 doc/lightning-bkpr-channelsapy.7 doc/lightning-bkpr-dumpincomecsv.7 doc/lightning-bkpr-inspect.7 doc/lightning-bkpr-listaccountevents.7 doc/lightning-bkpr-listbalances.7 doc/lightning-bkpr-listincome.7 doc/lightning-blacklistrune.7 doc/lightning-check.7 doc/lightning-checkmessage.7 doc/lightning-checkrune.7 doc/lightning-close.7 doc/lightning-connect.7 doc/lightning-commando.7 doc/lightning-commando-blacklist.7 doc/lightning-commando-listrunes.7 doc/lightning-commando-rune.7 doc/lightning-createonion.7 doc/lightning-createinvoice.7 doc/lightning-createrune.7 doc/lightning-datastore.7 doc/lightning-datastoreusage.7 doc/lightning-decodepay.7 doc/lightning-decode.7 doc/lightning-deldatastore.7 doc/lightning-delexpiredinvoice.7 doc/lightning-delforward.7 doc/lightning-delinvoice.7 doc/lightning-delpay.7 doc/lightning-disableinvoicerequest.7 doc/lightning-disableoffer.7 doc/lightning-disconnect.7 doc/lightning-emergencyrecover.7 doc/lightning-feerates.7 doc/lightning-fetchinvoice.7 doc/lightning-fundchannel.7 doc/lightning-fundchannel_start.7 doc/lightning-fundchannel_complete.7 doc/lightning-fundchannel_cancel.7 doc/lightning-funderupdate.7 doc/lightning-addpsbtoutput.7 doc/lightning-fundpsbt.7 doc/lightning-getroute.7 doc/lightning-invoice.7 doc/lightning-invoicerequest.7 doc/lightning-keysend.7 doc/lightning-listchannels.7 doc/lightning-listclosedchannels.7 doc/lightning-listdatastore.7 doc/lightning-listforwards.7 doc/lightning-listfunds.7 doc/lightning-listhtlcs.7 doc/lightning-listinvoices.7 doc/lightning-listinvoicerequests.7 doc/lightning-listoffers.7 doc/lightning-listpays.7 doc/lightning-listpeers.7 doc/lightning-listpeerchannels.7 doc/lightning-showrunes.7 doc/lightning-listsendpays.7 doc/lightning-makesecret.7 doc/lightning-multifundchannel.7 doc/lightning-multiwithdraw.7 doc/lightning-newaddr.7 doc/lightning-notifications.7 doc/lightning-offer.7 doc/lightning-openchannel_abort.7 doc/lightning-openchannel_bump.7 doc/lightning-openchannel_init.7 doc/lightning-openchannel_signed.7 doc/lightning-openchannel_update.7 doc/lightning-pay.7 doc/lightning-parsefeerate.7 doc/lightning-plugin.7 doc/lightning-preapproveinvoice.7 doc/lightning-preapprovekeysend.7 doc/lightning-recover.7 doc/lightning-recoverchannel.7 doc/lightning-renepay.7 doc/lightning-renepaystatus.7 doc/lightning-reserveinputs.7 doc/lightning-sendinvoice.7 doc/lightning-sendonion.7 doc/lightning-sendonionmessage.7 doc/lightning-sendpay.7 doc/lightning-setchannel.7 doc/lightning-setconfig.7 doc/lightning-setpsbtversion.7 doc/lightning-sendcustommsg.7 doc/lightning-signinvoice.7 doc/lightning-signmessage.7 doc/lightning-splice_init.7 doc/lightning-splice_update.7 doc/lightning-splice_signed.7 doc/lightning-staticbackup.7 doc/lightning-txprepare.7 doc/lightning-txdiscard.7 doc/lightning-txsend.7 doc/lightning-unreserveinputs.7 doc/lightning-utxopsbt.7 doc/lightning-wait.7 doc/lightning-waitinvoice.7 doc/lightning-waitanyinvoice.7 doc/lightning-waitblockheight.7 doc/lightning-waitsendpay.7 doc/lightning-withdraw.7 doc/lightning-ping.7 doc/lightning-stop.7 doc/lightning-signpsbt.7 doc/lightning-sendpsbt.7 doc/lightning-getinfo.7 doc/lightning-listtransactions.7 doc/lightning-listnodes.7 doc/lightning-listconfigs.7 doc/lightning-help.7 doc/lightning-getlog.7 doc/reckless.7 doc/lightning-listsqlschemas.7 doc/lightning-sql.7 /home/lightning/share/man/man7
install -m 644 doc/lightningd.8 doc/lightning-hsmtool.8 /home/lightning/share/man/man8
install -m 644 README.md LICENSE /home/lightning/share/doc/c-lightning

1 CPU seems to be 100% busy with cargo build --quiet --bin cln-grpc for 10+ minutes now. I guess I'll just wait, but the whole build never took more than a few minutes on this machine.

Sjors commented 12 months ago

Update: the installer finished after about 15 minutes total. If this can't be made faster, then at least I think it should be more verbose and show some progress.

vincenzopalazzo commented 11 months ago

I suspect that was was cargo build --quiet --bin ... to take long time? I have the same behavior

This can be caused if you change your rust version, and the recompilation of all the crates (that are a few) is trigger

Sjors commented 11 months ago

The lightning user has its own rust installation. I didn't change it, but maybe CLN did?

Current version is cargo 1.67.1 (8ecd4f20a 2023-01-10)

daywalker90 commented 11 months ago

i think i saw a commit for cln that updated the rust version and it might be that cargo updated its repo which takes a long time (git resolving deltas). I gave it a quick search and found this: https://github.com/rust-lang/cargo/issues/11014

vincenzopalazzo commented 11 months ago

The lightning user has its own rust installation. I didn't change it, but maybe CLN did?

No, we did not, we do not force the rustc version at the moment but it is possible that cargo is rebuilding.

If this happens could you run a patch version of the Makefile where you remove the cargo build --silent and use just cargo build? I can make a patch for you if you like

Sjors commented 11 months ago

Good idea, I'll try that if it happens again.

Sjors commented 8 months ago

It's being slow again while upgrading to v24.02, but I can't find the cargo build --silent line.

Cargo version before install: cargo 1.67.1

Sjors commented 8 months ago

It was a bit faster than last time, only about 5 minutes. Cargo version after install is unchanged.

vincenzopalazzo commented 8 months ago

git grep returned some item

cln-rpc/Makefile:12:    cargo build ${CARGO_OPTS} --example cln-rpc-getinfo
cln-rpc/Makefile:15:    cargo build ${CARGO_OPTS} --example cln-plugin-startup
cln-rpc/Makefile:18:    cargo build ${CARGO_OPTS} --example cln-plugin-reentrant
plugins/Makefile:257:   cargo build ${CARGO_OPTS} --bin cln-grpc

Cargo version after install is unchanged.

The cln-rust library is changed in this case, causing rebuilding (maybe) but I do not know why it is taking all this time :/

Sjors commented 8 months ago

Oh I guess --silent is in CARGO_OPTS? Will try that next time.

I did a git clean -dfx, ./configure ... after this to see what would happen. This time it built quickly.