Maximum call stack size exceeded

Globalebill commented 12 months ago

error reported when upgrading BTCPay server to 1.11.4, when trying to admin the LN using RTL the following error is shown and in the logs. it seems the ln rest api keeps restarting every minute:

saubyk/c-lightning-rest:0.10.5 "/sbin/tini -g -- ./…" 10 hours ago Restarting (1) 32 seconds ago generated_clightning_bitcoin_rest_1

similar error to reported on https://github.com/Ride-The-Lightning/RTL/issues/1263

RTL admin page error:

ERROR: GetInfo => Get Info Error: {"name":"RequestError","message":"Error: getaddrinfo ENOTFOUND clightning_bitcoin_rest","cause":{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"clightning_bitcoin_rest"},"error":{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"clightning_bitcoin_rest"},"options":{"url":"http://clightning_bitcoin_rest:3001/v1/getinfo","rejectUnauthorized":false,"json":true,"headers":{},"method":"GET","qs":{},"simple":true,"resolveWithFullResponse":false,"transform2xxOnly":false}}

Error: RangeError: Maximum call stack size exceeded at /usr/src/app/utils/webSocketServer.js:81:15 at processTicksAndRejections (node:internal/process/task_queues:96:5) Waiting /root/.nbxplorer/btc_fully_synched to be created... The chain is fully synched Reading config file warn: --- Starting the cl-rest server --- warn: --- cl-rest api server is ready and listening on 0.0.0.0:3001 --- warn: --- cl-rest doc server is ready and listening on 0.0.0.0:4001 --- warn: {} /usr/src/app/utils/webSocketServer.js:81 throw new Error(err); ^

from docker logs generated_clightning_bitcoin_rest_1

Globalebill commented 12 months ago

as a note the LN is up and running and processing payments, bitcoin-lightning-cli.sh getinfo returns node info and in BTC Lightning section on BTCPay the node is showing as online, https://amboss.space/node/03ec989184aa887337f9766d3c8dfa90a26b82f642eb89758432a9d6f18489f1c8 is the node

NicolasDorier commented 11 months ago

So RTL and Lightning Rest works fine on my own personal node. This user says the problem started when he updated.

I asked @Globalebill to check lightning-cli and it works fine. The node doesn't seem to be restarting or anything. I confirm that lightning rest and the lightning deamon are sharing the same unix socket.

I tried to review lightning Rest code and didn't see anything wrong either. I am a bit lost on this one. @saubyk

Globalebill commented 11 months ago

i can confirm the LN is running fine, shows as online in BTCPay and on amboss.space, also receives and processing lightning payments no issues, the issue is we cannot admin the node using RTL admin panel from BTC Lightning page in BTCPay server it opens /rtl/error with the error: Getaddrinfo Enotfound Clightning_bitcoin_rest

saubyk commented 11 months ago

For some reason c-lightning-rest service is not able to start. Seems to be running into a websocketserver issue at boot time and failing to start.

@Globalebill which version of core lightning are you on?

Globalebill commented 11 months ago

@saubyk i am running the docker install of BTCPay version © BTCPay Server v1.11.4+79b2f1652 (Altcoins) upgraded from within BTCPay maintenance, from docker ps saubyk/c-lightning-rest:0.10.5 btcpayserver/lightning:v23.08 shahanafarooqui/rtl:0.14.0

as a note, v1.10 was working fine, issue only started after upgrade to 1.11.4

NicolasDorier commented 11 months ago

There isn't any saved data in ln rest config, and he already rebooted.

@Globalebill just to make sure, your server isn't a raspberry pi or something of the sort? It's just a normal VPS?

NicolasDorier commented 11 months ago

The Lightning REST service is connecting to core lightning, if it wasn't the case the error would have been ECONNREFUSED, not ENOTFOUND.

saubyk commented 11 months ago

{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"clightning_bitcoin_rest"}

Per chatGPT...this seems to be a DNS resolution problem.

ENOTFOUND: This error code suggests that a certain address or domain name cannot be found.
getaddrinfo: This is a system call used to resolve domain names into IP addresses.
hostname: The domain name or address that the system failed to resolve. In this case, it's clightning_bitcoin_rest.

It seems that the system is trying to resolve the hostname clightning_bitcoin_rest but cannot find it.

Globalebill commented 11 months ago

There isn't any saved data in ln rest config, and he already rebooted.

@Globalebill just to make sure, your server isn't a raspberry pi or something of the sort? It's just a normal VPS?

it is a dedicated server, running OS

Linux btcpay 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux

Globalebill commented 11 months ago

{"errno":-3008,"code":"ENOTFOUND","syscall":"getaddrinfo","hostname":"clightning_bitcoin_rest"}

Per chatGPT...this seems to be a DNS resolution problem.

ENOTFOUND: This error code suggests that a certain address or domain name cannot be found.

getaddrinfo: This is a system call used to resolve domain names into IP addresses.

hostname: The domain name or address that the system failed to resolve. In this case, it's clightning_bitcoin_rest.

It seems that the system is trying to resolve the hostname clightning_bitcoin_rest but cannot find it.

which IP should it resolve ? localhost ? maybe a hosts file entry

NicolasDorier commented 11 months ago

@saubyk I am surprised lightning rest is even using DNS.

But your may be right, there is a NODE setting on https://github.com/btcpayserver/btcpayserver-docker/blob/master/docker-compose-generator/docker-fragments/bitcoin-clightning.yml#L35

@Globalebill there isn't anything to do normally, it should just work. Can you DM me on btcapy chat? I think the DNS issue route is worth exploring

NicolasDorier commented 11 months ago

I see no env.NODE reference in the source code... weird.

NicolasDorier commented 11 months ago

@saubyk I checked with @Globalebill

He can ping the lightning node container. (clightning_bitcoin)
He can ping his own container (clightning_bitcoin_rest)

Might it be some trickery in NodeJS about DNS resolution?

getaddrinfo seems to be the linux function to call DNS. Somehow it doesn't resolve the host, but ping does...

Something is strange though look at the CURL request

{"url":"[http://clightning_bitcoin_rest:3001/v1/getinfo","rejectUnauthorized":false,"json":true,"headers":{},"method":"GET","qs":{},"simple":true,"resolveWithFullResponse":false,"transform2xxOnly":false}

Notice that the server is trying to call itself. Why ??

saubyk commented 11 months ago

So, I think I sent you down the wrong path. Apologies as it was late in the night for me and I wasn't thinking clearly.

Notice that the server is trying to call itself. Why ??

That log line is RTL's, which is trying to call c-lightning-rest url, so that's fine.

The root cause is not the DNS error, it's the fact the c-lightning-rest is not able to initiate completely, which results in RTL not being able to invoke the URL successfully.

I think the root cause is this:

/usr/src/app/utils/webSocketServer.js:81 throw new Error(err);

For some reason there is a webSocketServer error at the start which is resulting in c-lightning-rest failing to initialize.

Let me discuss this with @ShahanaFarooqui and see if we can dig any further here.

NicolasDorier commented 11 months ago

Ah yes, I was scratching my head, I thought the ENOTFOUND was sent by ln-rest... But it is RTL, it makes sense.

Note we tried to downgrade Rest down to 0.9.0: Same issues. This is the weirdest bug(?) I've seen for a while...

NicolasDorier commented 11 months ago

Ah, may it be related to the user having huge amount of invoices? You call listinvoices, it might blow up the stack somehow.

NicolasDorier commented 11 months ago

I tried to ask to use EXECMODE dev, but sadly the logs aren't useful at all and doesn't show information that can help.

ShahanaFarooqui commented 11 months ago

Ah, may it be related to the user having huge amount of invoices? You call listinvoices, it might blow up the stack somehow.

No, it doesn't seem to be the reason. I have tested RTL with million invoices in past and though it was slow, it still worked.

IMHO, the reason is recursive calling of subscribeToAnyInvoice and c-lightning-REST is calling it recursively on assumption that it is already able to connect to CLN node.

So the issue is that c-lightning-REST is unable to connect to CLN.

Globalebill commented 11 months ago

@ShahanaFarooqui @NicolasDorier @saubyk hi guys, just a quick message to see if any possible solutions might have been found for the above, many thanks in advance

ShahanaFarooqui commented 11 months ago

I haven’t find anything yet but I am actively working on this issue. So hopefully I should have an update by tomorrow.

ShahanaFarooqui commented 11 months ago

@Globalebill Can you please provide:

Complete error stack from c-lightning-REST container?
Total number of invoices (bitcoin-lightning-cli.sh listinvoices | jq '.invoices | length')
Max call stack size (good to know, not necessary though). Steps to get the size:
1. Go to c-lightning-REST shell (docker exec -it sh)
2. ps aux
3. Get the PID of "node cl-rest.js"
4. cat /proc//limits | grep "Max stack size"

@saubyk, @NicolasDorier Most probably, the issue is related to spread operator used in listInvoicesToSubscribe (...data.invoices.map(inv => inv.pay_index || 0)) but I would be more confident after reproducing and fixing it locally. So let us wait for the confirmation on number of invoices and stack size.

Globalebill commented 11 months ago

hi @ShahanaFarooqui

i dont see node cl-rest.js from ps aux but stack size for c-lightning/lightning_connectd, clightning channel PIDs and other clightning PIDs are all the same: Max stack size 8388608 unlimited bytes

stack size for node rtl /etc/BTCPayServer/btcpayserver-docker # cat /proc/14936/limits | grep "Max stack size" Max stack size 8388608 unlimited bytes

total invoices from bitcoin-lightning-cli.sh listinvoices | jq '.invoices | length' = 134804

dont know if this is important but, 0 bytes for clrest.js in /etc/BTCPayServer/btcpayserver-docker -rw-r--r-- 1 root root 0 Sep 13 08:56 clrest.js

log from generated_clightning_bitcoin_rest_1: warn: --- Starting the cl-rest server --- warn: --- cl-rest api server is ready and listening on 0.0.0.0:3001 --- warn: --- cl-rest doc server is ready and listening on 0.0.0.0:4001 --- warn: {} /usr/src/app/utils/webSocketServer.js:81 throw new Error(err); ^

Error: RangeError: Maximum call stack size exceeded at /usr/src/app/utils/webSocketServer.js:81:15 at processTicksAndRejections (node:internal/process/task_queues:96:5) Waiting /root/.nbxplorer/btc_fully_synched to be created... The chain is fully synched Reading config file warn: --- Starting the cl-rest server --- warn: --- cl-rest api server is ready and listening on 0.0.0.0:3001 --- warn: --- cl-rest doc server is ready and listening on 0.0.0.0:4001 --- warn: {} /usr/src/app/utils/webSocketServer.js:81 throw new Error(err); ^

Error: RangeError: Maximum call stack size exceeded at /usr/src/app/utils/webSocketServer.js:81:15 at processTicksAndRejections (node:internal/process/task_queues:96:5) Waiting /root/.nbxplorer/btc_fully_synched to be created... The chain is fully synched Reading config file warn: --- Starting the cl-rest server --- warn: --- cl-rest api server is ready and listening on 0.0.0.0:3001 --- warn: --- cl-rest doc server is ready and listening on 0.0.0.0:4001 ---

NicolasDorier commented 11 months ago

@ShahanaFarooqui I do not believe the issue is related to lightning rest not able to connect to c-lightning, because if that's the case, the error is different and clear on lightning rest. (ECONNECT error I believe)

btw, if you have some stealth debug-in-prod docker image we can run, you can just ask @Globalebill to replace the image in $BTCPAY_DOCKER_COMPOSE to see what is going on.

ShahanaFarooqui commented 11 months ago

@NicolasDorier, yeah the connection wasn't the issue. It was the size of invoices array.

Finally the error was reproduced locally and tested that the fix is working now. So we can directly release the next version 0.10.6 with the fix.

@saubyk Please confirm whenever the docker image is ready to be updated.

saubyk commented 11 months ago

Tagging 0.10.6 soon

NicolasDorier commented 11 months ago

ping @Globalebill I bumped lightning rest on our deployment

Ride-The-Lightning / c-lightning-REST

Maximum call stack size exceeded #199