Lightning node IP not reachable from clearnet when using Cloudflare tunnel

[danielcharrua]

Hey guys, I was testing Lightning through Cloudflare tunnels and I've found some issues. Docs here https://docs.btcpayserver.org/Docker/cloudflare-tunnel/

When Cloudflare tunnel option is activated, the BTCPay instance can be accessed from clearnet and Tor without issues. The domain is resolved and tunneled to the BTCPay instance.

The problem comes when using Lightning and trying to connect to the node, to open a channel for example. Again when Cloudflare tunnel is activated, the "node info" page showed by BTCPay shows clearnet and Tor access. On the clearnet part the IP shown is the one provided by Cloudflare for the tunnel, the problem is that the IP is a shared and rotating one for the tunneling process.

You can see the problem here if connecting to my testnet node (you can't): 03cd163d76dac7fcca19317a2aae08191cf2a03e309d99ea234ba4454fbac0f0bd@188.114.97.8:9735

Some peer erros trying to connect to my node on the clearnet:

cegp@btcserver:~$ lightning-cli connect 03cd163d76dac7fcca19317a2aae08191cf2a03e309d99ea234ba4454fbac0f0bd@188.114.97.8:9735
{
   "code": 401,
   "message": "All addresses failed: 188.114.97.8:9735: Connection establishment: Connection timed out. 7mo4hxvvflixzro3aofexhfgqs6nj75jx3udbqsq666aqz2khfvykmad.onion:9735: need a proxy. "
}

At the same time peers using the onion address can reach my lightning node and connect with no issues here. Works as expected.

So we need to find a workaround to this issue and also inform it on docs for other users to know. One could be that when using the cloudflare tunnels the public node info page shows only the tor option and not the clearnet. Maybe there's an option to disable LND public IP and only using the onion address. I don't know if BTCPay would understand this and show only the onion address.

Another issue encountered when using BTCPay For Woocommerce V2 (through clearnet using cloudflare tunnel), the popup window to make the payment only shows the clearnet node info and not the Tor one

As you see, I need some help over here to solve this issue. Thank you.

[Goro2030]

Maybe there's a configuration in CloudFlare somewhere to make this work? I'm sure the BTCPay team tested this before releasing the segment.

[danielcharrua]

Hello @Goro2030 I don't think this is related to Cloudflare config but because of how tunnels works. It doesn't assign you a static IP and that's a must for LND (you can also setup a host but LND will try to figure out the IP behind).

The tunnel is resolved from inside Cloudflare when you send a request to the domain, you can't send a request directly to the IP because Cloudflare will not know where to send the request (the IP is also used for other clients also).

I think that BTCPay Server, when used behind a Cloudflare tunnel and you wanted to see LND node info, it must display only the Tor address.

The same with the BTCPay For Woocommerce V2 WordPress plugin. Even if you are connecting with BTCPay Server on checkout through clearnet, the LND node info must be only Tor.

Maybe @dennisreimann or @Kukks can give us some light here. Let us know your thoughts.

[dennisreimann]

I think the problem starts already with us not knowing that the Cloudflare fragment is being used there and that the address hence might need adaptation. This will be more involved to solve properly I think.

[danielcharrua]

@dennisreimann when using the Cloudflare fragment an env variable is used: CLOUDFLARE_TUNNEL_TOKEN maybe this is the flag you are talking about?

[dennisreimann]

@NicolasDorier Can we maybe use the real IP header value for this? Otherwise we could use the presence of the CLOUDFLARE_TUNNEL_TOKEN env value as a hint to discard the clearnet node info before displaying it.

[coinforensics]

Maybe adding the IP to RawExternalIPs in LND via LND_EXTRA_ARGS could help:

Add an ip:port to the list of local addresses we claim to listen on to peers. If a port is not specified, the default (9735) will be used regardless of other parameters

[danielcharrua]

@coinforensics maybe ExternalHosts also but the problem is still the same, I think you can't communicate to a Cloudflare's tunnel IP directly, you need to use the hostname (I could be wrong about this).

Another issue could be that we have something missing on the Cloudflare part. At this moment all http and https traffic is being redirected to the nginx container and BTCPay Server UI is rendered OK, but what about traffic going to the LND node on 9735? With the actual config we are routing also the node traffic?

[coinforensics]

As far as I now, it is not possible for a Lightning node to operate behind a "regular" Cloudflare tunnel. While Cloudflare does offer support for non-HTTP(s) traffic, this feature is only available with their paid Spectrum Enterprise Plan.

[danielcharrua]

What exactly is the traffic on LND? TCP? Looking at the docs on tunnels I see that TCP is supported (as other protocols), it can be used like tcp://localhost:9735

https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide/local/local-management/ingress/#supported-protocols

[danielcharrua]

Maybe the best here is to let the user select how he wanted to expose the LND node connection to the world.

For example, a user running an e-commerce store can use BTCPay Server in clearnet to make things easy to buyers that don't use Tor. But maybe with a simple option on the BTCPay Server UI (instead of detecting the use of any env) have a simple option to select what type of connection you wanted to show to the world on the LND node (Tor or clearnet) and have some control there.

This will solve the issue on detecting any env variable or the use of cloudflared and simply add an option to the UI. Also solves this issue when people run BTCPay Server through Cloudflare using LND.

[mountainlove]

Hy

I had to change my Provider an now i have my BtcPay Server up and running over a CF tunnel. I think i'm running in the same issue here, since i can not open some Lightning Channels to external Nodes anymore. My test is "pending" for days now and i'm unable to reach port 9735 over "subdomain.domain.xy:9735".

A answer in Mattermostchat sugestet, that i wil not be possible anyway with CF...

So i like to ask you for a hint. Is it not possible over all to open some Channels over clearnet an CF tunnels or should it work and i will have a bug in my config? Before changing to CF everything workes fine incl. opening channels. So the only thing i changed ist the connection through CF.

Thank you for your help.

[dennisreimann]

Cloudflare only handles requests to port 80 (http) and 443 (https), not anything beyond that. For Lightning peer connections you will need port 9735, which does not work with Cloudflare.

[danielcharrua]

Then we need to do something with the documentation on CloudFlare tunnels, maybe add this information for other users to know and use only Lightning with Tor.

Also would be nice that when the tunnel (container) is activated, the options that are shown to the en user are only Tor. I posted this issue and some ideas in 2022 but no solutions yet. Maybe there is no so many users using the cloudflare tunnel or no interest.

[dennisreimann]

It seems to be possible to open up other ports as well, at least if I understand this doc correctly. Can someone check whether or not this allows to circumvent the peer conecction problems?

https://developers.cloudflare.com/fundamentals/reference/network-ports/

[danielcharrua]

From my understanding that is not for tunnels. The tunnels have some different routing, see https://github.com/btcpayserver/btcpayserver-docker/issues/716#issuecomment-1531192471

[mountainlove]

I tried to get it working @dennisreimann . But you are right @danielcharrua... It seems that if you bypass the CF proxy you could open port 9735, but then 80 and 443 are not working anymore with the tunnel. Like i understand it might be possible with a paid plan and some additional CF Tools. So i think it is really not possible to open channels over clearnet and so one, when using a CF tunnel. If anyone has a working Solution, i would like to test it out...

How ever: In the Time of upcomming 5G Connections wich are making it easy to host your own BtcPayServer nearly everywhere and the used CG-NAT Solutions on most Networks from Providerside, CF Tunnels are providing really easy solution to expose your Server and run eg. a public Node or receive payments from Clearnetclients.

I hope, there will be a Solution one Day since i don't have the skills to solve it... Above was a Idea wich includes the Pageheader... Sounds not bad...

[danielcharrua]

@dennisreimann @NicolasDorier is there a way to select what node information the payment page shows? If using CF tunnels showing clearnet address or IP is not working. But if we can show users, only tor connection details, they could connect and using a CF tunnel will be an option.

Let me know your thoughts. Thanks!

[dennisreimann]

We'll remove the legacy checkout view in the upcoming BTCPay Server v2. We will have to fix it on another level, until then I still think using the reverse SSH tunnel approach is the best way to fix it.

https://docs.btcpayserver.org/Deployment/ReverseSSHtunnel/