search

bacalhau-project / bacalhau

Compute over Data framework for public, transparent, and optionally verifiable computation
https://docs.bacalhau.org
Apache License 2.0
716 stars 89 forks source link

Running bacalhau serve adds lots of peers to private ipfs swarm #3095

Closed chrisalys closed 5 months ago

chrisalys commented 11 months ago

I wish to run bacalhau as a private network along side a private ipfs swarm, but when I run bacalhau serve I suddenly get lots of unwanted peers.

Ipfs set up:

Then I install and run bacalhau serve

Lots of ipfs swarm peers turn up...

See complete proc below:

root@1647a005-9c3a-42ca-92ae-8e430b6de687:~# wget https://dist.ipfs.tech/kubo/v0.24.0/kubo_v0.24.0_linux-amd64.tar.gz
--2023-12-12 23:53:34--  https://dist.ipfs.tech/kubo/v0.24.0/kubo_v0.24.0_linux-amd64.tar.gz
Resolving dist.ipfs.tech (dist.ipfs.tech)... 209.94.78.1, 2602:fea2:3::1
Connecting to dist.ipfs.tech (dist.ipfs.tech)|209.94.78.1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 39124911 (37M) [application/gzip]
Saving to: ‘kubo_v0.24.0_linux-amd64.tar.gz’

kubo_v0.24.0_linux-amd64.tar.gz         100%[============================================================================>]  37.31M  15.3MB/s    in 2.4s

2023-12-12 23:53:37 (15.3 MB/s) - ‘kubo_v0.24.0_linux-amd64.tar.gz’ saved [39124911/39124911]

root@1647a005-9c3a-42ca-92ae-8e430b6de687:~# tar -xvzf kubo_v0.24.0_linux-amd64.tar.gz
kubo/LICENSE
kubo/LICENSE-APACHE
kubo/LICENSE-MIT
kubo/README.md
kubo/install.sh
kubo/ipfs
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~# cd kubo/
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# sudo bash install.sh
Moved ./ipfs to /usr/local/bin
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# ipfs --version
ipfs version 0.24.0
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# ipfs init -e
generating ED25519 keypair...done
peer identity: 12D3KooWSCKtdF6fd14mhthKzshcm79xiQazj9EV3cCLJDYcYeLD
initializing IPFS node at /root/.ipfs
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# ipfs bootstrap rm --all
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb
removed /dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt
removed /ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
removed /ip4/104.131.131.82/udp/4001/quic-v1/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# ipfs id | grep ID
        "ID": "12D3KooWSCKtdF6fd14mhthKzshcm79xiQazj9EV3cCLJDYcYeLD",
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# cat << EOF > /etc/systemd/system/ipfs.service
>
> [Unit]
> Description=InterPlanetary File System (IPFS) daemon
> Documentation=https://docs.ipfs.io/
>
> [Service]
> Type=notify
> Environment="IPFS_PATH=~/.ipfs"
> ExecStart=/usr/local/bin/ipfs daemon
> ExecStop=/usr/local/bin/ipfs shutdown
> Restart=on-failure
> RestartSec=15s
>
> [Install]
> WantedBy=multi-user.target
> EOF
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo#
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo#
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# systemctl enable ipfs --now
Created symlink /etc/systemd/system/multi-user.target.wants/ipfs.service → /etc/systemd/system/ipfs.service.
root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# ipfs swarm addrs
12D3KooWSCKtdF6fd14mhthKzshcm79xiQazj9EV3cCLJDYcYeLD (9)
        /ip4/127.0.0.1/tcp/4001
        /ip4/127.0.0.1/udp/4001/quic-v1
        /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiDHI4bgiuXU6IYSSiSsdnKT88t3vRepNiYD0mZZSgSYIQ/certhash/uEiDxTBrx30mjdLIBmgwUWEPBrOki_xDL_eJ4bcqNkRtBfQ
        /ip4/192.168.122.135/tcp/4001
        /ip4/192.168.122.135/udp/4001/quic-v1
        /ip4/192.168.122.135/udp/4001/quic-v1/webtransport/certhash/uEiDHI4bgiuXU6IYSSiSsdnKT88t3vRepNiYD0mZZSgSYIQ/certhash/uEiDxTBrx30mjdLIBmgwUWEPBrOki_xDL_eJ4bcqNkRtBfQ
        /ip6/::1/tcp/4001
        /ip6/::1/udp/4001/quic-v1
        /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDHI4bgiuXU6IYSSiSsdnKT88t3vRepNiYD0mZZSgSYIQ/certhash/uEiDxTBrx30mjdLIBmgwUWEPBrOki_xDL_eJ4bcqNkRtBfQ

Watch for 10 mins.. no change above..

listening looks like:

lsof -i -P -n | grep LISTEN
systemd       1            root  289u  IPv6   71899      0t0  TCP *:6566 (LISTEN)
systemd       1            root  310u  IPv6   65590      0t0  TCP *:22 (LISTEN)
systemd-r  1057 systemd-resolve   14u  IPv4   61565      0t0  TCP 127.0.0.53:53 (LISTEN)
brltty    15860            root   20u  IPv6 1439484      0t0  TCP [::1]:4101 (LISTEN)
brltty    15860            root   21u  IPv4 1523219      0t0  TCP 127.0.0.1:4101 (LISTEN)
ipfs      17823            root   15u  IPv4 1516506      0t0  TCP *:4001 (LISTEN)
ipfs      17823            root   17u  IPv6 1516507      0t0  TCP *:4001 (LISTEN)
ipfs      17823            root   25u  IPv4 1508125      0t0  TCP 127.0.0.1:5001 (LISTEN)
ipfs      17823            root   26u  IPv4 1508126      0t0  TCP 127.0.0.1:8080 (LISTEN)
cupsd     18002            root    6u  IPv6 1437070      0t0  TCP [::1]:631 (LISTEN)
cupsd     18002            root    7u  IPv4 1437071      0t0  TCP 127.0.0.1:631 (LISTEN)
#install bacalhau
curl -sL https://get.bacalhau.org/install.sh | bash

ipfs swarm addrs are still the same a this point, then...

bacalhau serve --node-type=requester --peer=none --ipfs-connect=/ip4/127.0.0.1/tcp/5001 --private-internal-ipfs=false

Apart from a lot of warnings, which I'm sure people are aware of.. suddenly my private ipfs swarm has got lots of peers!

root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# ipfs swarm addrs | wc -l
1888

e.g.

root@1647a005-9c3a-42ca-92ae-8e430b6de687:~/kubo# ipfs swarm addrs | head -30
12D3KooW9qq9znLdxDkDe95T1Xv7vtPyHpRR9XF5ZZkz48kxFak4 (3)
        /ip4/154.12.242.103/tcp/4001
        /ip4/154.12.242.103/udp/4001/quic
        /ip4/154.12.242.103/udp/4001/quic-v1
12D3KooW9rgXJJUSAQ4UyEr9dYCfskDUGhjeQg1VLtMXVUvGfuvM (10)
        /ip4/104.207.153.253/udp/4001/quic-v1
        /ip4/104.207.153.253/udp/4001/quic-v1/webtransport/certhash/uEiAD8OhYYtrMXoTU7GqoDUj0DLE_wYSfXjYikqbe3vPKnQ/certhash/uEiArAe1ucyEiipo4fW7j9TVl75umqFX_Nu5lSQsnNndnnA
        /ip4/127.0.0.1/tcp/4001
        /ip4/127.0.0.1/udp/4001/quic
        /ip4/127.0.0.1/udp/4001/quic-v1
        /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiAD8OhYYtrMXoTU7GqoDUj0DLE_wYSfXjYikqbe3vPKnQ/certhash/uEiArAe1ucyEiipo4fW7j9TVl75umqFX_Nu5lSQsnNndnnA
        /ip6/::1/tcp/4001
        /ip6/::1/udp/4001/quic
        /ip6/::1/udp/4001/quic-v1
        /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiAD8OhYYtrMXoTU7GqoDUj0DLE_wYSfXjYikqbe3vPKnQ/certhash/uEiArAe1ucyEiipo4fW7j9TVl75umqFX_Nu5lSQsnNndnnA
12D3KooW9sYdMkefShXKGT7SPcwLMgQ4NpUphMLGtRoBJmL48boG (3)
        /ip4/175.45.180.102/tcp/4001
        /ip4/175.45.180.102/udp/4001/quic
        /ip4/175.45.180.102/udp/4001/quic-v1
12D3KooW9tMu3udX3zCnd17cdHjB55akzTL3DfbxGPStDGMJT3mj (3)
        /ip4/193.201.15.63/udp/4001/quic
        /ip4/193.201.15.63/udp/4001/quic-v1
        /ip4/193.201.15.63/udp/4001/quic-v1/webtransport/certhash/uEiCdpsZxPYjxoQozxQ7kolV7UPOZBRrrNnM5r6OCOhgbQQ/certhash/uEiA4z0Ej2cTgz5rNAIcfl6wcrdBjXEdfV00f0w24pTAfvQ
12D3KooW9tQVcxbjywWpxccY7i3RvpkXdDHaVifHK5oT7qcMr8fS (3)
        /ip4/45.77.5.109/tcp/4001
        /ip4/45.77.5.109/udp/4001/quic
        /ip4/45.77.5.109/udp/4001/quic-v1
12D3KooW9wKrxn2Co8B1fpWPyiZmK5DNMeNCqkJsaBzJTAwmA7fq (4)
        /ip4/54.151.182.40/tcp/4001
        /ip4/54.151.182.40/udp/4001/quic
chrisalys commented 11 months ago

I tried with a swarmkey as to make it more private...

ipfs swarm addrs starts like this:

12D3KooWAbVphZ5ZJQgTqgJ8WtRVDBLKVryiP2DQDYSb4oHMSshF (3)
        /ip4/10.147.2.4/tcp/4001
        /ip4/127.0.0.1/tcp/4001
        /ip6/::1/tcp/4001
12D3KooWCVHjQfVnBbjS5aDZzNjQBjn8oJzDSXY8AiApABcr3hqt (1)
        /ip4/10.147.2.6/tcp/4001

good..

I then run: `bacalhau serve --node-type=requester,compute --peer=none --ipfs-connect=/ip4/127.0.0.1/tcp/5001 --private-internal-ipfs=false

I still get a few added, for about a minute or so (as below), then they disappear leaving just local and my one other private test node (as above), I guess this is because the swarmkey makes the swarming give up.

But of course, I don't want to advertise outside at all...

Q: Can we stop bacalhau adding any peers to the local private ipfs network???

12D3KooWAQpZzf3qiNxpwizXeArGjft98ZBoMNgVNNpoWtKAvtYH (2)
        /ip4/35.245.161.250/tcp/4001
        /ip4/35.245.161.250/udp/4001/quic
12D3KooWAbVphZ5ZJQgTqgJ8WtRVDBLKVryiP2DQDYSb4oHMSshF (3)
        /ip4/10.147.2.4/tcp/4001
        /ip4/127.0.0.1/tcp/4001
        /ip6/::1/tcp/4001
12D3KooWBCBZnXnNbjxqqxu2oygPdLGseEbfMbFhrkDTRjUNnZYf (2)
        /ip4/34.145.201.224/tcp/4001
        /ip4/34.145.201.224/udp/4001/quic
12D3KooWCVHjQfVnBbjS5aDZzNjQBjn8oJzDSXY8AiApABcr3hqt (1)
        /ip4/10.147.2.6/tcp/4001
12D3KooWH3rxmhLUrpzg81KAwUuXXuqeGt4qyWRniunb5ipjemFF (2)
        /ip4/35.245.215.155/tcp/4001
        /ip4/35.245.215.155/udp/4001/quic
12D3KooWJM8j97yoDTb7B9xV1WpBXakT4Zof3aMgFuSQQH56rCXa (2)
        /ip4/35.245.41.51/tcp/4001
        /ip4/35.245.41.51/udp/4001/quic
12D3KooWLfFBjDo8dFe1Q4kSm8inKjPeHzmLBkQ1QAjTHocAUazK (2)
        /ip4/34.86.254.26/tcp/4001
        /ip4/34.86.254.26/udp/4001/quic
simonwo commented 11 months ago

The issue is that even when the user supplies an IPFS Connect flag, we still ask the IPFS node to connect to any swarm peers that are part of our config: https://github.com/bacalhau-project/bacalhau/blob/7e74d437ddcdd1d7304c671bff4e7f74a92b84f9/cmd/cli/serve/util.go#L168-L175 And by default, the Bacalhau production node peers will be in the config: https://github.com/bacalhau-project/bacalhau/blob/7e74d437ddcdd1d7304c671bff4e7f74a92b84f9/pkg/config/configenv/production.go#L57-L77

So for a start, a workaround to get the behaviour you want should be to explicitly set the swarm peers to an empty array in your config file, i.e. 

Node:
    IPFS:
        SwarmAddresses: []

But then yes, we should think about whether this behaviour makes sense. Under what circumstances will a user have asked to connect to an existing IPFS node and then expect Bacalhau to modify the configuration of that node? That seems unlikely, so we can probably remove messing with any config in the --ipfs-connect case.

chrisalys commented 11 months ago

Thanks for this. Good explanation.

Perhaps a few typical scenario based yaml configs come with the install; reducing decisions in code and will be generally less opaque, as people can see the configs laid out in the yaml... alter without compiling, etc.

thanks again

simonwo commented 11 months ago

Perhaps a few typical scenario based yaml configs come with the install; reducing decisions in code and will be generally less opaque, as people can see the configs laid out in the yaml... alter without compiling, etc.

+1 to this idea, providing example config files with comments would be very helpful.

aronchick commented 10 months ago

@michaelhoepler - please add this to docs that need writing

wdbaruni commented 5 months ago

embedded ipfs node has been deprecated in favour of connecting to your own node https://github.com/bacalhau-project/bacalhau/pull/4061