search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.03k stars 3k forks source link

ipfs.pubsub.ls() never shows more than 5 channels #8632

Closed christroutner closed 2 years ago

christroutner commented 2 years ago

Checklist

Installation method

ipfs-update or dist.ipfs.io

Version

go-ipfs version: 0.11.0
Repo version: 11
System version: amd64/linux
Golang version: go1.16.12

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [],
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "2GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": false,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "12D3KooWMo6DWVXnBQEizaVCubghoP4ZsBzDn68DsnC7CPmCQRvn"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "none"
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/10.0.0.0/ipcidr/8",
      "/ip4/100.64.0.0/ipcidr/10",
      "/ip4/169.254.0.0/ipcidr/16",
      "/ip4/172.16.0.0/ipcidr/12",
      "/ip4/192.0.0.0/ipcidr/24",
      "/ip4/192.0.2.0/ipcidr/24",
      "/ip4/192.168.0.0/ipcidr/16",
      "/ip4/198.18.0.0/ipcidr/15",
      "/ip4/198.51.100.0/ipcidr/24",
      "/ip4/203.0.113.0/ipcidr/24",
      "/ip4/240.0.0.0/ipcidr/4",
      "/ip6/100::/ipcidr/64",
      "/ip6/2001:2::/ipcidr/48",
      "/ip6/2001:db8::/ipcidr/32",
      "/ip6/fc00::/ipcidr/7",
      "/ip6/fe80::/ipcidr/10"
    ],
    "ConnMgr": {
      "GracePeriod": "",
      "HighWater": 30,
      "LowWater": 10,
      "Type": ""
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": true,
    "EnableHolePunching": true,
    "RelayClient": {
      "Enabled": true
    },
    "RelayService": {},
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  },
  "preload": {
    "enabled": false
  }
}

Description

I am building an app that generates many pubsub connections. I'm using ipfs-http-client to control a go-ipfs v0.11.0 node. When I run ipfs.pubsub.ls() only returns 5 results. Every time. I'm sure there should be more than 5 connections, but it's only ever 5.

As new connections are made, older connections roll out. I'm not sure if this means the pubsub connection is severed (which is what I'm afraid of), or if I'm only seeing a window of the full list. But here is an example of this rolling array:

status: New message published to pubsub channel psf-ipfs-coordination-001
status: Announced self on psf-ipfs-coordination-001 pubsub channel at 1/1/2022, 7:54:15 PM
status: New pubsub message recieved from 12D3KooWMo6DWVXnBQEizaVCubghoP4ZsBzDn68DsnC7CPmCQRvn on channel psf-ipfs-coordination-001
...
status: pubsubs: [
  "psf-ipfs-coordination-001",
  "12D3KooWT2tDg2pHwKz84Htzh2KsfSUZbZXmoAUFnaSwUVrDsag3",
  "/orbitdb/zdpuAkxr81CZgu6jV3kakHwQb9VCKBFnWZWaUZoxQ6kFHBEcq/12D3KooWT2tDg2pHwKz84Htzh2KsfSUZbZXmoAUFnaSwUVrDsag322010204",
  "/orbitdb/zdpuArUPt3FMsdpuzhCchdYo5HFTxsWUaAzQjK3A1MNM8LwXG/12D3KooWMo6DWVXnBQEizaVCubghoP4ZsBzDn68DsnC7CPmCQRvn22010120",
  "/ipfs-pubsub-direct-channel/v1/12D3KooWMo6DWVXnBQEizaVCubghoP4ZsBzDn68DsnC7CPmCQRvn/12D3KooWT2tDg2pHwKz84Htzh2KsfSUZbZXmoAUFnaSwUVrDsag3"
]

...
status: New pubsub message recieved from 12D3KooWT2tDg2pHwKz84Htzh2KsfSUZbZXmoAUFnaSwUVrDsag3 on channel psf-ipfs-coordination-001
status: announcement recieved from 12D3KooWT2tDg2pHwKz84Htzh2KsfSUZbZXmoAUFnaSwUVrDsag3
...
status: New message published to pubsub channel psf-ipfs-coordination-001
status: Announced self on psf-ipfs-coordination-001 pubsub channel at 1/1/2022, 7:54:46 PM
status: New pubsub message recieved from 12D3KooWMo6DWVXnBQEizaVCubghoP4ZsBzDn68DsnC7CPmCQRvn on channel psf-ipfs-coordination-001
status: pubsubs: [
  "/ipfs-pubsub-direct-channel/v1/12D3KooWMo6DWVXnBQEizaVCubghoP4ZsBzDn68DsnC7CPmCQRvn/12D3KooWT2tDg2pHwKz84Htzh2KsfSUZbZXmoAUFnaSwUVrDsag3",
  "psf-ipfs-coordination-001",
  "12D3KooWT2tDg2pHwKz84Htzh2KsfSUZbZXmoAUFnaSwUVrDsag3",
  "/orbitdb/zdpuAkxr81CZgu6jV3kakHwQb9VCKBFnWZWaUZoxQ6kFHBEcq/12D3KooWT2tDg2pHwKz84Htzh2KsfSUZbZXmoAUFnaSwUVrDsag322010204",
  "/orbitdb/zdpuArUPt3FMsdpuzhCchdYo5HFTxsWUaAzQjK3A1MNM8LwXG/12D3KooWMo6DWVXnBQEizaVCubghoP4ZsBzDn68DsnC7CPmCQRvn22010120"
]

Note that from one log entry to another, the elements in the pubsubs array seems to shift. This is the output of ipfs.pubsub.ls().

welcome[bot] commented 2 years ago

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

Finally, remember to use https://discuss.ipfs.io if you just need general support.

christroutner commented 2 years ago

I just confirmed with the command line app for go-ipfs that the behavior is the same. Running ipfs pubsub ls shows 6 entries. They rotate, but it only ever shows 6.

aschmahmann commented 2 years ago

@christroutner this issue seems to be https://github.com/ipfs/js-ipfs/issues/3983 and related to browser HTTP limitations.

If we can help the browser out reasonably from go-ipfs then we can have an issue for dealing with that (cc @lidel)

christroutner commented 2 years ago

Thanks for the tip, @aschmahmann! This section on configuring an agent seems to be the fix that I need.

What I find odd, is that I observed the same behavior when using the go-ipfs CLI.

I'll close the Issue now, but I'll add to it if I discover new information as I try to incorporate http agents in my node.js apps.