search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.16k stars 3.01k forks source link

fuse pb regression: could not convert protobuf or raw node #9044

Open bmwiedemann opened 2 years ago

bmwiedemann commented 2 years ago

Checklist

Installation method

built from source

Version

go-ipfs version: 0.13.0
Repo version: 12
System version: amd64/linux
Golang version: go1.17.11

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [
      "/ip4/10.162.191.176/tcp/4001",
      "/ip4/176.9.1.211/tcp/34176",
      "/ip4/10.162.191.176/udp/4001/quic",
      "/ip4/176.9.1.211/udp/34176/quic"
    ],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/2620:113:80c0:8130:ec4:7aff:fe57:1808/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/2620:113:80c0:8130:ec4:7aff:fe57:1808/udp/4001/quic"
    ]
  },
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.236.179.241/tcp/4001/p2p/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM",
    "/ip4/128.199.219.111/tcp/4001/p2p/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu",
    "/ip4/104.236.76.40/tcp/4001/p2p/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64",
    "/ip4/178.62.158.247/tcp/4001/p2p/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd",
    "/ip6/2604:a880:1:20::203:d001/tcp/4001/p2p/QmSoLPppuBtQSGwKDZT2M73ULpjvfd3aZ6ha4oFGL1KrGM",
    "/ip6/2400:6180:0:d0::151:6001/tcp/4001/p2p/QmSoLSafTMBsPKadTEgaXctDQVcqN88CNLHXMkTNwMKPnu",
    "/ip6/2604:a880:800:10::4a:5001/tcp/4001/p2p/QmSoLV4Bbm51jM9C4gDYZQ9Cy3U6aXMJDAbzgu2fzaDs64",
    "/ip6/2a03:b0c0:0:1010::23:1001/tcp/4001/p2p/QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"
  ],
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "90GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Experimental": {
    "FilestoreEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "QUIC": false,
    "ShardingEnabled": false,
    "UrlstoreEnabled": true
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoFetch": false,
    "PathPrefixes": [],
    "RootRedirect": "",
    "Writable": false
  },
  "Identity": {
    "PeerID": "QmbBikAJAnTrry9XEVxc1G3YTVi13M1GHU2fk1m1Tag4Am"
  },
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": "",
    "StrictSignatureVerification": false
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {
      "GracePeriod": "20s",
      "HighWater": 900,
      "LowWater": 600,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "DisableRelay": false,
    "EnableAutoNATService": false,
    "EnableAutoRelay": false,
    "EnableRelayHop": false
  }
}

Description

I noticed that a number of my files stopped working in the fuse mount after upgrade from go-ipfs 0.12.0 to 0.13.0 e.g.

> ls /ipfs/bafybeighf5coeur6nejmyjzdatfcs6ohpo6ea5lhvhmrfecy6lvpird34m
2022-06-17T08:38:34.876+0200    ERROR   fuse/ipfs       readonly/readonly_unix.go:108   could not convert protobuf or raw node
2022-06-17T08:38:34.876+0200    ERROR   fuse/ipfs       readonly/readonly_unix.go:108   could not convert protobuf or raw node
ls: cannot access '/ipfs/bafybeighf5coeur6nejmyjzdatfcs6ohpo6ea5lhvhmrfecy6lvpird34m': No such file or directory

ipfs object get /ipfs/bafybeighf5coeur6nejmyjzdatfcs6ohpo6ea5lhvhmrfecy6lvpird34m
{"Links":[{"Name":"","Hash":"bafkreigbmv5dwljg4kfdrh3pqt7t57d3pykszknoyv2ykv2pen6mp352uu","Size":262144},{"Name":"","Hash":"bafkreidmx67g3ykxw6qb2sklnstrxnontt4plskqj27c6yjbpnntpgs5lq","Size":262144},{"Name":"","Hash":"bafkreid74vxd3o7a5za4hwixaoasfuesnymzqeyxvmxfxpejbrv2aya34i","Size":262144},{"Name":"","Hash":"bafkreiea3iy3qq343ugx6bik7yk7txuu7t62qg7hfwmskaokpllh2dgbay","Size":83759}],"Data":"\u0008\u0002\u0018��5 ��\u0010 ��\u0010 ��\u0010 ��\u0005"}

while ipfs cat still works fine.

It seems to affect all the bafybei (pb) objects, but the bafkrei (raw data) files work fine.

bmwiedemann commented 2 years ago

also reproduced with official https://github.com/ipfs/go-ipfs/releases/download/v0.13.0/go-ipfs_v0.13.0_linux-amd64.tar.gz

lidel commented 2 years ago

@Jorropo wonder if this might be related to invalid codec mapping in go-cid, see BREAKING CHANGE fix in https://github.com/ipfs/go-cid/releases/tag/v0.2.0

Really concerning that we have no tests for FUSE that would catch this regression – whoever will be fixing this, should also add sharness test for UnixFS DAG that has both dag-pb and raw file items.

Konubinix commented 1 year ago

By debugging with ipfs 0.12 and 0.17, using the followings command

echo a | ipfs add -
added Qmbvkmk9LFsGneteXk3G7YLqtLVME566ho6ibaQZZVHaC9 
cat /ipfs/Qmbvkmk9LFsGneteXk3G7YLqtLVME566ho6ibaQZZVHaC9
2023-05-04T13:02:23.318Z        ERROR   fuse/ipfs       readonly/readonly_unix.go:108   could not convert protobuf or raw node

I could find that there is a difference of path taken in https://github.com/ipfs/kubo/blob/a6f446a4bab78b3d8b814474f1532f7bebcdc9c1/fuse/readonly/readonly_unix.go#L92

In ipfs 0.12, nd is of type github.com/ipld/go-ipld-prime/datamodel.Node(*github.com/ipfs/go-unixfsnode._PathedPBNode) *{_substrate: *github.com/ipld/go-codec-dagpb._PBNode...}

It then goes in here https://github.com/ipfs/kubo/blob/master/fuse/readonly/readonly_unix.go#L95 and get the substrate and run mdag.ProtoNodeConverter on it and get the fnd.

In ipfs 0.17, nd is of type github.com/ipld/go-ipld-prime/datamodel.Node(*github.com/ipfs/go-unixfsnode/file.singleNodeFile) *{Node: github.com/ipld/go-ipld-prime/datamodel.Node(*github.com/ipfs/go-unixfsnode/data._Bytes) *{x: []uint8 len: 2, cap: 506, [97,10]}}

Then it goes in https://github.com/ipfs/kubo/blob/master/fuse/readonly/readonly_unix.go#L95 and ProtoNodeConverter returns an error because nd is not a dagpb.PBNode

I'm not familiar with those concepts, so the amount of knowledge to have to debug this issues seems a bit overwhelming to me.

But with some guidance, may be I can help.

Most of my day to day life relies on using ipfs mount and using ipfs content like any other files. It has been very annying to have to resort to ipfs cat, ipfs get or the gateway to just read a file.

Therefore, I am quite motivated to make this work. Of course, during the time I can spare to this task.

I guess my next action in this project would be to try to understand what are those PBNode about, right?

bmwiedemann commented 1 year ago

PB probably stands for ProtoBuf which is the encoding-method of metadata. E.g. the DAG-PB (Directed Acyclic Graph) contains a list of entries that each have Size, Name, CID

In (unrelated) https://github.com/bmwiedemann/ipfs-iso-jigsaw/blob/master/ipfsjigsaw.py I also construct a custom PBNode from python.

Konubinix commented 1 year ago

Thank you

Konubinix commented 1 year ago

Digging into ipfs, ipld, PBNode, ADL etc looks like a pretty big rabbit hole to me.

I realized that, in term of functionality, the ipfs gateway is very similar to the fuse mount point: it provides access to the data. So I figure there should be also a lookup-like method in the gateway code.

I intuitively think that if I find this code, I can either

Does this idea make sense?

bmwiedemann commented 1 year ago

This problem still exists in 0.21.0

Konubinix commented 1 year ago

BTW, my ongoing investigation on the subject is here -> https://konubinix.eu/braindump/f9f882e4-0d84-453e-b3aa-5e34e411b90d?title=ipfs_mount_0_12_2_does_not_work_for_several_files

It moves into my someday/maybe list, because I have more important matters now, and learning about ipfs, ipld, boxo and stuffs like that needs quite a lot of cognitive load.

But because I have the frustration everyday of having to work around this issue, I will definitely continue working on this, so stay tuned :-). Of course, if someone fix it before I do, I would be verrrrry grateful.

bmwiedemann commented 1 year ago

still broken in 0.22.0

bmwiedemann commented 1 year ago

I ran a git bisect between 0.12.2 and 0.13.0 and found

34aac1ee43168d18456ec8659f0024841541b0d2 is the first bad commit
commit 34aac1ee43168d18456ec8659f0024841541b0d2
Author: Adin Schmahmann <adin.schmahmann@gmail.com>
Date:   Wed May 4 11:32:38 2022 -0400

    chore: update deps

 go.mod | 15 +++++++--------
 go.sum | 35 ++++++++++++++++++++---------------
 2 files changed, 27 insertions(+), 23 deletions(-)

Specifically reverting this part, fixes fuse:

-   github.com/ipfs/go-graphsync v0.11.0
+   github.com/ipfs/go-graphsync v0.13.1
-   github.com/ipfs/go-unixfsnode v1.1.3
+   github.com/ipfs/go-unixfsnode v1.4.0

and I further narrowed it down to 12 commits in https://github.com/ipfs/go-unixfsnode/compare/v1.1.3...v1.2.0

bmwiedemann commented 1 year ago

It seems, https://github.com/ipfs/go-unixfsnode/pull/14 by @willscott broke fuse.

willscott commented 1 year ago

Thanks for tracking this down, @bmwiedemann !

It's not obvious to me exactly how the update of unixfsprime from 2 years ago broke fuse.

One thing that is immediately clear is that this is an unhelpful error message, in that the actual error is not printed. https://github.com/ipfs/kubo/pull/10102 It would be great to run with that PR and see what the actual error is

bmwiedemann commented 1 year ago

with 10102 ontop of 0.22.0 I get

2023-08-25T15:02:12.975+0200    ERROR   fuse/ipfs       readonly/readonly_unix.go:107   could not convert protobuf or raw node: expected protobuf dag node
ls: cannot access '/ipfs/bafybeiax2m67v77asgjtbdhyiqa37b7x6zmylz3ry6bziwwt63hrlyqz4a': No such file or directory
willscott commented 1 year ago

My guess at fixing the immediate regression is https://github.com/ipfs/go-unixfsnode/pull/67

rvagg commented 1 year ago

new version of go-unixfsnode released with that change and a few other fixes

lidel commented 11 months ago

Reopening as could not convert protobuf or raw node: expected protobuf dag node is still a problem for some CIDs in FUSE. Seems it was happening since 0.23.

Details and repro steps: https://github.com/ipfs/kubo/pull/10243#issuecomment-1841895501 – take a look if you had good intuition on this before :pray:

Luflosi commented 11 months ago

See also #10242, which I created just for this new problem.

hacdias commented 11 months ago

@Luflosi your issue is for the path prefix problem. Unfortunately, it also turns out that this issue (#9044) came back too. Both are being handled in #10243.

lidel commented 6 months ago

Triage notes:

We won't have bandwidth to fix this any time soon, but if someone has bandwidth to fix the remaining bug, and open PR with tests, I'm happy to review.