ipfs get doesn't work on files with "]" in the name

Arlodotexe commented 1 year ago

Checklist

[X] This is a bug report, not a question. Ask questions on discuss.ipfs.io.
[X] I have searched on the issue tracker for my bug.
[X] I am running the latest kubo version or have an issue updating.

Installation method

ipfs-update or dist.ipfs.tech

Version

Kubo version: 0.16.0
Repo version: 12
System version: amd64/windows
Golang version: go1.19.1

Config

Click to expand

```json { "API": { "HTTPHeaders": { "Access-Control-Allow-Origin": [ "http://example.io" ] } }, "Addresses": { "API": "/ip4/127.0.0.1/tcp/5001", "Announce": [], "AppendAnnounce": [], "Gateway": "/ip4/127.0.0.1/tcp/8080", "NoAnnounce": [], "Swarm": [ "/ip4/0.0.0.0/tcp/4001", "/ip6/::/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", "/ip6/::/udp/4001/quic" ] }, "AutoNAT": {}, "Bootstrap": [ "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt" ], "DNS": { "Resolvers": {} }, "Datastore": { "BloomFilterSize": 0, "GCPeriod": "1h", "HashOnRead": false, "Spec": { "mounts": [ { "child": { "path": "blocks", "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", "sync": true, "type": "flatfs" }, "mountpoint": "/blocks", "prefix": "flatfs.datastore", "type": "measure" }, { "child": { "compression": "none", "path": "datastore", "type": "levelds" }, "mountpoint": "/", "prefix": "leveldb.datastore", "type": "measure" } ], "type": "mount" }, "StorageGCWatermark": 90, "StorageMax": "10GB" }, "Discovery": { "MDNS": { "Enabled": true } }, "Experimental": { "AcceleratedDHTClient": true, "FilestoreEnabled": false, "GraphsyncEnabled": true, "Libp2pStreamMounting": false, "P2pHttpProxy": false, "StrategicProviding": false, "UrlstoreEnabled": false }, "Gateway": { "APICommands": [], "HTTPHeaders": { "Access-Control-Allow-Headers": [ "X-Requested-With", "Range", "User-Agent" ], "Access-Control-Allow-Methods": [ "GET" ], "Access-Control-Allow-Origin": [ "*" ] }, "NoDNSLink": false, "NoFetch": false, "PathPrefixes": [], "PublicGateways": null, "RootRedirect": "", "Writable": false }, "Identity": { "PeerID": "12D3KooWGCLGARP16J2Q67FeykYRxoAagJBVKG2c18ggkXQy7zgX" }, "Internal": {}, "Ipns": { "RecordLifetime": "", "RepublishPeriod": "", "ResolveCacheSize": 128 }, "Migration": { "DownloadSources": [], "Keep": "" }, "Mounts": { "FuseAllowOther": false, "IPFS": "/ipfs", "IPNS": "/ipns" }, "Peering": { "Peers": [ { "Addrs": [ "/dnsaddr/node-1.ingress.cloudflare-ipfs.com" ], "ID": "QmcFf2FH3CEgTNHeMRGhN7HNHU1EXAxoEk6EFuSyXCsvRE" }, { "Addrs": [ "/dnsaddr/node-2.ingress.cloudflare-ipfs.com" ], "ID": "QmcFmLd5ySfk2WZuJ1mfSWLDjdmHZq7rSAua4GoeSQfs1z" }, { "Addrs": [ "/dnsaddr/node-3.ingress.cloudflare-ipfs.com" ], "ID": "QmcfFmzSDVbwexQ9Au2pt5YEXHK5xajwgaU6PpkbLWerMa" }, { "Addrs": [ "/dnsaddr/node-4.ingress.cloudflare-ipfs.com" ], "ID": "QmcfJeB3Js1FG7T8YaZATEiaHqNKVdQfybYYkbT1knUswx" }, { "Addrs": [ "/dnsaddr/node-5.ingress.cloudflare-ipfs.com" ], "ID": "QmcfVvzK4tMdFmpJjEKDUoqRgP4W9FnmJoziYX5GXJJ8eZ" }, { "Addrs": [ "/dnsaddr/node-6.ingress.cloudflare-ipfs.com" ], "ID": "QmcfZD3VKrUxyP9BbyUnZDpbqDnT7cQ4WjPP8TRLXaoE7G" }, { "Addrs": [ "/dnsaddr/node-7.ingress.cloudflare-ipfs.com" ], "ID": "QmcfZP2LuW4jxviTeG8fi28qjnZScACb8PEgHAc17ZEri3" }, { "Addrs": [ "/dnsaddr/node-8.ingress.cloudflare-ipfs.com" ], "ID": "QmcfgsJsMtx6qJb74akCw1M24X1zFwgGo11h1cuhwQjtJP" }, { "Addrs": [ "/dnsaddr/node-9.ingress.cloudflare-ipfs.com" ], "ID": "Qmcfr2FC7pFzJbTSDfYaSy1J8Uuy8ccGLeLyqJCKJvTHMi" }, { "Addrs": [ "/dnsaddr/node-10.ingress.cloudflare-ipfs.com" ], "ID": "QmcfR3V5YAtHBzxVACWCzXTt26SyEkxdwhGJ6875A8BuWx" }, { "Addrs": [ "/dnsaddr/node-11.ingress.cloudflare-ipfs.com" ], "ID": "Qmcfuo1TM9uUiJp6dTbm915Rf1aTqm3a3dnmCdDQLHgvL5" }, { "Addrs": [ "/dnsaddr/node-12.ingress.cloudflare-ipfs.com" ], "ID": "QmcfV2sg9zaq7UUHVCGuSvT2M2rnLBAPsiE79vVyK3Cuev" } ] }, "Pinning": { "RemoteServices": {} }, "Plugins": { "Plugins": null }, "Provider": { "Strategy": "" }, "Pubsub": { "DisableSigning": false, "Router": "" }, "Reprovider": { "Interval": "6h", "Strategy": "all" }, "Routing": { "Methods": null, "Routers": null, "Type": "dht" }, "Swarm": { "AddrFilters": [], "ConnMgr": { "GracePeriod": "1m", "HighWater": 75, "LowWater": 45, "Type": "basic" }, "DisableBandwidthMetrics": false, "DisableNatPortMap": false, "RelayClient": {}, "RelayService": {}, "ResourceMgr": {}, "Transports": { "Multiplexers": {}, "Network": {}, "Security": {} } }, "foo": "foobar" } ```

Description

Repro steps:

Create an empty folder
Create an empty file in that folder, and name it anything. Include ] in the name.
Add the folder to Kubo with ipfs add -r ./path
Copy the folder CID from the terminal
Run ipfs get Qmfoo on the copied CID.
Observe the error: Error: invalid platform path: path components cannot contain any of [<>:"\|?*] : "test file].txt"

It's worth noting that this does not happen with the [ character.

CsterKuroi commented 1 year ago

reproduce in old version v0.9.1

after-egypt commented 1 year ago

On the Linux version 0.16.0 I didn't get any errors.

lidel commented 1 year ago

This is because this problem is specific to MS Windows.

Ref. https://learn.microsoft.com/en-us/windows/win32/fileio/naming-a-file:

The following reserved characters:

    < (less than)
    > (greater than)
    : (colon)
    " (double quote)
    / (forward slash)
    \ (backslash)
    | (vertical bar or pipe)
    ? (question mark)
    * (asterisk)

User-friendly way to resolve this is to

easy fix: provide user with meaningful error ("Microsoft Windows does not support" instead of vague "invalid platform path".
proper fix: add ipfs get --with-unsupported-filenames as opt-in (and mention it in the error returned to the user) – passing this parameter will allow Windows users to get data but replace unsupported characters in filenames with underscore (_).

franklovefrank commented 1 year ago

hi @Arlodotexe I am looking at this issue but cannot reproduce it. can you please post a stack trace?

susarlanikhilesh commented 1 year ago

@lidel this is not reproducible on windows.

Jorropo commented 1 year ago

@susarlanikhilesh files do not have names they are just bytes, directories have a list of name cid pairs.

You need to try to download the folder that contains a file with a ] not a file.

phillmac commented 1 year ago

@lidel the problem is, neither the right square bracket or left square bracket are restricted, and you can in fact have files and folder names contain those. the link nor the list of characters you referenced forbid using right and left square bracket.

Can we please get them removed from the filter list?

I specifically want the brackets preserved and not replaced with underscores because they are an important part of folder naming schemas that will break with them removed or replaced.

lidel commented 1 year ago

Ah, indeed. Tracked down the Windows-specific sanitization, it is in go-libipfs/tar (ipfs get RPC uses TAR behind the scenes): https://github.com/ipfs/go-libipfs/blob/302b2799386dea7afb72ba0b4c32a5c427215d06/tar/sanitize_windows.go#L12

Just like others noted the [ and ] are allowed in file names on Windows, but we try to sanitize them for some reason (was unable to find out why, code is 5 years old).

If anyone with Windows has time to test allowing them does not break ipfs get, please submit a fix (open a PR in go-libipfs and here)

ipfs / kubo