search

ipfs / kubo

An IPFS implementation in Go
https://docs.ipfs.tech/how-to/command-line-quick-start/
Other
16.03k stars 3k forks source link

Pinning Interface #1711

Open sroerick opened 9 years ago

sroerick commented 9 years ago

Hi. I've had some trouble pinning a large number of items, so Juan asked me to open a ticket here detailing my ideal UI.

  1. It would be neat if "ipfs pin add -r" could support multiple files, and/or text input. so i can use "ipfs pin add -r [listofhashes]" or "ipfs pin add -r [hashlist.txt]". Ideally, IPFS would run "ipfs refs" and retrieve a list of all the sub-hashes that are also to be pinned. Then, it would return something like "pinning 42 hashes, 237 subhashes." It would be nice to see a progress bar for each hash, as well as an indication that a hash being pinned indirectly had another hash as a parent.
  2. Error Handling. There MUST be some kind of error handling here. If something is being pinned and no blocks have been added within a given period of time, it should time out. At the end of the pin, IPFS should notify the user which pins have failed, with the option to export a list of broken pins to a file. Concurrency would also be useful here: E.G "ipfs pin add -r" would simultaneously try to pin multiple hashes. This way, if there was one hash with only a few seeds or no seeds, it would not prevent the rest of the pinning mechanism from completing. Concurrency should happen across hashes, NOT subhashes. If one hash containing many hashes is down, it is likely that most of the hashes contained within are also down. Identifying which hashes are unseeded is key here. This ability also will ultimately help the overall health of the network, by more easily identifying unseeded hashes we can more easily upload and fix.
  3. API support: For apps that are trying to build on top of this platform, having an API for pinning is pretty essential. Generally, the features outlined here should be duplicated in the API.
  4. "ipfs pin add -r" should also function in the same way as running "ipfs refs [hash] && ipfs get [hash] && ipfs pin add -r [hash]". Running multiple commands to successfully cache files is pretty wack.
  5. ipfs refs, ipfs get, and ipfs pin ought have clearer and more unified syntax. They're a little baffling. e.g. ipfs get runs recursively automatically, where ipfs pin does not. ipfs pin supports multiple hashes, while get and refs do not.
whyrusleeping commented 9 years ago
  1. It would be neat if "ipfs pin add -r" could support multiple files, and/or text input. so i can use "ipfs pin add -r [listofhashes]" or "ipfs pin add -r [hashlist.txt]". Ideally, IPFS would run "ipfs refs" and retrieve a list of all the sub-hashes that are also to be pinned. Then, it would return something like "pinning 42 hashes, 237 subhashes." It would be nice to see a progress bar for each hash, as well as an indication that a hash being pinned indirectly had another hash as a parent.

ipfs pin add -r can indeed take multiple arguments! even from a list of hashes in a file like ipfs pin add -r < hashlist. the pin code actually does run the same fetching code as ipfs refs except that pin sets a timeout on the retreival. This timeout is a tenative decision that we're still kinda up in the air about. As for printing the number of subhashes, we cant really do that until we have retrieved all of the refs. We do want to have some sort of progress bar though.

  1. Error Handling

Yeah, we need better error handling.

  1. API support

Everything you can do on the CLI can be done via the API.

  1. "ipfs pin add -r" should also function in the same way as running...

as mentioned earlier. It does, it just has a timeout in it, that you can set to whatever you want with the global --timeout flag. (ipfs pin add -r <hash> --timeout=100m)

whyrusleeping commented 9 years ago
  1. ipfs refs, ipfs get, and ipfs pin ought have clearer and more unified syntax. They're a little baffling. e.g. ipfs get runs recursively automatically, where ipfs pin does not. ipfs pin supports multiple hashes, while get and refs do not.

pin and refs can both support multiple arguments, but for get its unclear what the meaning of the -o option would be. I think more docs on the role of each command would be good.

jbenet commented 9 years ago

like what i'm hearing here. particularly progress bars per pin, that would be very nice indeed.

it may be easier to do a lot of these upgrades once we separate out ipfs-core and the commands.

also yeah, @whyrusleeping has been pushing to make pin add -r the default (i.e. ipfs pin add <foo> is recursive by default). this would mean adding -d or something, for direct. (am still ambivalent but yeah, i see the confusion)

MichaelMure commented 8 years ago

The gist of the linked bug above is that data requesting command are blocking for both the API (you have to keep a http request running) and the CLI (you have to keep a terminal running) to get something, which is IMHO bad for a ipfs user as well as app builder like me.

whyrusleeping commented 8 years ago

Primary deliverables here would be a progress bar, and improved error handling