search

ipfs-shipyard / ipfs-pack

ipfs-pack filesystem packing tool
MIT License
73 stars 16 forks source link

Maintaining Dynamic Packs #34

Open nothingismagick opened 6 years ago

nothingismagick commented 6 years ago

So, I have a fundamental question about maintaining packs. Can you, @whyrusleeping give me some advice?

I am having to manually rebuild my 34GB pack almost every day. This size of a pack is kind of the limit of usefulness, because that means that I have to tear down the daemon while I am rebuilding - which takes about 15 minutes. Our use case, however, foresees Packs growing to 3TB size (the size of a current consumer HDD) - and this is a problem, because packing that many files TAKES A VERY LONG TIME®.

These are the cases that I would like to look into:

  1. An unhashed file needs to be created and added to the pack.
  2. A hashed file needs to have its metadata changed. (Is this even possible? Can I just switch out the first block...?)
  3. A hashed file needs to be in a different directory.
  4. A hashed file was transcoded wrong, and needs to be unpinned (EVERYWHERE) and replaced.

In all cases, it is not really acceptable to kill the main ipfs-pack thread. Can I manually change or add a hash to the PackManifest instead of rebuilding the pack - while still serving? I have found two files that I need to replace (my file ingest was borked, need to do it again for the main file and transcode it for the second file). For now rebuilding the pack would work, but like I said above, our data volume is going to explode and we can't be offline for 30 minutes. It would be great to have commands like:

 $ ipfs-pack inject ./folder1/somefile.mp4 
 $ ipfs-pack move ./folder1/somefile.mp4 ./folder2/somefile.mp4
 $ ipfs-pack purge ./folder2/somefile.mp4

Of course, this could all be managed by integrating a watchfolder pattern and updating the PackManifest accordingly. I am not sure how possible that is, but it would really make the daemon superpowered. Like:

ipfs pack serve --watchfs

Of course it is also possible that I am trying to find ways to use ipfs-pack that are inappropriate anti-patterns and for which other tools exist. If that is the case, I would love to hear about a better way to do this...

PalinuroSec commented 4 years ago

hello, i have the same issue with a 450gb archive updated many times every day (a gnu/linux distribution repository)

do we have any chance to see this kind of use cases supported any soon?

hsanjuan commented 4 years ago

Unfortunately ipfs-pack is effectively not maintained these days. We'd be happy to take a new captain to move the project forward.

PalinuroSec commented 4 years ago

unfortunately i have no experience with the ipfs and ipfs-pack codebase and i have little to no time to drive it, but i would love to give some help with what fits my skills