Open mishmosh opened 1 month ago
I don't know about what hashing method is running under the hood, but this was the questions that I had when contemplating it.
Normally the entire file is passed through in chunks, and each chunk is passed in serial, and includes the previous chunk, which means that parallelizing this will not work. In the alternative it should be possible hash chunks in parallel, and then take the hashes of the parallel chunks and then do a sha256 of those chunks.
Additionally, I know that there is hardware acceleration and various hardware instructions for sha256, however I dont know how much you really will enjoy supporting different hardware, or what sort of applications which will automatically handle the accelerators and what level of support that they have.
I seem to remember it taking around 2 days to index ~4TB of models on a 2TB NVME cached ZFS 8 drive SAS array
@endomorphosis mind providing some answers to below questions? These will let us narrow down the areas that require optimization to better serve your use case (AI model data?):
ipfs add
or a different command else? Do you use any custom parameters?ipfs daemon
running while you perform the import? (or did you pass --offline
flag to skip announcements?)it looks like:
Mostly large language models mostly between 7GB to 70G using badgerds on huggingface repositories, IPFS daemon is running online and i use -r to archive the entire folder, e.g. https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF/tree/main
https://github.com/endomorphosis/ipfs_transformers/blob/main/ipfs_transformers/ipfs_kit_lib/ipfs.py#L73 /usr/local/bin/ipfs daemon --enable-gc --enable-pubsub-experiment
ipfs@workstation:/storage/cloudkit-models/Mixtral-8x7B-v0.1-GGUF-Q8_0$ time ipfs add mixtral-8x7b-v0.1.Q8_0.gguf added QmXVg3Ae6wRwbvkVqMwySyx6qdcVdjEy1iu8xHnwT9dAoB mixtral-8x7b-v0.1.Q8_0.gguf 46.22 GiB / 46.22 GiB [=============================================================================================================================================================================] 100.00% real 15m32.782s user 0m22.393s sys 1m54.226s
(copy from zfs to ssd, then zfs to zfs) barberb@workstation:/storage/cloudkit-models/Mixtral-8x7B-v0.1-GGUF-Q8_0$ time cp mixtral-8x7b-v0.1.Q8_0.gguf /tmp/mixtral.bin
real 1m59.033s user 0m0.681s sys 1m17.771s
barberb@workstation:/storage/cloudkit-models/Mixtral-8x7B-v0.1-GGUF-Q8_0$ time cp mixtral-8x7b-v0.1.Q8_0.gguf ../
real 2m28.763s user 0m0.624s sys 0m45.349s
Dual Xeon E5-V4 CPU with 8x WD gold and 1x Samsung 3x8 pcie lane enterprise NVME on ZFS I will try on my windows laptop workstation on only NVME using the desktop client in a second, and will update this, but it looks like ~ 50MB/s, and there the CPU utilization is always very low.
Windows nvme -> nvme (different devices) Intel(R) Xeon(R) CPU E3-1535M v6 using the desktop client 12m 30s 80% total cpu util (4 cores)
@endomorphosis Triage questions:
ipfs config show
? time ipfs add mixtral-8x7b-v0.1.Q8_0.gguf
taking the same amount of time if you do it in offline mode? (daemon shut down, or running in --offline
mode). In the past, it helped with importing things like 350GiB wikipedia (by skipping online tasks such as announcements, gc, etc) and performing expensive import this way and then restarting node might unblock you for now.syncwrite
to false
will speed things up significantly)devel@workstation:/tmp$ time ipfs add mixtral-8x7b-v0.1.Q8_0.gguf added QmXVg3Ae6wRwbvkVqMwySyx6qdcVdjEy1iu8xHnwT9dAoB mixtral-8x7b-v0.1.Q8_0.gguf 46.22 GiB / 46.22 GiB [===========================================================================================================================================================================================================] 100.00% real 14m8.824s user 12m19.404s sys 1m31.919s
running offline
fregg@workstation:/tmp$ time ipfs add mixtral-8x7b-v0.1.Q8_0.gguf added QmXVg3Ae6wRwbvkVqMwySyx6qdcVdjEy1iu8xHnwT9dAoB mixtral-8x7b-v0.1.Q8_0.gguf 46.22 GiB / 46.22 GiB [===========================================================================================================================================================================================================] 100.00% real 9m1.508s user 12m46.422s sys 2m11.760s
syncwrite to false
I do want to mention that I am writing a wrapper to import datasets into IPFS as well, where the number of files will be on the order of 7 million legal caselaw documents, I want to know if you have any feedback about what optimizations ought to be made for that instance?
Per #9678 I have an old branch with a bunch of optimizations for data import that I could rescue/cleanup. (edit: not for merging though as it hardcodes some stuff, just for using).
One low hanging fruit would be to add support for badger4 and pebble backends too. Badgerv1 is ages old.
Per #9678 I have an old branch with a bunch of optimizations for data import that I could rescue/cleanup. (edit: not for merging though as it hardcodes some stuff, just for using).
One low hanging fruit would be to add support for badger4 and pebble backends too. Badgerv1 is ages old.
I will look at that for my repositories https://github.com/endomorphosis/ipfs_transformers https://github.com/endomorphosis/ipfs_datasets
I would like to impress on your org (protocol labs) that I am trying to make this ergonomic for machine learning developers, and from what I have seen is that other projects (e.g. Bachalau, Iroh) are migrating away from libp2p / IPFS, because the project needs to be performance optimized.
I only have two more weeks I can spend on this hugging face bridge, and right now its more of a life raft in case the government decides to overregulate machine learning, than it is a viable solution that ML devs would turn to, but if something reasonably effective for decentralized low latency MLops existed I would probably stop development and just buy filecoins.
Checklist
Description
This is a followup on user @endomorphosis's comment in the Filecoin community discussions about IPFS hashing being slow.
Per @lidel in an ipfs-steering conversation on 2 April 2024: