search

ipfs-inactive / archives

[ARCHIVED] Repo to coordinate archival efforts with IPFS
https://awesome.ipfs.io/datasets
183 stars 24 forks source link

cdnjs #35

Open victorb opened 9 years ago

victorb commented 9 years ago

Would love to have a IPFS compatible fork of https://github.com/cdnjs/cdnjs serving files via IPFS. Super large repository though but will give it a try to develop the integration locally.

jbenet commented 9 years ago

cc @lgierth

davidar commented 9 years ago

On hold until ipfs add performance improves, too many small files

eminence commented 8 years ago

Ok well I bit the bullet and added this to IPFS. Here is the result: http://ipfs.io/ipfs/QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7

A few observations. Note that for all of the below comments, the daemon was not running, and I was IPFS version 0.3.11-dev

$ time ipfs object new unixfs-dir
QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn

real    0m14.855s
user    0m19.136s
sys     0m0.632s

This means that is took 6 hours to call object patch add-link for every top-level directory. I believe this 15 second delay is just some startup delay, so if I had the daemon running at this point, I believe this process would have been much faster (since there would be no startup delay)

achin@diax:~/devel$ du -sh cdnjs/ajax/libs/ ~/.ipfs/
22G     cdnjs/ajax/libs/
13G     /home/achin/.ipfs/

I took a look at every single hash to see what ones were duplicated. In total, about 180000 difference hashes were dedups at least once in this tree. The most dedupd hash was a small 1x1 PNG file with 12376 links. The second most deduped file is an empty file with 696 links.

I've uploaded a file that lists all dedups. I haven't done the analysis to see if it would account for the savings reported by du

$ time ipfs pin add -r QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7
pinned QmZJKsLpebYqHApRLeoLhj2NsXZ2JoXqhSWkxafxaBXYu7 recursively

real    7m24.915s
user    5m16.676s
sys     0m32.560s
davidar commented 8 years ago

@eminence awesome, thanks for tackling this and writing up the details :)

Cc @whyrusleeping @rht @diasdavid

eminence commented 8 years ago

Second day follow up notes:

But IPNS is still too slow. Or rather, it is not consistently fast. I wrote a script that requests the same file several times, and records how long it takes. The data for 60 requests is here. The summery is that most of requests are quick (about 0.5 seconds or less). But some requests take 10 seconds, others up to 60 seconds. When your website is loading many different assets (like cdnjs libs), this can result is very noticeable delays.

davidar commented 8 years ago

Note that I'd love a mode in ipfs add that just gives me the top-level hash of the thing that I'm adding, so I can dispense with the tail -1 stuff.

I think ipfs add -q does that

This manual tree management is very doable, but also annoying. I admit I've not been following the Files API stuff -- would it help here?

Probably, yes.

jbenet commented 8 years ago

I think ipfs add -q does that

ipfs add -q outputs only hashes. still need tail -n1. but yeah, we should add a flag that outputs only the last root. maybe --only-root ? or -Q, --root-quiet? idk.

jbenet commented 8 years ago

This manual tree management is very doable, but also annoying. I admit I've not been following the Files API stuff -- would it help here?

yes, very much so.

jbenet commented 8 years ago

But IPNS is still too slow. Or rather, it is not consistently fast. I wrote a script that requests the same file several times, and records how long it takes. The data for 60 requests is here. The summery is that most of requests are quick (about 0.5 seconds or less). But some requests take 10 seconds, others up to 60 seconds. When your website is loading many different assets (like cdnjs libs), this can result is very noticeable delays.

yes, ipns is still very slow. there's caching (the fast results), but we need to fix this at the dht query level

eminence commented 8 years ago

After testing with 0.4.0, the experience was much better!

If anyone is running a 0.4.0 node, the result is here: QmRnvPSCNmYHdYQAo6JUWJPW8uVQv7z6D9nSQmw5qbHVWy

jbenet commented 8 years ago

Good news. Though still waaaay too slow for my liking.

Adding concurrent add will help here. I believe we have not added this. On Tue, Dec 15, 2015 at 09:08 Andrew Chin notifications@github.com wrote:

After testing with 0.4.0, the experience was much better!

-

Memory usage was totally under control, and mostly stable the entire time.

I was able to add the entire libs directory at once. This took about 9 hours, which is much quicker than before. This isn't a totally fair comparison, since I used different hardware for each test (0.4.0 versus 0.3.8), but even trying to take into account the hardware differences, I believe 0.4.0 was faster

Rescanning the entire libs directory, after the initial add, took about 50 minutes. This would represent the time needed to do an incremental update. Obviously using the git diff trick from above would improve this, but 50 minutes seems pretty acceptable to me.

— Reply to this email directly or view it on GitHub https://github.com/ipfs/archives/issues/35#issuecomment-164775232.

eminence commented 8 years ago

For me, the gold standard of merkledags is probably git :) So I timed how long it takes to add this directory tree into a new git repo -- 41 minutes! About an order of magnitude faster than IPFS.

whyrusleeping commented 8 years ago

@eminence yeah, we've got a little ways to go still, but keep in mind that git uses a faster hashing algorithm, and doesnt chunk objects. I have one more changeset to apply that should get close to leveling the playing field. Just have to polish it a bit.

rht commented 8 years ago

In some cases, e.g. lots of 1MB files instead of 1KB files, ipfs is way faster than git https://github.com/ipfs/go-ipfs/pull/1973#issuecomment-157738243

rht commented 8 years ago

git uses a faster hashing algorithm, and doesnt chunk objects.

The chunking equivalent in git would be git gc, but only after several revisions?

jbenet commented 8 years ago

we should consider moving the default to https://blake2.net/ it's designed for this.

rht commented 8 years ago

That would explain rsync's speed, since it uses blake2. Perhaps it is possible to go ~O(rsync) with this.

It would be more effective, though, to use blake2 into 0.4.0 before its release (combining all the incompatible changes into one release?).

jbenet commented 8 years ago

It would be more effective, though, to use blake2 into 0.4.0 before its release (combining all the incompatible changes into one release?).

blake2 isnt incompatible. https://github.com/jbenet/multihash :)

jbenet commented 8 years ago

oh i guess it is, because it's not included in the 0.3.x codebase, right.

jbenet commented 8 years ago

yeah it would be nice, but idk if we can land it in time. @whyrusleeping wants to ship 0.4.0 soon. would be nice to add blake2 and ipld support, but not sure if we'll get there in time

eminence commented 8 years ago

Even though git is super fast here, ipfs isn't unusablely slow. as an end user, i have so many more things on my wishlist that are more important to me than speed

(edit: but i do indeed understand the desire to put all breaking changes into 1 release)

davidar commented 8 years ago

@rht What percentage of the total runtime is currently consumed by the hash function?

rht commented 8 years ago

...I did instead with testing dev0.4.0+blake2b. There is more cpu consumption and a speedup but not noticeable due to the jitter like in https://github.com/ipfs/go-ipfs/pull/2039#issuecomment-164978328. Perhaps it could be significant once dev0.4.0 goes to ~O(git) or ~O(rsync). Though I could have just checked the runtime percentage of the hashing.

Currently, both ipfs and git are slow for adding large things. edit: s/moving/adding/

Kubuxu commented 8 years ago

I re-archived it on my own. Available under fs:/ipns/cdnjs.ipfs.ovh

Notes: It took about 6h, I had one crash during that time.

xioustic commented 8 years ago

Anyone interested in a deterministic, efficient node script that wraps the command-line to do the git pull / git diff / ipfs add / ipfs object patch manual work that does this into the existing index you last put into the IPFS network? Could also handle the memory issue (restarting the daemon or sets of potential offending processes when they get near a threshold).

About four months ago I did this and I'll need to dig up my work sitting on one of my VPSes; simple dumb 'ipfs add -r' every time I got pulled was quite expensive. Might be reusable in other related archive targets (other big git repos that are more archive repositories than single codebases). Just requires 'git', 'ipfs' and 'node' on the path of the indexing machine. Maybe someone has already done this or perhaps the ipfs internals have improved enough to remedy the need for this?

I just found out about this "IPFS Archives" project at the Decentralized Web Summit, pretty exciting. @whyrusleeping still doing the workshop on IPFS Archives and Versioning, so tagging him.

Kubuxu commented 8 years ago

Me and @Magik6k are working on adding the cdnjs currently.

I added whole cdnjs some time ago, it is available under fs:/ipns/cdnjs.ipfs.ovh but I think most of that data is currently gone (due to problems with local IPFS repo).

Adding cdnjs is nice stress test for IPFS: https://github.com/ipfs/go-ipfs/issues/2823 https://github.com/ipfs/go-ipfs/issues/2828

Kubuxu commented 8 years ago

Note about publishing the cdnjs:

We shouldn't use /ipns/ to reference the cdnjs in HTML. IPNS currently doesn't work with browser level caching (no HTTP 304 response code if path is IPNS path).

notslang commented 8 years ago

Updated CDNJS hash: https://ipfs.io/ipfs/QmPJnEf5933cXteZmaMJkphCW1CtpcMMVx7N6rUr8cZAok And a little script for generating it, in case anyone else wants to try:

#!/bin/bash
HASH=$(ipfs object new unixfs-dir)
for FILE in cdnjs/ajax/libs/*; do
  LIB=$(basename $FILE)
  echo "adding $LIB"
  LIB_HASH=$(ipfs add -r -H -q "$FILE" | tail -n 1)
  HASH=$(ipfs object patch $HASH add-link $LIB $LIB_HASH)
done
echo "final hash: $HASH"

...However, this doesn't handle symlinks, which do exist in CDNJS. I've gotta decide how to deal with those.

eminence commented 8 years ago

What needs handling exactly (with regard to symlinks)?

notslang commented 8 years ago

If you click on one right now (like https://ipfs.io/ipfs/QmPJnEf5933cXteZmaMJkphCW1CtpcMMVx7N6rUr8cZAok/zocial) it's broken, whereas https://cdnjs.cloudflare.com/ajax/libs/zocial/1.2.0/css/zocial.css works.

eminence commented 8 years ago

Ahh. I see, yes. I would just pretend the symlink doesn't exist and just add the contents of the directory. Let IPFS's intrinsic de-dup handle it from there

eminence commented 8 years ago

BTW, I am trying to pin this hash, but I can't download everything. Are you still seeding it, @slang800 ?

Kubuxu commented 8 years ago

@eminence you are better of, instead of pinning it right away, doing ipfs refs -r $HASH, this way you will see progress, just make sure to set high GC limit (the cdnjs will be about 20GB) and pin it afterwards.

eminence commented 8 years ago

Actually, I am already doing that (using ipfs refs). I was trying to run ipfs refs QmWLDMm1CC2E5f7M5dcbi9hW9K9XuckFUfGc39GRgGYtQC, but after about 30 hours of waiting nothing came back. But I just restarted the command just now and it returned instantly. Not sure what that means... ipfs bug?

eminence commented 8 years ago

Or maybe I just can't connect to @slang800 node?

When I run ipfs dht findprovs QmeQYktmYVqAbRju1boAyimC9J1dAACYY7h5KNqfEEEamb, nothing at all is returned. But if I request that hash via the ipfs.io public gateways, it loads almost immediately. Then ipfs dht findprovs returns something:

> ipfs dht findprovs QmeQYktmYVqAbRju1boAyimC9J1dAACYY7h5KNqfEEEamb
QmVyqFjQJTqVmKRBk4sL9F9Af7fCRdA9YNK845NSHRD8zJ
QmSoLer265NRgSp2LA3dPaeykiS1J6DifTC88f5uVQKNAd
> ipfs swarm peers |wc -l
117

So I'm a little confused about what's actually happening here

Kubuxu commented 8 years ago

It is possible that you can't connect with @slang800 node directly. Maybe his node failed to penetrate NAT and only connected to SolarNet nodes/can't accept connections, this would be why you can see things work via ipfs.io but not through your own instance.

notslang commented 8 years ago

Sorry - I was hosting this on my desktop and turned it off this morning to move my desk to the opposite side of the room. It should be up now. :D

Kubuxu commented 8 years ago

@slang800 can you share your peerID (result of ipfs id -f"<id>\n") for debugging?

notslang commented 8 years ago

Sure, it's QmVKZAcpqrrQoBsc7eyEFnqFjHTDAUdJWMTBDJ34jR7ueU

victorb commented 7 years ago

Submitted an updated build based on github.com/cdnjs/cdnjs commit 4fabd85c986d57a61e0fbd8504cf15d67f60ada6 here: https://github.com/ipfs/archives/pull/82

New hash would be: QmRrnfFUgx81KZR9ibEcxDXgevoj9e5DydB5v168yembnX - https://ipfs.io/ipfs/QmRrnfFUgx81KZR9ibEcxDXgevoj9e5DydB5v168yembnX

It's stored at Pollux right now.

PeterDaveHello commented 7 years ago

@cdnjs maintainer here, anything I can help here?

victorb commented 7 years ago

@PeterDaveHello Thanks for checking in here! I think something that would be really useful is adding "ipfs.io" as one of the CDN providers, however, I'm not sure if you have support for adding more, currently it's just Cloudflare there without the ability to change.

If we did that, we would need to setup the updating/adding to be more automatic, right now it's me doing a manual git pull and adding it to IPFS, then making a PR here. The flow would be something like:

What you think?

PeterDaveHello commented 7 years ago

I'm afraid that we can't do that as we update the library and website every 5~10 mins since the 3k+ libraries update very frequently. This is all automatically, without manual review and merge.

I also wonder if it's a good idea to push ipfs when I don't understand this project enough, when we're going to provide service officially, we'll have responsible on that, especially when there is anything wrong, so, sorry, that might not be something I can do right now. Maybe I can help update the files from my side if you want, currently, the files looks dated on ipfs.

victorb commented 7 years ago

I'm afraid that we can't do that as we update the library and website every 5~10 mins since the 3k+ libraries update very frequently. This is all automatically, without manual review and merge.

Yeah, understandable, and making it all automatic is a much better way to go from the get-go so makes sense.

Something else we can do from our side is having the same interface you run on cdnjs.com but slightly modified to hook-up into our version of cdnjs, and deployed on cdnjs.ipfs.io or something like that. Would need to make sure it's always up-to-date, which will take some effort but not be super hard.

Maybe I can help update the files from my side if you want, currently, the files looks dated on ipfs.

Yeah, as I mentioned, the process is right now manual but in reality, should be fully automated. Will have some more thoughts about this at a later point.

Thanks for jumping in here and sharing your thoughts 👍

PeterDaveHello commented 7 years ago

@VictorBjelkholm thanks! Let me know if I can help update cdnjs on ipfs more up-to-update and frequently :)

magik6k commented 7 years ago

IPFS stores files much like git does, so updating it 'live' shouldn't really be a problem. This updating could be done using the ipfs files or ipfs object api quite easily.

The best way I can see it done in case of cdnjs is to have a tool that would apply updates based on which files changed in git commits.

Only thing I'm not sure about is how IPNS would react to that frequency of updates.

PeterDaveHello commented 7 years ago

Yeah we can try to integrate that in our buildScript