Closed mcollina closed 1 year ago
We want to go with 1)
https://github.com/libp2p/js-libp2p-crypto/issues/130 -- https://github.com/nodejs/node/issues/15116, it does require to default to min Node.js 10 though, which is fine as we already agreed to that -- https://github.com/ipfs/community/pull/350.
For the sake of continuing with the rest of Benchmarks, you can use a pre-generated keypair by passing it --privateKey
on init.
https://github.com/ipfs/js-ipfs#optionsinit
» jsipfs init --help
jsipfs init [config] [options]
Initialize a local IPFS node
Positionals:
config Node config, this should JSON and will be merged with the default
config. Check https://github.com/ipfs/js-ipfs#optionsconfig [string]
Options:
--version Show version number [boolean]
--silent Write no output [boolean] [default: false]
--pass Pass phrase for the keys [string] [default: ""]
--help Show help [boolean]
--bits, -b Number of bits to use in the generated RSA private key
(defaults to 2048) [number] [default: "2048"]
--emptyRepo, -e Don't add and pin help files to the local storage [boolean]
--privateKey, -k Pre-generated private key to use for the repo [string]
As a datapoint for comparison, go-ipfs takes 0.17s
» time ipfs init
initializing IPFS node at /Users/imp/.ipfs
generating 2048-bit RSA keypair...done
peer identity: QmQSqNaY5KK8ygzSfvptqbvVqswQ9rmC6ShVKw7Hk3cW5L
to get started, enter:
ipfs cat /ipfs/QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme
ipfs init 0.17s user 0.05s system 85% cpu 0.261 total
@daviddias @alanshaw do you want us to do that change (put it in our queue of things to do), or do you plan to make that change in the js-ipfs team?
Note that the problem is not going to fix it for the browser. Is that ok?
As a datapoint: go-ipfs takes 0.17s
Currently js-ipfs is at 10 seconds because of that generation. Creating the actual repo and adding a file takes ~100 ms (after the certificate is created) which is already pretty good. I hope we can get it into the same ballpark of the Go implementation.
Startup speed of Go is superior to the one of Node.js, so it would be extremely hard to reach the same speed.
do you want us to do that change (put it in our queue of things to do)
@vasco-santos @jacobheun would you be able to ship https://github.com/libp2p/js-libp2p-crypto/issues/130 soon or would you prefer to let @mcollina handle it?
Either way, for the browsers, could you just add a JSON object with a bunch of keys before starting the profiling? Or alternatively, just load the key from a local http endpoint through a single get request, taking out all the keypair calls from the profiling graph and cutting the time spent on generating it.
Either way, for the browsers, could you just add a JSON object with a bunch of keys before starting the profiling? Or alternatively, just load the key from a local http endpoint through a single get request, taking out all the keypair calls from the profiling graph and cutting the time spent on generating it.
I'm more concerned about fixing the issue for browsers rather than the benchmarking itself.
I would like to focus on the testbed and libp2p
interop as soon as possible.
If @mcollina can handle it, it would be great. Otherwise, I can also shift priorities to tackle it.
@mcollina go ahead and have fun trying the internals of libp2p :D
Just wanted to shim in saying that node-forge
key generation is fast in both browser and node environments. What they do is that they use the node api if available, then use the webcrypto (subtle) api if available and fallback to a JS implementation.
We can do the same as node-forge
and fallback to keypair
for the JS implemrntation. See https://github.com/digitalbazaar/forge/blob/5478021a7bdbbbc3b2ea981f28a0487b204cea10/lib/rsa.js#L896
Is there any update on this? We're building a chat app on OrbitDB, which creates a lot of files, and this is showing up as the big resource hog in our profiling too.
We're using this in node, not the browser, so the node-forge
solution mentioned above sounds good.
@holmesworcester key generation only happens when you init a new node, not when adding files to ipfs etc.
Can you open a new issue with the results of your profiling?
js-ipfs is being deprecated in favor of Helia. You can https://github.com/ipfs/js-ipfs/issues/4336 and read the migration guide.
Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see https://github.com/ipfs/js-ipfs/issues/4336).
This issue is most likely resolved in Helia, please try it out!
As part of the benchmarking initiative, we have identified a first bottleneck in js-ipfs. Adding a file to a local empty repo creates a new public/private key (see https://github.com/ipfs/benchmarks/blob/master/tests/local-add.js). This operation takes 3-6 seconds on our laptop, making the user experience on new repositories problematic. Also, it skews all the possible benchmarks, because it is hard to gain 3-6 seconds lost in this activity. During our call earlier this week, we talked about using a pre-generated certificate for benchmarking/improving the actual data transfer. However, this does not solve the problem for creating a new repository and new users.
We have conducted our analysis using node clinic. Here is a doctor visualization of that benchmark:
As it's evident, the event loop is blocking for a severe amount of time. We generated a flamegraph of the same activity:
As you can see, the vast majority of time is spent in the https://www.npmjs.com/package/keypair module. We can see two recommendations:
Note that approach 1 only works for Node.js, while 2 is essentially needed for browser environments.