Closed mrousavy closed 1 year ago
For example, for atob
and btoa
this JSI library by @craftzdog is 4x faster than base64-js: https://github.com/craftzdog/react-native-quick-base64
I'd love to make the pbkdf2 function asynchronous (run the algorithm on a separate Thread, return a Promise) to not block the entire JS interaction (pressing buttons, updating loading percentage, etc.)
nvm it only takes 1ms if written in C++, no need to make it async.
I did it.
@zemse @mirceanis @ricmoo how do I swap the pbkdf2 function?
The one from this library takes 14 seconds:
ether's pbkdf2 took 14279.91887497902ms
and my pure C++ JSI implementation takes 0.001 second:
C++ pbkdf2 took 1.4742500185966492ms
..so it's almost 10.000x faster.
I'd love to make that open source if there's a way to swap the implementations.
So I patched @ethersproject/pbkdf2/lib/browser-pbkdf2.js
to use my custom C++ implementation instead of the JS one, and here's the results:
💰 Importing wallet from phrase: ....
💰 Successfully imported wallet in 45114.42308330536ms!
💰 Importing wallet from phrase: ....
💰 Successfully imported wallet in 1120.2051666378975ms!
...so I brought it down from 45 seconds to 1 second, a 45x performance improvement.
I still wonder why it takes a full second to import the wallet from the passphrase, I am even using the native C++ atob
, btoa
and Buffer
implementations, will debug further - but at least now the app is useable.
Can you try using resolutions in package.json to override the pbkdf2 subpackage? (I'm currently afk otherwise would include an example)
I'll try to play with resolutions too, right now I have used patch-package
to locally patch it - quick and dirty but it works.
Since getting the wallet still takes more than a second on a relatively new iPhone, I'm not done with optimizing yet. Unfortunately I'm having trouble getting the flamegraph profiler working in React Native, do you maybe know off the top of your head which other functions are causing those performance issues that are worth looking into? I can also implement those in C++
Just as an FYI, I've been adding an option to v6 to register custom implementations for certain crypto methods, so this should be easier to swap out in v6.
Do you have an ETA for v6?
@ricmoo this is interesting for us too. actually we have been tracking this using another issue - https://github.com/ethers-io/ethers.js/issues/1503
im betting both will get fixed by the new swapouts.
I'm working on v6 as fast as I can and hope to get a beta out before the end of January. There are a few priority changes needed for another minor bump though, which might bump that a bit.
@mrousavy this patch seems to be the end to my quest.But still unable to get it working throwing "undefined" while using it.Could you shed some light on the usage
There is something that we can do to help you release v6?
@mrousavy this patch seems to be the end to my quest.But still unable to get it working throwing "undefined" while using it.Could you shed some light on the usage
I know nothing about jsi but i think you still need to have the module itself installed in the global object somehow. @mrousavy can you please share the pbkdf2 c++ module too so we can try it out?(if i missed it or i misunderstood something please correct me)
@mrousavy hmm, patch seem don't used, global always "undefined", peformace when run createRandom() on andoird very bad
Well obviously you still need the native function, otherwise the patch won't work. That's the whole idea, moving it to native. 😅
The code is closed source for a client of mine (at Margelo), but we will release something that speeds up this operation (and all other crypto operations) by magnitudes soon.
@ricmoo I think I mentioned this before, but we're essentially bringing crypto
to React Native with C++ implementations instead of JS browser-polyfills, so this will be a huge performance improvement 🔥
I'll DM you on Twitter once we are ready to test the integration with ethers
@ricmoo When will V6 be released? My application is about to go into production, so I may have to switch to web3.js
There is a v6 beta available now, but I haven’t got any meaningful documentation together yet.
If the new Wallet(utils.randomBytes(32))
method doesn’t work for you (which should be instant, even on react-native), you could probably trivially just implement HDNode using native libraries, or there are probably already existing libraries for that part designed for RN.
But if you go the Web3.js route, I’d be curious how easy it is to switch between the two.
There is a v6 beta available now, but I haven’t got any meaningful documentation together yet.
If the
new Wallet(utils.randomBytes(32))
method doesn’t work for you (which should be instant, even on react-native), you could probably trivially just implement HDNode using native libraries, or there are probably already existing libraries for that part designed for RN.But if you go the Web3.js route, I’d be curious how easy it is to switch between the two.
new Wallet(utils.randomBytes(32))
does not generate mnemonic words.web3.js integration with react-native is more difficult and has no documentation. If you can tell us how to use it in V6 here.Thanks.
Same here. I would prefer to use v6 beta and put my app in production. Rather than switch between libraries.
Anyway u could give us a complete example on how to use v6 ?
On Thu, 14 Apr, 2022, 17:35 linxianxi, @.***> wrote:
There is a v6 beta available now, but I haven’t got any meaningful documentation together yet.
If the new Wallet(utils.randomBytes(32)) method doesn’t work for you (which should be instant, even on react-native), you could probably trivially just implement HDNode using native libraries, or there are probably already existing libraries for that part designed for RN.
But if you go the Web3.js route, I’d be curious how easy it is to switch between the two.
new Wallet(utils.randomBytes(32)) does not generate mnemonic words.web3.js integration with react-native is more difficult and has no documentation. If you can tell us how to use it in V6 here.Thanks.
— Reply to this email directly, view it on GitHub https://github.com/ethers-io/ethers.js/issues/2250#issuecomment-1099046177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAASYUZR5PUHRK34JGCTYJDVE7YIVANCNFSM5HA54GVA . You are receiving this because you commented.Message ID: <ethers-io/ethers .@.***>
Anyway u could give us a complete example on how to use v6 ?
Until the official docs are up here's an example to use custom pbkdf2 implementation.
IMP: This code example needs a fast pbkdf2 implementation in RN. Just copy pasting this below snippet would NOT WORK.
import { Wallet } from "ethers";
import { pbkdf2 } from "@ethersproject/crypto";
// IMP: this code WON'T WORK by simple copy paste in React Native, you need to replace the following with a
// working implementation of pbkdf2 in RN.
import { pbkdf2Sync } from "crypto"; // fast implementation in your enviornment
/**
* @param {*} password Uint8Array
* @param {*} salt Uint8Array
* @param {*} iterations number
* @param {*} keylen number
* @param {*} algo "sha256" | "sha512"
* @returns BytesLike
*/
function customPbkdf2(password, salt, iterations, keylen, algo) {
console.log("custom pbkdf2");
return pbkdf2Sync(password, salt, iterations, keylen, algo);
}
// registers custom implementation for pbkdf2
pbkdf2.register(customPbkdf2);
console.log("creating wallet");
const wallet = Wallet.createRandom();
console.log(wallet.address);
I would prefer to use v6 beta and put my app in production.
Note that v6 is in very early beta as mentioned by ricmoo.
Anyway u could give us a complete example on how to use v6 ?
Until the official docs are up here's an example to use custom p
import { Wallet } from "ethers"; import { pbkdf2 } from "@ethersproject/crypto"; import { pbkdf2Sync } from "crypto"; // fast implementation in your enviornment /** * @param {*} password Uint8Array * @param {*} salt Uint8Array * @param {*} iterations number * @param {*} keylen number * @param {*} algo "sha256" | "sha512" * @returns BytesLike */ function customPbkdf2(password, salt, iterations, keylen, algo) { console.log("custom pbkdf2"); return pbkdf2Sync(password, salt, iterations, keylen, algo); } // registers custom implementation for pbkdf2 pbkdf2.register(customPbkdf2); console.log("creating wallet"); const wallet = Wallet.createRandom(); console.log(wallet.address);
I would prefer to use v6 beta and put my app in production.
Note that v6 is in very early beta as mentioned by ricmoo.
This will report an error,did you try it on RN?
Which line does it error on?
import { pbkdf2Sync } from "crypto"; // fast implementation in your enviornment
function customPbkdf2(password, salt, iterations, keylen, algo) {
console.log("custom pbkdf2");
return pbkdf2Sync(password, salt, iterations, keylen, algo);
}
You need to replace the above code with a pbkdf2 method that works nicely in RN.
Which line does it error on?
import { pbkdf2Sync } from "crypto"; // fast implementation in your enviornment function customPbkdf2(password, salt, iterations, keylen, algo) { console.log("custom pbkdf2"); return pbkdf2Sync(password, salt, iterations, keylen, algo); }
You need to replace the above code with a pbkdf2 method that works nicely in RN.
Pardon me where can i get the v6 code to try this out?
@andra961 it’s the v6-beta
branch of this repo, or npm install ethers@beta
. There is no documentation yet.
@mrousavy Do you have the pure C++ JSI implementation mentioned in the reply above available for public? The example here is for swapping the pbkdf2 ethers would use.
@mrousavy Do you have the pure C++ JSI implementation mentioned in the reply above available for public? The example here is for swapping the pbkdf2 ethers would use.
He already said it's closed source
I find solution for slow when create random wallet ` import hdkey from 'ethereumjs-wallet/dist.browser/hdkey'; import bip39 from 'bip39'; import {Contract, ethers, Wallet} from 'ethers'; const getPrivateKeyFromSeed = (mnemonic: string) => { const root = hdkey.fromMasterSeed(bip39.mnemonicToSeed(mnemonic)); const derivedNode = root.derivePath(ethers.utils.defaultPath); const privateKey = derivedNode.getWallet().getPrivateKeyString(); return privateKey; };
const createWallet = () => { let entropy: Uint8Array = randomBytes(16); const mnemonic = entropyToMnemonic(entropy); const privateKey = getPrivateKeyFromSeed(mnemonic);
return { wallet: new ethers.Wallet(privateKey), mnemonic, }; }; `
I find solution for slow when create random wallet ` import hdkey from 'ethereumjs-wallet/dist.browser/hdkey'; import bip39 from 'bip39'; import {Contract, ethers, Wallet} from 'ethers'; const getPrivateKeyFromSeed = (mnemonic: string) => { const root = hdkey.fromMasterSeed(bip39.mnemonicToSeed(mnemonic)); const derivedNode = root.derivePath(ethers.utils.defaultPath); const privateKey = derivedNode.getWallet().getPrivateKeyString(); return privateKey; };
const createWallet = () => { let entropy: Uint8Array = randomBytes(16); const mnemonic = entropyToMnemonic(entropy); const privateKey = getPrivateKeyFromSeed(mnemonic);
return { wallet: new ethers.Wallet(privateKey), mnemonic, }; }; `
How can you make bip39 work on react native?
I find solution for slow when create random wallet
import hdkey from 'ethereumjs-wallet/dist.browser/hdkey'; import bip39 from 'bip39'; import {Contract, ethers, Wallet} from 'ethers'; const getPrivateKeyFromSeed = (mnemonic: string) => { const root = hdkey.fromMasterSeed(bip39.mnemonicToSeed(mnemonic)); const derivedNode = root.derivePath(ethers.utils.defaultPath); const privateKey = derivedNode.getWallet().getPrivateKeyString(); return privateKey; }; const createWallet = () => { let entropy: Uint8Array = randomBytes(16); const mnemonic = entropyToMnemonic(entropy); const privateKey = getPrivateKeyFromSeed(mnemonic); return { wallet: new ethers.Wallet(privateKey), mnemonic, }; };
How can you make bip39 work on react native?
https://www.npmjs.com/package/bip39 this work on RN
I find solution for slow when create random wallet
import hdkey from 'ethereumjs-wallet/dist.browser/hdkey'; import bip39 from 'bip39'; import {Contract, ethers, Wallet} from 'ethers'; const getPrivateKeyFromSeed = (mnemonic: string) => { const root = hdkey.fromMasterSeed(bip39.mnemonicToSeed(mnemonic)); const derivedNode = root.derivePath(ethers.utils.defaultPath); const privateKey = derivedNode.getWallet().getPrivateKeyString(); return privateKey; }; const createWallet = () => { let entropy: Uint8Array = randomBytes(16); const mnemonic = entropyToMnemonic(entropy); const privateKey = getPrivateKeyFromSeed(mnemonic); return { wallet: new ethers.Wallet(privateKey), mnemonic, }; };
How can you make bip39 work on react native?
https://www.npmjs.com/package/bip39 this work on RN
I tried and it didn't. Pretty sure it requires shims to work. Are you using something like rn-nodeify?
It took me one day to find a solution!
npm i events assert bip39 buffer hdkey crypto-browserify stream-browserify
npm i -D @types/hdkey // if your project uses typescript
metro.config.js
module.exports = {
// add resolver extraNodeModules, like webpack alias
resolver: {
extraNodeModules: {
crypto: require.resolve('crypto-browserify'),
stream: require.resolve('stream-browserify'),
},
},
...
};
index.js
// add this line
global.Buffer = require('buffer').Buffer;
create wallet
const bip39 = require('bip39');
import HDKey from 'hdkey';
const createWallet = () => {
const entropy = ethers.utils.randomBytes(16);
const mnemonic = ethers.utils.entropyToMnemonic(entropy);
const hdkey = HDKey.fromMasterSeed(bip39.mnemonicToSeedSync(mnemonic));
const derivedNode = hdkey.derive(ethers.utils.defaultPath);
const privateKey = ethers.utils.hexlify(derivedNode.privateKey);
const { address } = new ethers.Wallet(privateKey);
return {
address,
mnemonic,
privateKey
}
}
My current solution is to directly use react-native-scrypt and react-native-aes-crypto to replace scrypt's method and pbkdf2's method. I modified a few js with patches in a less elegant way. if you need it you can download it here.
My current solution is to directly use react-native-scrypt and react-native-aes-crypto to replace scrypt's method and pbkdf2's method. I modified a few js with patches in a less elegant way. if you need it you can download it here.
this is pretty cool! is it possible for you to structure ur code as a separate library that overrides the ethers and other libraries ? like so https://www.tutorialkart.com/nodejs/override-function-of-a-node-js-module/
any project will set an override so that your library patches are taken - https://www.stefanjudis.com/today-i-learned/how-to-override-your-dependencys-dependencies/
this will make ur library very quick to adopt.
Did someone succeeded to use Marc's solution with his custom C++ ?
Yes, I did.
I'm working on releasing the package as open-source today or tomorrow, stay tuned. Follow me on Twitter for the announcement
Hey - we've just published the fully native C++ library for Crypto. 🎉🥳
As I said before, doing crypto in JS simply won't work for mobile apps (React Native) since JS is too slow in such cases. That's why libraries like BN.js or crypto-browserify are a huge bottleneck in React Native applications. Instead, we at Margelo implemented those functions fully natively in C++ to offer huge speedups (in BN.js we measured up to 300x improvements and in QuickCrypto up to 58x improvements!)
As of today, I recommend to install both react-native-quick-crypto and react-native-bignumber in your projects and use them as drop-in replacements for free app speedups!
Install:
yarn add react-native-bignumber
yarn add react-native-quick-crypto
Then add them to your babel.config.js (see the READMEs for details).
This makes ethers.js run smoothly on React Native! Would be cool if we could add that to the README or docs of ethers.js? cc @ricmoo
@mrousavy Amazing work !
We'll use it on Ko :)
@ricmoo ethers.js currently doesn't use crypto
(nor react-native-quick-crypto
) if it is available, not sure where the try/import/catch is happening (or if you guys are doing that), but just wanted to make sure that y'all are aware of this.
I had to manually patch node_modules/@ethersproject/pbkdf2/...
to use our crypto.pbkdf2Sync
function (which is implemented natively in react-native-quick-crypto) to improve the performance for wallet creation.
const start = performance.now()
const wallet = ethers.Wallet.createRandom()
const end = performance.now()
console.log(`Creating a Wallet took ${end - start} ms.`)
Before patching @ethersproject/pbkdf2
(with your JS-based implementation):
Creating a Wallet took 16862 ms
After patching @ethersproject/pbkdf2
(with our native C++/JSI react-native-quick-crypto implementation):
Creating a Wallet took 289 ms
Here's the patch I used:
@ethersproject+pbkdf2+5.6.1.patch
You can DM me on Twitter if you have any questions or want to collab on getting this integrated with ethers.js. :)
@mrousavy I couldn't get it to work with Expo.
There was no change in performance.
This implementation only works with prebuild ?
@anarkrypto I was using an expo managed workflow and apply the @ethersproject/project/pbkdf2
patch applied above. However I found that it won't work in the managed workflow, and you'll have to eject to bare workflow if you want it to work due to the native code modification that will happen under the hood. It's not enough to apply the changes in the babel.config.js
you have to patch the pbkdf2 in order to ensure that ethers works performantly when creating new wallets. Hope this helps. I can confirm that the patch does work and lowers wait times from ~6-10 seconds [depending on the device, beforehand it was 6 seconds on new iPhones like the iPhone 13 pro, and 10-12 seconds on iPhone 7s, all the way to sub to single second waits for all phone's on all builds for new wallet creation in production builds for iOS, I haven't built it yet for android but I assume that it works as well, but maybe some build.gradle
changes will have to be made.
Note: I got the error that it doesn't work in Expo Go and that's exactly the point remember this is drop in native code, you're going to have to use less convenient local building tools like the react-native cli or expo dev-client builds from now on to build the app for emulators locally, and the nice things that come with testing expo go on a real phone on the same network also goes away, hope this helps 😄
@anarkrypto send me an dm on twitter w/ a link to your expo codebase or if it's closed source invite me as a contributor to your org and I can show you what it looks like to get it working.
If you just need bip39
then https://www.npmjs.com/package/@mfellner/react-native-bip39 is a good solution.
But for my case, I also need ed25519-hd-key
, ... so upgrade to RN 0.70+
and ready to face a lot of deprecated errors is my only solution for now.
Step 1: create a patch js file with the following content:
const crypto = require("react-native-quick-crypto")
exports.pbkdf2 = crypto.pbkdf2Sync
Step 2: update the module-resolver
config in babel.config.js
:
{
alias: {
// other alias ....
'@ethersproject/pbkdf2': 'path/to/your/patch.js',
}
}
Step 3: npm start --reset-cache
You could try to disable hermes for android and it would work faster:
In my case:
const seed = await bip39.mnemonicToSeed(mnemonic);
tooks 1500ms with hermes and without hermes 336ms
This is available in v6, using the .register
functions on each of the hash, hmac, etc. functions.
For example:
ethers.keccak256.register((data: Uint8Array) => {
yourCustomLibHere.hash(data);
});
See the src.ts/crypto/
files for more details. I'll be adding more docs for RN, Expo, etc. to help too.
Thanks! :)
@mrousavy amazing work! Improvement was from 4108ms
down to 210ms
replacing @ethersproject/pbkdf2/lib/browser-pbkdf2
with your implementation of pbkdf2Sync
🥳
Thank you to everyone in this thread and particularly @ricmoo and @mrousavy for their work with these libraries.
Using the thread above I was able to get Ethers v6.6 and react-native-quick-crypto (RNQC) working with RN. Without using RNQC Ethers v6.6 was able to create a wallet in around 1300ms which was a big improvement from Ethers v5.7 (around 5500ms). However when using RNQC, Ethers V6.6 creates a new wallet in 300ms dropping to 50ms for any additional requests.
I will include my setup for anyone wishing to work with Ethers v6:
import { getRandomValues } from 'react-native-quick-crypto';
global.getRandomValues = getRandomValues;
export * from '@ethersproject/shims';
Then imported 'ethers-setup' into the App.tsx file.
import * as ethers from 'ethers';
import { pbkdf2Sync } from 'react-native-quick-crypto';
ethers.pbkdf2.register(
(
password: Uint8Array,
salt: Uint8Array,
iterations: number,
keylen: number,
algo: 'sha256' | 'sha512'
// eslint-disable-next-line max-params
) => {
return ethers.hexlify(pbkdf2Sync(password, salt, iterations, keylen, algo));
}
);
export * from 'ethers';
I now import this into my project instead of the usual import { ethers } from 'ethers';
Awesome stuff @martin-opensky - glad to see you can benefit from those speedups.
I believe there's also other functions than pbkdf2 that can be replaced and should be much faster - @ricmoo do you have any more info on that?
Yeah great point @mrousavy. I am intrigued to see whether the encrypt wallet method could be improved as it takes 30 seconds+ during testing. It seems this process relies on scrypt but I struggled to get the RNQC library to work with ethers, using the ethers.scrypt.register approach.
I did have a question @mrousavy if you could confirm the security of the native implementations? Ethers uses noble-hashes which has been audited. Can you think of any security issues that could arise by replacing these functions with native implementations you have written?
A lot of issues have been opened about React Native being slow. For example, creating a wallet takes 33 seconds on an iPhone 11 Pro - I don't even want to find out how slow that is on an older Android device.
This code:
Takes half a minute:
I'm wondering if there is any workaround to make this code faster, as it's definitely not a solution to let the user wait for 5 minutes until he can use the app.
I can create a library for the crypto polyfills that uses a pure C/C++ implementation accessed synchronously through JSI for maximum performance, but I'd need help to find out what algorithms are slow and where I can swap them.