Open aulneau opened 4 years ago
Yeah, the hard part is getting all the hyperswarm features into the browser. I've got hyperswarm-web working, mostly, but it's missing the connection deduplication code.
Try using webpack to compile your project, but replacing hyperswarm
with hyperswarm-web
. From there see if there are any methods missing in hyperswarm-web and send a PR for stubs on those methods (or actually implement the functionality if that seems doable).
@RangerMauve thanks for the quick reply!
Yeah -- actually none of the hyperswarm specific stuff is an issue wrt to using in the browser. I had aliased your -web
package and that worked great. The dependency on noise-peer
is what is causing much of the issues. https://github.com/emilbayes/noise-peer/issues/2, due to their underlying dependencies on sodium
and it not having certain encryptions ported to javascript. https://github.com/sodium-friends/sodium-universal#compatibilty. I don't have enough context or domain knowledge to know how to improve or swap out these deps with more browser compatible ones.
Ah, yes. I think this is the same issue we had in dat-sdk, try checking out the webpack config there which replaced sodium-native with a fork by Geut. That should hopefully work.
That got it further along. I had to make some modifications to @geut/discovery-swarm-webrtc/lib/utils
const toHex = buff => {
if (typeof buff === 'string') {
return buff;
}
// this is new, previous Buffer.isBuffer would fail with a `uint8array`
// being passed here via noisePeer.keygen();
if (buff && buff.buffer) {
const str = Buffer.from(buff.buffer).toString('hex');
return str;
}
throw new Error('Cannot convert the buffer to hex: ', buff);
};
I think @mafintosh or @martinheidegger recently mentioned that sodium-javascript got updated with the missing crypto functions. Maybe it'd make sense to try using that instead of the @geut fork?
I think the @geut/sodium-javascript-plus
fork is working correctly, the issue seems to be coming from within hyperswarm-web
and its dependency on @geut/discovery-swarm-webrtc
. I'll see if I can a minimal repo set up to show what's happening
See here -> https://github.com/aulneau/p2plex-next.js
Had to change this: https://github.com/aulneau/p2plex-next.js/blob/master/vendor/discovery-swarm-webrtc/lib/utils.js#L6-L9 to accept the buffer correctly. The next error which I have a screenshot above is linked to this line of hyperswarm-proxy
https://github.com/RangerMauve/hyperswarm-proxy/blob/master/messages.js#L80
I feel like it's related to buffer and not using the right one, or something along the lines of that.
Hmm, is p2plex.publicKey
a Buffer?
Hmm, is
p2plex.publicKey
a Buffer?
It is a Uint8Array
Cool, I think it's a Buffer when running in Node.js, so it'd be good to convert it to a buffer inside the key generation in p2plex. That's probably what's causing all these weird issues.
Try changing the get publicKey()
function to look like this:
get publicKey () {
return Buffer.from(this.keyPair.publicKey)
}
That moved things forward!! This seems to be an issue now:
onstatickey: (remoteStaticKey, done) => {
const publicKey = Buffer.from(remoteStaticKey);
done();
const dropped = info.deduplicate(this.publicKey, publicKey);
if (dropped) return;
I think for that we need to add some stubbed methods for the info
object coming out of webrtc.
This info object is defined here: https://github.com/RangerMauve/hyperswarm-web/blob/master/index.js#L53
I think I've stubbed out the deduplicate method in hyperswarm-proxy already: https://github.com/RangerMauve/hyperswarm-proxy/blob/master/client.js#L191
Actually, I'm not sure why that might be happening. Is there a stack trace you could post?
It could be that we need to change this.keyPair
to have buffers for secretKey
and publicKey
using something similar to the getter. Like this.keyPair.publicKey = Buffer.from(this.keyPair.publicKey)
and similar for this.keyPair.secretKey
or whatever it's called
yep, this seems to be making it work.
get keyPair() {
const _keypair = this.opts.keyPair || noisePeer.keygen();
return {
publicKey: Buffer.from(_keypair.publicKey),
secretKey: Buffer.from(_keypair.secretKey),
};
}
Running this
const plex1 = p2plex()
const plex2 = p2plex()
plex1.on('connection', (peer) => {
peer.receiveStream('example').on('data', (data) => {
console.log('Got data from', peer.publicKey, ':', data.toString('utf8'))
plex1.destroy()
plex2.destroy()
})
})
plex2.findByPublicKey(plex1.publicKey).then((peer) => {
peer.createStream('example').end('Hello World!')
})
Does not seem to cause any data console logs though. Additionally, the default wss server is down -> wss://hyperswarm.mauve.moe
Dang. 🤔 Are WebRTC connections not going through, do you think?
I'll have to play around with it more, seems like things are happening via the wss connection:
You've been so helpful, thank you.
I wonder if you'd be open to help out getting my mental model of how this could function within my app? It's a decentralized document editor built with blockstack auth and storage. I think I'll have to generate the keypairs that this uses for each user, and persist that, and then when they enter a document they would do something along the lines of peer.createStream(doc.id)
, at least that's how it's currently set up with the webrtc stuff I have now. Eventually once I understand this better, I'd love to write a provider to use p2plex for yjs.
Since the last release sodium-javascript
has crypto_aead_chacha20poly1305P
support, but crypto_secretstream_xchacha20poly1305
is still missing, which is required for secretstream-stream
, a dependency of noise-peer
.
@aulneau That sounds like a good plan TBH. 😁 p2plex will also be handy got multiplexing across documents and stuff.
By the way, would you be down to submit a PR with whatever changes you added to hyperswarm-web / p2plex that made things work for you?
I have found another issue unfortunately, this time related to the use of noise-peer
, which uses secretstream-stream
. There is a reference to exports.HEADERBYTES = sodium.crypto_secretstream_xchacha20poly1305_HEADERBYTES
which is undefined in browser usage. Do you have anymore leads related to how to switch out noise peer for something more web compat?
Best t
Could the upgrade of noise-peer have affected this? Try using the older version and see if that helps.
Could it be that upgrade is required for compatibility?
Could the upgrade of noise-peer have affected this? Try using the older version and see if that helps.
Nah, it was the next issue regardless of version bump. Seems like noise-peer has used secretstream for a long time.
I don't really know the difference between crypto_secretstream_xchacha20poly1305
and crypto_aead_chacha20poly1305P
is. Would there be a way to alias the one that is currently implemented in sodium-javascript
?
Jeeze, yeah I have no clue. 😅
I'm also down for switching the encryption layer with whatever works instead.
@aulneau worth opening an issue and pinging emilbayes or tinchoz
Jeeze, yeah I have no clue. 😅
I'm also down for switching the encryption layer with whatever works instead.
@RangerMauve would this possibly work? https://github.com/paulmillr/noble-secp256k1
@aulneau It doesn't look like that library supports streaming encryption, but otherwise it looks good enough. 🤷
It would be cool to have this working in a browser 🚀
This is such a cool project! I started playing around with implementing this in a next.js react application, but found that certain dependencies don't really work in the context of a browser. Do you have any ideas on how to replicate or replace certain libs in this package to work universally?
Thanks!