Trying to deduplicate

[YerkoPalma]

Hi 👋 I'm trying to deduplicate connections on this demo but I can't seem to make it right.

I did this before in a minimal hyperswarm setup like this

  const peerId = Buffer.alloc(crypto_generichash_BYTES)
  randombytes_buf(peerId)
  // dedup
  socket.write(peerId.toString('hex'))
  socket.once('data', (data) => {
    info.deduplicate(peerId, data)
  })

But when I tried something similar with this demo I've got an error

* Connecting to channel global
events.js:298
      throw er; // Unhandled 'error' event
      ^

Error: Could not parse row ebe785283234ca52aef5044fd1981f8ff792d94e8dc97d1655...
    at DestroyableTransform.parseRow [as mapper] (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\ndjson\index.js:19:28)
    at DestroyableTransform.transform [as _transform] (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\split2\index.js:33:21)
    at DestroyableTransform.Transform._read (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\through2\node_modules\readable-stream\lib\_stream_transform.js:184:10)
    at DestroyableTransform.Transform._write (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\through2\node_modules\readable-stream\lib\_stream_transform.js:172:83)
    at doWrite (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\through2\node_modules\readable-stream\lib\_stream_writable.js:428:64)
    at writeOrBuffer (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\through2\node_modules\readable-stream\lib\_stream_writable.js:417:5)
    at DestroyableTransform.Writable.write (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\through2\node_modules\readable-stream\lib\_stream_writable.js:334:11)
    at Socket.ondata (_stream_readable.js:708:22)
    at Socket.emit (events.js:333:22)
    at addChunk (_stream_readable.js:297:12)
Emitted 'error' event on DestroyableTransform instance at:
    at errorOrDestroy (internal/streams/destroy.js:128:12)
    at DestroyableTransform.onerror (_stream_readable.js:745:9)
    at DestroyableTransform.emit (events.js:321:20)
    at DestroyableTransform.parseRow [as mapper] (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\ndjson\index.js:19:14)
    at DestroyableTransform.transform [as _transform] (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\split2\index.js:33:21)
    [... lines matching original stack trace ...]
    at DestroyableTransform.Writable.write (C:\CLC\VIDA\Web\Comun\ypalma\hypercore-playground\node_modules\through2\node_modules\readable-stream\lib\_stream_writable.js:334:11)

I'm doing it wrong or something? Would highly appreciate some help Thanks :)

[RangerMauve]

I think the problem there is that you're writing a raw hex string to the stream, but it's expecting everything to be in ndjson format and throwing a parsing error. Might want to use the Peer API for sending the hex key.

Also, hyperswarm-web sadly doesn't support info.deduplicate yet. 😅 A PR to hyperswarm-proxy would be very much appreciated, and in the meantime you could probably track connection streams and the peer IDs and deduplicate within your app.

[YerkoPalma]

I'm not using hyperswarm-web (yet), so, for now using the send method from Peer worked. I don't get the hyper stack so well right now, that's why I'm just playing around, if I can figure out how to implement dedup in hyperswarm-web, I've definetly will be sending a PR.

thanks