search

garbados / dat-gateway

In-memory Dat to HTTP gateway
https://dat.bovid.space/
61 stars 13 forks source link

Websocket replication #7

Closed RangerMauve closed 5 years ago

RangerMauve commented 6 years ago

This adds a websocket server which listens on locahost:3000/:daturlhere and creates a replication stream for the hyperdrive archive.

The purpose is to enable browsers to not only view dat content, but to also be able to load entire dat archives or publish their own archives (for instance updates) to the network.

The long term goal is to enable the DatArchive API in browsers.

RangerMauve commented 6 years ago

Should I have made a PR against https://github.com/pfrazee/dat-gateway instead? :P

RangerMauve commented 6 years ago

@garbados would you be interested in a collaborator for the project?

I think the current issue with travis is related to websocket-stream not working in the CI environment.

garbados commented 6 years ago

Interesting. I’ve been following the discussion but I honestly don’t know much about DatArchive or WebSockets so I’ve refrained from comment. I’ll take a closer look at this branch when I have time. How else can I help?

RangerMauve commented 6 years ago

Well, DatArchive is just a wrapper over top of dat / hyperdrive for BeakerBrowser.

Websockets are basically TCP streams created from an HTTP request, so they can be used anywhere a duplex stream is needed.

Hyperdrive uses duplex streams to do replication, so I'm using a websocket stream to replicate between the hyperdrive created by dat-gateway (via dat librarian), and another one that's created on the client-side (e.g. a browser).

With regards to help, I'm not sure what I can do to get the CI to work, and I think it'd be really important to get this merged in for people to seriously start using it for stuff like dat-fox.

coveralls commented 6 years ago

Pull Request Test Coverage Report for Build 38

Changes Missing Coverage Covered Lines Changed/Added Lines %
index.js 14 23 60.87%
<!-- Total: 14 23 60.87% -->
Totals Coverage Status
Change from base Build 35: -4.02%
Covered Lines: 86
Relevant Lines: 109
💛 - Coveralls
RangerMauve commented 6 years ago

I wasn't waiting for the dat archive to load properly, and I wasn't creating the replication stream.

Tested it with dat-archive-web and was able to create a new archive, replicate it to the gateway, and load the Dat URL from Beaker (the only peer was the gateway).

It should be good to go now.

RangerMauve commented 6 years ago

@garbados What's left to get this merged in?

garbados commented 6 years ago

@RangerMauve I haven't had time yet to review this in detail, sorry. It looks like tests are still failing but maybe I can help with that. I hope we can merge this soon :+1:

garbados commented 6 years ago

Tests fail intermittently from what appear to be low-level bugs having to do with UDP sockets:

node: ../deps/uv/src/unix/udp.c:82: uv__udp_finish_close: Assertion `handle->send_queue_count == 0' failed.
Aborted (core dumped)

I feel like I've seen this before with Dat, and IPFS for that matter, though I don't know what it means or how to debug it.

RangerMauve commented 6 years ago

Oh jeeze. I'm not sure what the UDP errors could be from since I've only really added stuff for websockets.

According to the checks, it's saying that the CI is working now, but it's the code coverage that's failing.

RangerMauve commented 5 years ago

🔥🎉🎉🎉🎉🎉💃💃