search

brave / brave-browser

Brave browser for Android, iOS, Linux, macOS, Windows.
https://brave.com
Mozilla Public License 2.0
17.85k stars 2.33k forks source link

IPFS Integration - Roadmap and discussion #819

Closed bsclifton closed 4 years ago

bsclifton commented 6 years ago

Carried over from https://github.com/brave/browser-laptop/issues/9556

@diasdavid wrote:

Hi there brave team! I'm David, from the IPFS project.

I'm looking into the IPFS integration into the Brave browser and I'm hoping you could clarify some design details, just so that I'm sure I understand how everything is pieced together.

Right now, the most straightforward way is to mimic what WebTorrent did and create an extension. I've noticed that it gets its own WebPage, located at https://github.com/brave/browser-laptop/tree/master/js/webtorrent and that this page gets bundled in with WebPack -- https://github.com/brave/browser-laptop/blob/master/webpack.config.js#L153-L169 --. I can create something similar, no problem.

  • Where do I capture ipfs://, dweb:// and even http(s)://ipfs.io/ipfs/ urls so that they get resolved through the IPFS Extension?
  • How do I create a control panel like the one I see Dashlane/1Password image
  • Currently, we have full web applications that are hosted and served through IPFS. We can start by just adding IPFS support for download of files but ideally, we would like to see pages be rendered. Any advice on how to hint Brave the content type? Drop it into the dom like it's 🔥?

Thanks in advance! Super excited to do this :)

drbh commented 6 years ago

What are the next steps to shipping Brave with IPFS?

My understanding is that Brave is going to use a custom build of Chromium because it does not support custom protocols without the web+ prefix.

Quoted from arewedistributedyet/issues/23

Brave will use custom build of Chromium: they could override whitelisting rules as a part of the build, but it makes more sense to try to solve it upstream in Chromium first.

However he road block is upstream; getting Chromium to support DWeb protocols - which seemingly has been officially started on Aug 17 2018

Quoted by asanka@chromium.org from /chromium/issues/detail?id=651311

I'll start an intent-to-implement.

Assuming the protocols get supported and are included in a near future build of Chromium. What will the general integration process look like?

Will the navigator.registerProtocolHandle still be used or will this not be needed anymore?

If it is used, is the sample code below seem close to the general concept? Redirecting ipfs:// traffic to a ipfs gateway?

navigator.registerProtocolHandler("ipfs",
                                  "https://gateway.ipfs.io/ipfs/%s",
                                  "IPFS");
lidel commented 6 years ago

On HTTP-based Protocol Handler

When requirement for web+ prefix is removed, we can add code to register redirect-based handler on first use.

The problem with having navigator.registerProtocolHandler without webextension support via something like manifest.json/protocol_handlers is that it comes with severe UX limitations described in https://github.com/arewedistributedyet/arewedistributedyet/issues/23#issuecomment-412892777 (previous comments are also a good read).

Still.. better than nothing :)

On Native Protocol Handler

As noted in https://github.com/brave/browser-laptop/issues/9556#issuecomment-376943891 API at navigator.registerProtocolHandler is just a redirect-based kludge that makes URLs work and open, but in the longer term, we'll need a real, programmable protocol handler.

An experimental, native protocol handler API is being designed as WebExtension API at mozilla/libdweb/#protocol-api and ipfs:// is WIP at https://github.com/ipfs-shipyard/ipfs-companion/pull/533. My hope is that when libdweb experiment matures to the point it is safe to include it in upstream Firefox at some point, we could ask Chromium to implement it as a part of WebExtension APIs as well. We could also think about at implementing it in Brave without waiting for Chromium, of course.

It is in a relatively a far future, and we could plan for delivering integration without native handler (see below).

Integration Milestones

Assuming the protocols get supported and are included in a near future build of Chromium. What will the general integration process look like?

Short Term: Choosing HTTP backend for IPFS Companion

Without native protocol handler API we can't fetch data from IPFS directly and need to use HTTP gateway so URLs still work and can be opened by HTTP-based handler registered with navigator.registerProtocolHandler.

HTTP Gateway can be provided by either locally running go-ipfs or js-ipfs running in Node (js-ipfs running in browser context can't open TCP port to serve HTTP requests). We could also just use public HTTP Gateway (go-ipfs) at https://ipfs.io/ipfs/* but that is a centralized fallback without doing actual p2p.

In the short term I would go with go-ipfs due to reasons listed in https://github.com/ipfs-shipyard/ipfs-companion/issues/312#issuecomment-385066360.

I see three ways to deliver integration:

  1. The go-ipfs is shipped and orchestrated by Brave: would be something similar to Tor Tabs (very good UX, IPFS works out of the box)
  2. Ask user to install ipfs-desktop on first use of IPFS addresses (bad UX, surface for errors).
  3. Use public HTTP gateway for reads coupled with embedded js-ipfs for writes (+ user has an option to install go-ipfs manually and switch to local node in preferences)

If (1) is not possible then go with (2) to minimize maintenance burden and user confusion.

I would love to see (1) – what are the next steps to move in that direction?

Long Term: Adding Missing APIs for Native Handler, Discovery and Transport

To get all benefits of IPFS (and other p2p protocols) we need at least:

drbh commented 6 years ago

The go-ipfs is shipped and orchestrated by Brave: would be something similar to Tor Tabs (very good UX, IPFS works out of the box)

I really like shipping go-ipfs with Brave! 👍

Not only is it great UX, but it add a competitive advantage to Brave from a product growth perspective. Similar to how Tor Tabs incentives users to move from the commercial browsers to Brave (because of its added feature set)

How do we go about adding a program the way Tor Tabs did.

Looking through the project in brave-laptop I find

browser-laptop/blob/master/app/tor.js

/**
 * Tor daemon management.
 *
 * This doesn't actually manage the tor daemon: the parts that did are
 * commented out.  Rather it just watches the tor daemon's control
 * port file for activity and connects to its control socket.
 *
 * @module tor
 */

and browser-laptop/blob/master/tools/package_tor.js

which seems to pull the binaries from S3 (interesting)

const torS3Prefix = 'https://s3.us-east-2.amazonaws.com/demo-tor-binaries/'
var torPath = process.argv.slice(2)[0] // npm run package-tor torPath
if (torPath === undefined) {
  torPath = path.join('app', 'extensions', 'bin')
}

Does this suggest we need to host the IPFS go binaries somewhere and these are pulled in and installed with Brave?

Is there someone who worked on TorTabs and has insight to the process of shipping a binary with Brave?

bsclifton commented 6 years ago

cc: @riastradh-brave @darkdh

drbh commented 6 years ago

@bsclifton thanks for suggesting help from @riastradh-brave and @darkdh.

Looking at their commits pertaining to Tor build packaging and the build_tor_scripts I've pieced together these basic next steps:

i've added a repo at /drbh/ipfs_build_scripts as a starting point for the build scripts.

also i've created a fork of brave/browser-laptop and added a branch called ipfsPackageBuild at /drbh/browser-laptop with what seems like the necessary files for brave to fetch the binaries.

The signature need to be updated in package_ipfs after the binaries are correctly built, and some adjustments to ipfs version are needed.

I'm most confused about the build process for the ipfs_build_scripts, because the tor_build_scripts seem to add dependencies in a specific way like openssl. Since go-ipfs's only dependency seems to be Golang, can we build the binaries straight from a golang:1.10-stretch base image?

If anyone has insight on the build process this would be greatly appreciated. I hope my boilerplate code is helpful.

lidel commented 6 years ago

@drbh thank you, the insight is really appreciated!

Yes, our official Docker image for go-ipfs is built on top of golang:1.10-stretch, there should not be any additional moving pieces apart from Dockerfile itself.

We also have https://github.com/ipfs/distributions which is responsible for fetching and publishing pre-built artifacts at https://dist.ipfs.io/#go-ipfs website. I can reach out to people and see how much of already existing process can be reused.

Before we dive into details, I have a question about ipfs_build_scripts: is build from source mandatory? Are you building Tor for Linux with a specific goal, eg. to solve dynamic linking issues present in the original build or something like that, or is it just a security/transparency policy?

I ask because we already have pre-built go-ipfs binaries for platforms you target with Brave and we could download them from there just like it is done with Tor on win32 here. It would simplify entire setup by removing a lot of moving pieces.

drbh commented 6 years ago

@lidel your a genuis! Pointing out how Tor win32 is built was super helpful.

Before we dive into details, I have a question about ipfs_build_scripts: is build from source mandatory?

Retrospectively it seems like overkill to build from source. I was doing so only because I was following the path that the Tor binaries needed.

I understand now that they only built them from source for dynamic linking purposes - it seems that the Windows binary doesn't need the same dyn linking that the darwin and linux bins do

I ask because we already have pre-built go-ipfs binaries for platforms you target with Brave

Based on this we have two paths forward - including some sample code to get close to the intended goals

  1. Directly downloading the ipfs precompiled binary - unzipping and storing. Edits would take place in /tools/package_ipfs.js 
    
const request = require('request');
const os = require('os');
const fs = require('fs')

const tmpPath = ${os.tmpdir()}/ipfs-binary; var binary; var opts = { "uri":"https://dist.ipfs.io/go-ipfs/v0.4.17/go-ipfs_v0.4.17_linux-amd64.tar.gz", gzip: true }

console.log(tmpPath) request(opts, function (err, res, body) { fs.writeFile(tmpPath, body, 'binary', console.log); })


2. Creating a bash shell build env - similar to how TorTabs did, to build rename and push binaries to S3
`Edits would take place in https://github.com/drbh/ipfs_build_scripts`

!/bin/sh

set -eu rm -rf go-ipfs/

export IPFS_WIN32_VERSION="0.4.17" export BRAVE_TOR_VERSION="5"

curl -fsSl "https://dist.ipfs.io/go-ipfs/v0.4.17/go-ipfs_v${IPFS_WIN32_VERSION}_windows-amd64.zip" -o ipfs-win32-$IPFS_WIN32_VERSION.zip

unzip ipfs-win32-$IPFS_WIN32_VERSION.zip

cp go-ipfs/ipfs.exe . mv ipfs.exe ipfs-$IPFS_WIN32_VERSION-darwin-brave-$BRAVE_TOR_VERSION.exe



It seems like the first way of fetching the binaries within Brave is the easiest path forward. However it is noted that Tor binaries have been compiled and pushed to a S3 location. Maybe there is value in this data flow, however it seems unneeded for IPFS because it doesn't depend on dynamic linking.

I can take on the `/tools/package_ipfs.js` file and downloading IPFS directly if this seems like the best way forward
lidel commented 6 years ago

@drbh To keep momentum going let's go with direct download for now (1) and see if that works well enough. If it turns out we need custom build for some platforms or Brave decides they want to host files at S3 to control user experience end-to-end, we can always add step (2) at a later stage.

FYI dist.ipfs.io has a valid DNSLink, which means fetching from https://ipfs.io/ipns/dist.ipfs.io/go-ipfs/v0.4.17/ (or any other public gateway) will also work.

If we have that, next step would be making sure ipfs daemon is initialized and configured for Brave. Some things from the top of my head:

darkdh commented 6 years ago

for packaging binary, I suggested taking a look at https://github.com/brave/brave-core/pull/316, we use different method in brave-browser.

drbh commented 6 years ago

@darkdh thanks for the suggestion. I am looking at /brave-core/pull/316 now and I don't fully understand where the binary is pulled and packaged.

Looking in /brave-core/tree/master/browser/extensions I see a few files that reference Tor but they seem to only reference a Tor client update.

Questions

  1. Where is TorTab's Tor packaged?
  2. What is the difference of packaging in Brave Browser vs Brave Core?
  3. Are parts of Brave Browser's Tor client related to the Brave Core Tor client?
  4. Is packaging IPFS in Brave Browser less reliable in some way then Brave Core?

Any clarification is greatly appreciated! Thanks

drbh commented 6 years ago

Updates!

In addition - the biniary is pulled via dist.ipfs.io

FYI dist.ipfs.io has a valid DNSLink, which means fetching from https://ipfs.io/ipns/dist.ipfs.io/go-ipfs/v0.4.17/ (or any other public gateway) will also work.

Issues

ipfs daemon starts when the ipfs.js script is loaded into a node command line.

Error

iipfs-brave-fail

Above code that fails:

mystart () {
    console.log('My Start')

    server.start((err) => {
      if (err) { throw err }
      console.log('Server Start')
      console.log(f)

// FAILS HERE - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
      f.spawn(options, (err, ipfsd) => {
        if (err) { throw err }
        console.log('Deamon Start')
        // this._process = ipfsd

        console.log('http://' + ipfsd.api.apiHost + ':' + ipfsd.api.apiPort + '/webui')

        ipfsd.api.id(function (err, id) {
          if (err) { throw err }

          // console.log(id.api.apiPort)
          // ipfsd.stop(server.stop)
        })
      })
    })

    console.log('Server Has Been Started')
  }

I don't exactly understand why the build fails when using npm start the install and packaging process succeeds with npm install

IPFS packaging and barebones daemon management - ipfsPackageBuild branch

lidel commented 6 years ago

@drbh apologies for not getting to it earlier, slipped thru my notifications.

The main use for ipfsd-ctl is test suite of IPFS itself, so it may be rough around edges specific to our use case.

Are you still blocked by the above error? I see you work on a fork of browser-laptop, which is the old Muon-based codebase. Shouldn't we aim for brave-browser/brave-core instead? (following these steps)

Update: I think a safe path would be to create a PR to brave-core-crx-packager that creates CRX bundle with IPFS binary similar to one already present for Tor, and then get some feedback on what are the next steps for the Chromium version.

drbh commented 6 years ago

Just submitted a PR https://github.com/brave/brave-core-crx-packager/pull/21 following the Tor CRX bundling

da2x commented 6 years ago

Will creators be able to earn Brave Rewards for content distributed over IPFS?

Creators could claim their IPNS by publishing the brave-payments-verification.txt under their IPNS. Brave could verify this using a public HTTP gateway such as https://cloudflare-ipfs.com/ipns/<id>/.well-known/brave-payments-verification.txt.

The Brave client would have to recognize and attribute time spent on http://127.0.0.1:8080/ipns/<id>/*, https://*/ipns/<id>/* (public gateways), ipns://<id>/*, and dweb://ipns/*<id> to a normalized address like ipns://<id> and handle this in the rewards system.

mathiasrw commented 6 years ago

@da2x This is such a good idea. Lets implement IPFS first and then integrate it into the attention economy!

lidel commented 5 years ago

Notes on bringing IPFS to Brave in 2019

I had a chat with @bbondy about ways we can deliver better IPFS experience in Brave, given where both projects are right now.

Below is a short (well, I tried..) summary of takeaways and low hanging fruits we were able to identify to speed up IPFS integration.

Today: IPFS works in Brave, but UX could be much better

IPFS Companion extension can be installed in Brave from Chrome Store, with some caveats:

Future: how it could work

We want to ship native IPFS support with Brave in 2019. What we mean by that in UX terms is:

MVP:

Ideal:

How to get there?

In 2019 there are two ways to do enable it, each comes with own set of challenges:

(A) Bundling go-ipfs binary with Brave and talking to it via ipfs-companion

(B) Embedded js-ipfs running in ipfs-companion (WebExtension context)

Which one is more feasible?

While (A) was the way we planned to integrate in 2018, (B) is self-contained (execution context is limited to ipfs-companion extension), which may be easier to manage, audit and ship today, as IPFS project has a lot of relevant plumbing in place in js-ipfs and ipfs-companion thanks to mozilla/libdweb and other work that happened in recent months.

Shipping Brave-powered js-ipfs node in ipfs-companion being blessed_extension (B) may simply take less work.

Sidenote: both (A) and (B) can benefit from native Protocol Handler API:

What are the next steps?

IPFS roadmap for bringing IPFS to Brave in 2019 will include (P0 - priority):

Note: "Bundling go-ipfs with Brave" is still a possibility, and work done on extension responsible for binary updates may be resumed and used in the future, but we want to see if we can expose HTTP Gateway over chrome.sockets.tcpServer. If that is possible, we may be able to ship sooner than later.

If js-ipfs approach takes too much time, or (P0/MVP) does not work as expected, we can always focus back on bundling go-ipfs.

da2x commented 5 years ago

@lidel js-ipfs doesn’t support IPNS or DNSLink. You’d have to implement it in the Companion, which we discussed briefly in relation to the TTL and DNS caching behavior. Supported in go-ipfs. Speaking of DNSLink, does trying to access a transparently HTTP gatewayed resource over an DNSLink mapped domain count as “the first time user tries to access IPFS resource”? Detecting this is non-trivial without either doing DNSLink requests for every domain; or bundling a list of IPFS enable domains and gateways (like the HSTS preload list).

lidel commented 5 years ago

@da2x js-ipfs is actually making good progress on those fronts:

IPNS over pubsub and DHT will be enabled by default in js-ipfs 0.35, but we can experiment with it today as it is opt-in since v0.34.0.

DNSLink lookup already works. In web browser context js-ipfs is delegating lookups to our public gateway (runtime/dns-browser.js), and in the future this can be improved by using a dedicated WebExtension API (like this one), falling back to one of public DNS-over-HTTPS providers or implementing Node.js dns API using chrome.sockets.*.

Does trying to access a transparently HTTP gatewayed resource over an DNSLink mapped domain count as “the first time user tries to access IPFS resource”?

I think good candidates for "detecting IPFS without ipfs-companion" (very light on resources):

See also Detect IPFS hints and display extension install prompt (#3045)

MidnightLightning commented 5 years ago

Thank you for making strides in this direction! One aspect of a deeper integration between Brave and IPFS is more intuitive viewing/loading of IPFS-hosted files, based on MIME type. In https://github.com/ipfs-shipyard/ipfs-webui/issues/920 I brought up the fact that the way that the embedded video player in Chrome doesn't work ideally with how IPFS loads data at a low-level; even if a Brave user doesn't have an IPFS node running of their own, I think there's an opportunity there for how the Brave browser itself loads and views videos and pictures if it detects it's being loaded via IPFS could be optimized.

MicahZoltu commented 5 years ago

While full IPFS integration would certainly be nice, we can gain a significant portion of the benefits of IPFS by implementing the much simpler process of validating IPFS hashes. The browser can be pre-configured with known IPFS gateways (including localhost) and have a configuration page for adding more. When a user browses a page that matches a configured IPFS regex (e.g., https?://cloudflare-ipfs.com/ipfs/([^/])+) it would validate the hash for the fetched file matches the hash in the URL. When it doesn't match, a security pop-up can be presented to the user just like if the SSL key didn't validate.

While this doesn't give us the guaranteed availability that IPFS does, it does give us the security of knowing that the data we are receiving is the expected data, thus allowing people to trustlessly utilize IPFS gateways (at the moment, IPFS gateways serve almost no purpose since you still have to trust them, and they can censor content).

For me, and I think many others, this is a big win from IPFS and I believe the amount of work required to achieve this is much smaller than the amount of work required to do full P2P IPFS fetching of content.

lidel commented 5 years ago

@MicahZoltu fysa we are tracking Verifiable HTTP Gateway Responses in https://github.com/ipfs/in-web-browsers/issues/128 :+1:

nileshtrivedi commented 5 years ago

What's the current state of native IPFS integration in Brave?

lidel commented 5 years ago

We are exploring Option (B) from https://github.com/brave/brave-browser/issues/819#issuecomment-456039555:

kirkins commented 5 years ago

Any updates on IPFS?

lidel commented 4 years ago

Yes!

One-click-install

Stable release of IPFS Companion can be enabled directly via Brave Settings (chrome://settings/extensions):

2019-09-20--14-43-17

By default, it works like regular ipfs-companion and External node is used by default, assuming user has go-ipfs or IPFS Desktop installed and running.

Experimental embedded JS IPFS node

Brave users (oh the pun..) can opt-in and try experimental Embedded + chrome.sockets (see /docs/node-types for more details).

Note: embedded js-ipfs will remain experimental opt-in until performance/ux gets closer to one with External node (js-ipfs is missing native DHT, uses delegated routing for now).

2019-09-20--14-44-37

2019-09-20--14-45-06

Where to track work on Companion and Brave integrations?

Meta issue on embedded js-ipfs is at https://github.com/ipfs-shipyard/ipfs-companion/issues/716, we also track v1 and v2 stages of IPFS integration in ipfs-companion/projects/4, and of course feature/ipfs label in this repo.

There also a broader IPFS Browser Update published on blog.ipfs.io. Expect Brave-specific one in following weeks.

bbondy commented 4 years ago

Duping to this one, we can create other follow-ups from that too which are more specific and smaller than this issue. Thanks everyone for the discussions above. https://github.com/brave/brave-browser/issues/10220