Initial discussion

[vesparny]

Open Web Traffic

This project is inspired by an awesome @staltz's article

Goal

Create an open and fair google analytics alternative.

How

My idea is to use hypercore to store events in a log, that is also shareable using feed.discoveryKey with other peers.

I imagine an event to be something like:

{
  type: 'pageVisit',
  when: '2017-04-26T08:43:00.088Z',
  path: '/open-analytics.html'
}

This event will be stored in the feed in its raw format.

Then

The hypercore's feed is just a raw data container. We'd probably want to feed a database with incoming events in order to visualize them properly in a nice dashboard.

function processNewEntry (entry) {
  // store the entry in a database
}

feed.on('append', () => {
  feed.get(feed.length, processNewEntry)
})

Open points

This is an open discussion, and there are a lot of open points:

• how do we store information like unique page views without abusing user's privacy?
• same for average visit duration
• where do we store events for making them available to a dashboard? I was thinking about PouchDb. Perhaps better alternatives exist?
• If a website gets a fair amount of traffic, the hypercore's log might get quite big soon. I am wondering if storing events there is a viable approach.

Thanks to everyone willing to give their input/opinion/help here 😀

[staltz]

Hi! Great initiative. Trying to understand correctly, with Hypercore, who would store the data? Would random hypercore peers store it? Is it opt-in or opt-out?

[vesparny]

The Idea is to log every event in a local hypercore instance, and make the feed.discoveryKey available to the public. The feed will be replicated to any other peer.

This way anyone will be able to downlaod a local version of the raw data by doing something like

const hyperdrive = require('hypercore')
const swarm = require('hyperdiscovery')

KEY = '01842485b8b40ca9de3995fe4e51d48c4b895e5903a9c37102aa11a40e7d88d5' 
const feed = hyperdrive('./replicated.dataset', KEY, { live: true, valueEncoding: 'json' })
swarm(feed)

feed
  .createReadStream({
    tail: true,
    live: true
  })
  .on('data', console.log)

[vesparny]

On the other hand, while processing incoming request, we can do 2 things:

• log data to hypercore, exposing it to any connected peer as well
• create a readStream from the feed, and use it to process the raw data and do anything we want (the idea is to store raw data in a database in order to be able to display charts)

function processNewEntry (event) {
// store the event somewhere.
// I was thinking about PouchDB, but it can be anything
}

feed
  .createReadStream({
    tail: true,
    live: true
  })
  .on('data', processNewEntry)

[vesparny]

We can even create a realtime dashboard serving raw data through WebSockets or sse .

[vesparny]

Hi @staltz, I've put some effort in the project in the past days and released the first version. Now the readme should be clearer.

I'd really love to have a feedback from you.

[NetOpWibby]

This is pretty cool! I currently use Gauges for analytics and I think it's neat to see on a world map where my visitors are coming from. I'll probably fork fair-analytics when I'm ready to delve into it but I really like what you have thus far (I'm super anti-Google).

[vesparny]

@NetOperatorWibby It's not about being anti-google. To me it's all about respecting users :)

[NetOpWibby]

I'm aware of that, but it's because you are respecting users, you're doing the exact opposite of what Google does.