search

zielmicha / fc00.org

Hyperboria (CJDNS network) map
http://fc00.org
GNU General Public License v3.0
63 stars 21 forks source link

False links are shown on the map #4

Open zielmicha opened 9 years ago

zielmicha commented 9 years ago

Either a bug in collection util or someone sends incorrect links on purpose.

nsapa commented 9 years ago

This node http://h.fc00.org/#fccc:64de:a7a9:54ec:4812:b5b0:20a7:fbd8 only have one peer: http://h.fc00.org/#fc87:f729:c42b:8a74:9e05:916f:7b80:6df4

I checked what sendGraph.py send ( http://pastebin.com/mq2TgJVH ) and it's correct. Maybe you should store in the database whom added what.

AwesomePatrol commented 9 years ago

My Guess is that not only physical connections are shown/sent (from cjdns's README.md):

Routers have a responsibility to "keep in touch" with other routers that are physically close by and numerically near to their address.

My node has only 3 peers and even after I regenrated cjdroute.conf, restarted, etc. more peers appear on the map after a while.

zielmicha commented 9 years ago

It's not the case - each node typically contacts about 150 nodes. (and it worked correctly before, even when there were more peers in the routing table)

wt., 18.08.2015 o 00:25 użytkownik Aleksander Mistewicz < notifications@github.com> napisał:

My Guess is that not only physical connections are shown/sent (from cjdns's README.md):

Routers have a responsibility to "keep in touch" with other routers that are physically close by and numerically near to their address.

My node has only 3 peers and even after I regenrated cjdroute.conf, restarted, etc. more peers appear on the map http://h.fc00.org/#fc06:6aac:efc3:9032:58b6:4d68:0ca0:dd05 after a while.

— Reply to this email directly or view it on GitHub https://github.com/zielmicha/fc00.org/issues/4#issuecomment-131981733.

AwesomePatrol commented 9 years ago

It gets crazy.

Nodes: 469 Links: 4105

Try adding to database who add whom and reseting. Maybe accept graphs sent via hyperboria only?

interfect commented 9 years ago

My nodes have a bunch of fake links as well. Perhaps links should not be believed unless they come from several places.

zielmicha commented 9 years ago

I've added origin information to database. I believe that incorrent information was sent by ams.smash-net.org. Ping @weuxel

@interfect Your approach is not resistant to sybil attacks.

Weuxel commented 9 years ago

Incorrect in what way? I am using your sendGraph.py script.

interfect commented 9 years ago

Nothing that crowdsources the data can be immune to Sybil attacks, can it? Unless you add in an element like proof of work or signatures from the involved nodes.

On Wed, Sep 2, 2015 at 10:38 AM, Norman Rieß notifications@github.com wrote:

Incorrect in what way? I am using your sendGraph.py script.

— Reply to this email directly or view it on GitHub https://github.com/zielmicha/fc00.org/issues/4#issuecomment-137182549.

zielmicha commented 9 years ago

And now I got false links from other sources. This may be a bug in collection script.

Kubuxu commented 8 years ago

Confirming, h.n1.kubuxu.ovh should be linked to and only to h.vs1.kubuxu.ovh it is linked to two other.

kyrias commented 8 years ago

I've made a customized version of sendGraph.py that instead of using getLink to find the links it issues a getPeers against each node in the nodestore.This should theoretically prevent the false links, since the remote node should just reply to the getPeers call with its actual peers, but since it has to query each node instead of just getting info already known by cjdroute it's quite a lot slower.

The question is why cjdroute has false links in its tables though, or if it's some bug in the API causing it to give incorrect data by accident. It would be interesting to get a dump of the full nodestore and all of the links from a node that's submitted false links.

It would probably be a good idea to try to parallelize it somewhat though, to run multiple queries at once, but haven't done any work on that yet.

kyrias commented 8 years ago

For reference, it takes around 30 seconds on my laptop, with 130 nodes and it finds 226 edges.

kyrias commented 8 years ago

Another interesting question: Does every submission from the nodes sending false links contain those false links, or is it a once-off thing?

zielmicha commented 8 years ago

It seems that the errors are only for nodes that are far away (> 3 hops?). I don't know if every submission returns errors, but I guess most do.

zielmicha commented 8 years ago

@kyrias Can you share your modified script?

kyrias commented 8 years ago

Current version is available here, though I've only tested it with Python 3 and it requires hyperboria/python-cjdns.

Kubuxu commented 8 years ago

There are three options IMHO:

  1. Try fixing cjdns (might be not worth it as routing algs will change in close future).
  2. Use getPeers (quite slower). 3 Implement some sort of filtering algorithm so if link is reported once or twice it is not shown.

It could be done by keeping track of how frequently on average links are reported ie. if each report tells us about 5% links in the network then if link is reported in only 0.5% of reports don't show it.

It might not be easy but either way will be beneficial.

zielmicha commented 8 years ago

I've updated the sendGraph.py script based on @Kubuxu sGNG.py. fc00.org will now reject submissions from old versions of the script, we'll see if new false links will be introduced.

kyrias commented 8 years ago

Well, mine, but hmm.. Would be nice if there was a simple way to test if a certain link exists.

Edit: Or, if we save the submission IP and timestamp of every submitted link, not necessarily in the DB, it might be easier to detect false links semi-automatically, hmm...

Kubuxu commented 8 years ago

Not mine but thanks. :p

also now you do not give option to add cjdns path by default. It would be nice to have it back.

kyrias commented 8 years ago

It's useless right now because it won't work with the python library in the cjdns repo, and for a git clone of the python-cjdns repo we should just let pip handle it instead.

Kubuxu commented 8 years ago

Ok, I just didn't know that cjdns lib is on pip.

kyrias commented 8 years ago

@zielmicha Hmm, I see 7 nodes without any links now.. Are they old nodes that are still there but their old links have expired, or are they new nodes that have submitted their node without any links?

zielmicha commented 8 years ago

I've removed all old nodes. If node doesn't respond to get_peers request it may be left without any links (get_peers returns empty set).

kyrias commented 8 years ago

Hmm, true, so basically, the empty nodes are cases where node A has node B in its nodestore, possibly it has it added as a peer. But node B happens to be down, but it's still in the NodeStore for node A. So when node A tries to submit data to fc00 it will add node B to its list of known nodes, but not add any links, since when querying node B for its peers it didn't respond...

This does begs a question though, if a node submitted doesn't have a link, should we drop the node instead of inserting it into the db?

Another thought, since we now only add links that have been confirmed by a remote node, maybe the timeout values for the graph generation should be adjusted somewhat.

A node timeout of 3 hours doesn't sound too bad, maybe up it to 4-5 in case some remote node doesn't submit data more often, since I've noticed that right now the number of nodes goes down a lot occasionally.

But I feel that the link timeout of 7 days is a bit high now that all links are guaranteed to be links that a remote node says that it is peered with. Lowering it to maybe around 1 day should make it fine if the link happens to be down for a little while, but makes the map more accurate overall.

Thoughts?

zielmicha commented 8 years ago

I've changed edge timeout to 1 day and node timeout to 5 hours.