search

Tribler / tribler

Privacy enhanced BitTorrent client with P2P content discovery
https://www.tribler.org
GNU General Public License v3.0
4.74k stars 445 forks source link

Dispersy mini: Dispersy no longer stores any data in databases #2778

Closed synctext closed 5 years ago

synctext commented 7 years ago

Remove the double storage principle of Tribler. Dispersy becomes a state-less and clean overlay.

Each Dispersy community is now responsible for their own storage. We now have significant overlap between the tribler and dispersy database. Move much of this responsibility in several iterations upwards. Downwards we move the low-level connectivity stuff. The walker becomes responsible for storing all public keys, interaction records, IP addresses, etc. See: https://github.com/Tribler/dispersy/issues/61

libtribler_architecture

We have several issues to reach dispersy mini and refactor 14000 Dispersy lines. Braindump :

wc -l Tribler/dispersy/*.py
   397 Tribler/dispersy/authentication.py
   298 Tribler/dispersy/bloomfilter.py
   335 Tribler/dispersy/candidate.py
  3703 Tribler/dispersy/community.py
  1296 Tribler/dispersy/conversion.py
   424 Tribler/dispersy/crypto.py
   417 Tribler/dispersy/database.py
    99 Tribler/dispersy/destination.py
   705 Tribler/dispersy/dispersydatabase.py
  2273 Tribler/dispersy/dispersy.py
   325 Tribler/dispersy/distribution.py
   376 Tribler/dispersy/endpoint.py
    17 Tribler/dispersy/exception.py
     0 Tribler/dispersy/__init__.py
   219 Tribler/dispersy/member.py
   615 Tribler/dispersy/message.py
    46 Tribler/dispersy/meta.py
   671 Tribler/dispersy/payload.py
   234 Tribler/dispersy/requestcache.py
   112 Tribler/dispersy/resolution.py
   393 Tribler/dispersy/statistics.py
   124 Tribler/dispersy/taskmanager.py
   458 Tribler/dispersy/timeline.py
   335 Tribler/dispersy/util.py
 13872 total

Roadmap note: instead of tweaking this layer for years and having fun, we need to be pragmatic. So no 10 month refactoring job and 7 months of nasty bug hunting. Clean-up and move on to real productive work within 6 months!

Good luck!

egbertbouman commented 7 years ago

Maybe we could even look at the byte stats that go through the socket (with tunnel prefix?) instead of using notifications from the TunnelCommunity. Just an idea.

qstokkink commented 7 years ago

If we want to be pragmatic and fast, we should follow a concrete plan. So I hereby present:

Quinten's 6 month concrete action plan

I believe the best way to approach this, is starting by refactoring out the current Dispersy walker into a separate module. Then we would write the new walker interface as we would like it to be in the future. If this does not fit the old interface, the old walker would have to be adjusted to keep everything functional.

This is the first milestone which can be merged into the repository.

At this point the new walker can be created. It would consist (roughly) of the following 3 components, which can be implemented (more-or-less) concurrently:

  1. A peer management system (public key to IP address, reputation, etc. - covers candidate.py, member.py, database.py)
  2. A peer discovery system (puncture, authentication, id etc. and conversion.py)
  3. A high-level trust based mechanism implementation for delivering peers through the interface (multichain community)

Once this is done, another milestone would be able to be merged into Dispersy: dropping the old walker and using the new one. At this point backward compatibility will probably be broken (if the peer discovery system uses Protocol Buffers messages already).

After this, we should probably finish off with refactoring the requestcache.py, community.py and dispersy.py.

Based on existing work and my intuition, this would be the time allocation:

Milestone Time (Weeks) Details
1 3 Walker interface
2.a 4 Peer management
2.b 4 Peer discovery
2.c 3 Low-level multichain
3 4 Odd jobs
18 TOTAL 1 EMPLOYEE
15 TOTAL 2 EMPLOYEES
26 AVAILABLE

Feedback or alternatives are welcomed.

devos50 commented 7 years ago

Just a mental note here: I think that Dispersy communities should adhere to the separation of concerns principle where each community has it's own functionality and responsibilities. The communication between these communities should be as little as possible i.e. they should be able to run in isolation.

synctext commented 7 years ago

Dispersy-identity message: remove and always include full public key. Remove sha1 usage everywhere in Dispersy

egbertbouman commented 7 years ago

@devos50 I agree, but as far as I know communities already run separately. Are you referring to some specific instance where communities communicate with each other where they shouldn't?

@synctext I completely agree with removing the dispersy-identity/dispersy-missing-identity message stuff. However, note that this requires upgrading all communities to v2 (allchannel,channel,multichain! are using v1), so the old communities would no longer be accessible. If we are going to do this, we may as well use curve25519/ed25519 members everywhere. Then we can remove the M2Crypto dependency. Also a good time to switch to protocol buffers...

synctext commented 7 years ago

@egbertbouman Yes to V2, yes for curve25519, yes to protocol buffers. Cleanup year..

qstokkink commented 7 years ago

The first refactoring is in: https://github.com/qstokkink/py-ipv8.

qstokkink commented 7 years ago

Small update: just now the first bytes were sent to - and received from - a tracker, marking the first backwards compatible communication.

Notable changes between IPv8 and Dispersy so far are (1) the removal of the M2Crypto dependency, (2) the removal of authentication.py, conversion.py, destination.py, distribution.py and resolution.py and (3) removal of meta objects and dramatic downsizing of message.py.

We still have quite a way to go (many messages to convert to the new system) for full on backwards compatibilty though.

qstokkink commented 7 years ago

Now that I can communicate with Dispersy, I had some fun and captured a little under 10 seconds of peer discovery:

output

qstokkink commented 7 years ago

Some internal imagery for my own convenience: 20170707_172047 Green: hole punching package Blue: peer discovery package Red: pitfall Black: note

EDIT: In other news, I am on a roll: network_view The walker/peer discovery now uses a networkx graph to represent its peers (orange is a walkable candidate and green is a verified member). Conveniently this allows future scientists to (1) make fancy graph based walking decisions and (2) plot this from within IPv8 (or export it).

qstokkink commented 6 years ago

Cool update: the basic (no hidden services) TunnelCommunity is integrated and working in IPv8.

EDIT: The TrustChain and hidden services extensions are now also in. Excluding attestation logic, IPv8 is now feature complete. Ergo, on a level playing field with our current technology stack (regarding messaging, anonymization and distributed ledger tech.).

Whereas the non-hiddenservices TunnelCommunity and Dispersy functionality could be tested over the wild Internet, the TrustChain and hidden services were implemented blind. Therefore, I will now commence writing tests to make sure that these latest additions still work as advertised.

qstokkink commented 6 years ago

The first test is up and running for the TunnelCommunity 🎉 . Performance is notably better.

Doing the following:

Now takes 0.03~0.04 seconds, instead of 20 seconds. So:

500 times faster

synctext commented 6 years ago

making impressive progress.. 40msec, interesting speedup metric.

Obviously I'm anxiously awaiting if IPv8 also has an impact on Bittorrent tunnel MBps.. But cleaning, refactoring, and bugfree code is what Tribler needs first.

qstokkink commented 6 years ago

@synctext I think MBps will go up a little bit due to the cleaner back-end. It won't be as big of a boost as using multiple sockets/multiprocessing though.

Also, progress:

synctext commented 6 years ago

year 2018 toughts.. move whole Libtorrent API into a storage wrapper. becomes part of the IPv8 stack API: trust,identity, secure communication and swarm-based storage. Merely 1 bulk and 1 control socket for all DHT, miniDispersy, swarming and crypto matters. not a priority, but clean vision.

qstokkink commented 6 years ago

More progress: TrustChain now also has tests and appears to be working correctly. Now I'll move on to the boring stuff like (1) hooking up a coverage check to see if I am missing lines in my tests, (2) writing a README.md with all the depedencies, (3) writing docstrings and other assorted documentation and (4) run the tests on other platforms.

Overall, IPv8 has all of the beta 1 functionality complete:

IPv8 does not (yet) include:

At some point, all of this functionality should probably be shaped into some nice API.

synctext commented 6 years ago

congrats! now the boring part..

Do we need unverified messages in the future? (no idea what they really are..) crypto==NULL and no authentication could be avoided.

qstokkink commented 6 years ago

@synctext What I mean with verified is that the messages are correctly signed (message public key and message signature match). In the past Dispersy also added replay attack prevention, but this ate disk space and CPU time and wasn't used in the tunnel and TrustChain communities. So I left it out in IPv8 (for now).

The TunnelCommunity and the TrustChainCommunity use their own verification mechanism. So, for the more advanced communities unverified messages are also available. Furthermore, the messages are smaller and faster to construct, making them convenient for NAT hole punching.

I do agree that this is a potential security risk for first-time community developers. That is why I made verified messages the default option.

qstokkink commented 6 years ago

Testing update: now on 82% branch coverage, 132 tests, time in tests 5.721 seconds, time with overhead (test suite/script setup and teardown) 14.496 seconds. Moving on to other boring parts.

qstokkink commented 6 years ago

We are now sporting a README: https://github.com/qstokkink/py-ipv8 Now, just writing extended documentation 🤢 and running the tests on other platforms remain (I'll probably use Jenkins for this).

UPDATE: Unit tests and coverage run automatically on Jenkins (unix) when I push to the repository: https://jenkins.tribler.org/job/pers/job/test_ipv8_qstokkink/lastSuccessfulBuild/artifact/coverage/index.html UPDATE2: unit tests are also succeeding on Windows: https://jenkins.tribler.org/job/pers/job/test_ipv8_win_qstokkink/lastSuccessfulBuild/artifact/coverage/index.html

qstokkink commented 6 years ago

Ok, I also added missing documentation, which completes the feature set for beta 1. Anyone should now be able to make full use of IPv8's functionality. Time for 4 days of vacation 🌦 😎.

synctext commented 6 years ago

As IPv8 is including web-of-trust and authentication features, please read: https://doi.org/10.6028/NIST.SP.800-63-3 This documents the hard-core NIST guidelines on Digital Identity:

IAL2: Evidence supports the real-world existence of the claimed identity and verifies that the
applicant is appropriately associated with this real-world identity. IAL2 introduces the need for
either remote or physically-present identity proofing. Attributes can be asserted by CSPs to RPs
in support of pseudonymous identity with verified attributes.
qstokkink commented 6 years ago

For my own convenience, this is the executive summary of the NIST identity standards:

IAL Description
1 Self-assert
2 IAL1 + third party assert + pseudonyms
3 IAL2 + third party attribute attestation + physical presence
AAL Description
1 Some authentication protocol
2 Multifactor (at least 2) authentication
3 AAL2 + hardware authenticator + authenticated verifiers
FAL Description
1 Signature from federation
2 Encrypted signature from federation for RP
3 FAL2 + AAL1+
synctext commented 6 years ago

Neat trick to possibly implement for 2018: as measure of last bootstrap failure resort... join the hard-coded #IPv8-bootstrap IRC channel and meet others just like you and talk to them. Inspired by the 8 Bitcoin bootstrap features: https://en.bitcoin.it/wiki/Satoshi_Client_Node_Discovery

synctext commented 6 years ago

Supersingular Elliptic Curve Isogenies is a quantum-resistant cryptosystem. How mature is this work and their Python? Realistic for a 2018 roadmap issue?

qstokkink commented 6 years ago

Alright, I fed the same Community to both Dispersy and IPv8 and had two nodes exchange 10 ping-pong messages with each other. IPv8 is not even in the same ballpark as Dispersy: output Do bear in mind that these are the fastest message settings available for both platforms. Also note that Dispersy was designed for bandwidth efficiency and IPv8 for throughput.

synctext commented 6 years ago

@qstokkink nice! Upcoming Tuesday we hope to be able to present a "messages per second" graph of IPv8 for various settings or just a single number. Plus a number for Dispersy. Performance regression testing would be great to have from this early stage onwards see this amazing example of professional performance testing. Full speed ping-pong is a great performance measure. Another is unconstrained send rate and the rate at which a receiver can still process incoming UDP IPv8 messages without them pilling up somewhere.

synctext commented 6 years ago

btw if a master student showed me your 10 ping-pong speed graph I would send them away to redo it :-)

The linear part from message 3 until message 10 from Dispersy node 1 and 2 is perfect linear, although these are really 7 unique incoming message events taking 15 seconds.

qstokkink commented 6 years ago

@synctext Note that this is just a quick graph to explore the output, not something which should be published. I was also expecting more of a "usual" cumulative distribution function with buffers filling up etc. The main cause for it being so linear is probably the fact that the ping<->pong is synchronous (each ping<->pong waits for the last to finish).

Also, there is probably some hidden 'feature' in Dispersy causing a forced 1 second delay between transmitting two of the same message (it's not actually perfectly linear there is a variance of a few milliseconds). Update: the 1 second delay seems to be hidden somewhere on the receiving side (I tested this by sending directly over the endpoint). You can compensate for this 'feature' by taking the time difference between a pong and the next synchronous ping of the couterparty instead. This puts Dispersy at a time of roughly 21ms/message and IPv8 at roughly 2ms/message.

Given these results I would also say that (1) it indeed makes more sense to just present throughput as a number and (2) we should just test sending messages asynchronously to give Dispersy a fighting chance.

Update 2: I never managed to find this in the past but I finally found out that the 1 second delay (for some reason) starts here: https://github.com/Tribler/dispersy/blob/a68fbe7eba4a6542c8d64975ac0504a0cff5e5b6/endpoint.py#L234 Some weird Twisted scheduling going on apparently. Update 3: The cause of the 1 second delay was a badly implemented timeout for message delivery.

qstokkink commented 6 years ago

Spurred on by the above results I decided to rip out all of the endpoint nonsense from Dispersy and simply exchange messages between communities directly. This is now an unfair comparison as IPv8 is actually sending through its endpoints, but it is still faster: output Note that this is still synchronous communication. I named this supercharged Dispersy version Dispersy+ in the graph. Also, the bend in the beginning of the Dispersy+ plot is due to authorize messages.

Here is the asynchronous case: output_async

qstokkink commented 6 years ago

Alright here is some publishable output of large experiments: manualep standaloneep These are box-and-whisker plots of the round-trip time of messages in Dispersy vs IPv8 using (1) no intermediate routers and (2) 1 intermediate router. Like before, this is a polished up/faster version of Dispersy.

synctext commented 6 years ago

IPv10 trolling at IETF but has good intro text:

- Three short-term solutions (CIDR, Private addressing, and NAT) were
  introduced in the mid-1990s but even with using these solutions,
  the IPv4 address space ran out in February, 2011 as announced by
  IANA, The announcement of depletion of the IPv4 address space by
  the RIRs is as follows:

  * April, 2011:      APNIC announcement.
  * September, 2012:  RIPE NCC announcement.
  * June, 2014:       LACNIC announcement.
  * September, 2015:  ARIN announcement.

- 19 years have passed since IPv6 was developed, but no full migration
  happened till now and this would cause the Internet to be divided
  into two parts, as IPv4 still dominating on the Internet traffic
  (85% as measured by Google in April, 2017) and new Internet hosts
  will be assigned IPv6-only addresses and be able to communicate with
  15% only of the Internet services and apps.
qstokkink commented 6 years ago

@synctext yes, this is hilarious: https://twitter.com/The_Road_Series/status/868140346177355777

qstokkink commented 6 years ago

On the IPv8 attestation component: As it turns out, calculating the Weil pairing on a finite field Elliptic curve using a distortion field is kind of difficult. Only one real implementation is publicly available and complete, and it is written in Mathematica. So, to facilitate the Python implementation, I got myself a Mathematica license from the TU Delft and formalized the Mathematica implementation as pseudocode: formulae.pdf Other than this, the rest of the implementation should be smooth sailing.

Update: I got the Mathematica code up and running, time for polishing and porting.

synctext commented 6 years ago

impressive! A 4 page algorithm. Guess we really want to line up a big pile of testcases to compare both implementations eventually.

qstokkink commented 6 years ago

Making progress translating the Mathematica code into Python. afbeelding

Implementations are rare, no open source library is feature complete. The only other library available has a performance of:

The PBC library is designed to be the backbone of implementations of pairing-based cryptosystems,
thus speed and portability are important goals. It provides routines such as elliptic curve
generation, elliptic curve arithmetic and pairing computation. Thanks to the GMP library, despite
being written in C, pairings times are reasonable. On a 1GHz Pentium III:
Fastest pairing: 11ms
Short pairing: 31ms

Note that Type B is not implemented in this library. However, this is critical for the implementation of the Weil pairing (Abelian mapping) for finite field Elliptic curves using distortion maps.

qstokkink commented 6 years ago

I needed coefficient based operators for the Python implementation, which I worked out here (mostly for my own reference): ffformulea Update: And here it is implemented in Python: ffformulea_python

devos50 commented 6 years ago

Idea (came up during meeting): create an identity validation block in your TrustChain.

qstokkink commented 6 years ago

Alright, the Weil pairing seems to be functional now (Python produces the same value as Mathematica): preview (Mathematica is above the black bar, Python is under the black bar)

qstokkink commented 6 years ago

Attestation update, I can now perform the Stanford encryption. Here is some raw output:

CIPHERTEXTS (ENCODED BIT VALUE: 0):
4467 + 75x
3019 + 4629x
2318 + 4070x
2318 + 4070x
1088 + 1834x
1318 + 2225x
4203 + 1184x
3534 + 2558x
3019 + 4629x
2318 + 4070x
1318 + 2225x
4061 + 1743x
4392 + 5738x
1088 + 1834x
976 + 3255x
976 + 3255x
5067 + 3979x
5067 + 3979x
63 + 5237x
4467 + 75x

DECODED (CIPHERTEXT EXPONENTIATION WITH SK, SHOULD EQUAL g^0=1):
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
synctext commented 6 years ago

Great doc examples:

btw always central list of servers to bootstrap: https://github.com/libp2p/specs/blob/master/4-architecture.md#443-bootstrap-list

qstokkink commented 6 years ago

After some major optimizations: managed the first 512-bit-primes-key generation. Still some room left for speedup. Encryption/decryption itself is pretty fast.

RANDOM CIPHERTEXTS:
27819787572934406955043645093393842401993071469366681292238424796137619677306643704822401176837845947645204143850293453891591259579432512637721892788528947 + 127269305053866411856240263656167770550683818740502830977586904429068538960419847382428445626306588169955492852281714997188187203300826967806657664100386085x
121320047998478831289340101355181059721687546361944999842911173946274855846740576508835311135677350097401565566500597307401060517315873749593281997325211185 + 992665056501391192073459270486308812253456417209749851122631094903993455368064223832184992685028381073760637676338743854592830166268112690103770515787088x
165726942691611537722565503796304737203706713276071135536436385769342229544214397727932176683554222107632984298683789248179118639577498865888136382261567108 + 157981475677911198276430776020436291396737662344945486541624376254311901520421268911513231119734547280233950160326096430574824182053524070957352877914390203x
130200020437030443544603127176135889224888771494049811957338425255413285082108104855732607682362196233329342927242209686602158047616746305848018292520711198 + 132226935570814425704817021574265878850011584614806793295121561638924729783155632379015289305596310697652941681758402857655796258409317249088508406774493176x
110719948931283941057187983691216140403502429301765269439516963474941613604084883765420684437779497587737016002276630820468184617712931968441709253554338703 + 60370133285663657060401187796729473649240385523812726231277187547826580731991976511524927508285634624107970657913672507046750069321843085770165183862120155x
60652512444857048143098802167520153984106429919197478769828080742343336036642581067893127045534660471801170453490067391764835883979185761152398633272722296 + 46905838323311527982788437853097496490405274315352299117203260928536230764240664226596160478447508959549929193823392085873516411326405319290104775956058288x
66165640286271503179560314555603371912689710010813730692678814176529489335148211794322595647537743595087106348292111183020170970602269205326336585136077977 + 84845234921119080616624114880680483221074918260088667086855131218672763883656947087990370652102649458956895152076020617730362634954839970824753571025851212x
110682688728521575412758157767852881565718286772888923800500319599995110639261152305448742802048614322157332759789589414643355903097395498285532000163152076 + 119615694458238726986906378850051493185090803986815471090937147976779480888462315913299990289707636915395192905101200790759249175203345556644941994139851335x
165970072031676447730638238197191305516211228227854547150412912143639617564931467583799628730062799195340040423629935744750773992167459601608267352620111173 + 103214293313527887854043870308368258982524781740017606269573765609598664885292255104086760925690704177075858611417274898765475646618240491777422402349883664x
77497255732997188725991095890352485981599596699104475860171469784130169470584820932129009210911014419713539646544516310742667472897876341854503019039145005 + 73514075927872420531775351073912512956003967378501056795679890191010660234819259991010683932051230528639714962547149365034975941038126831701971335118475324x
59098145107605158921046050296354854412778472144398939942945658504189233290320403048726759691714746395864572084219318631161238505105760344010024244697020284 + 94929046558930929094187164600200273555499860427264253446115533927906013582742021193406356557070146550343821486284117887655821795569539853342891138760948831x
28414691139304030668669687052297775523533775487133723385101025862281651075666806795190038766397114259509309098811139037205869907680396160493104331089996949 + 106220645424240243588172370987508966386281324914639081960321926504970664098696062081166187313427343050793121466531534230771199145391243751570754963273429321x
123774743349305464935212416574016236445211208767843718367576419920059679632051231647078082860690936289877238858080827262694036400770440377500783533273787883 + 35571857516981593067757520372604831936845092063992518466151119674769687775690683526232862435528988650127411223593213727625737704484339099103213094456874418x
42944656913437718781153554975259492048986385633566985221615601337209481418141625087417673148108659075013760373975618763505078978766146778905747320510511522 + 40499550451974623016246374636960043598776271751443633630666531273847067370258777945089279550996120615107500493453265835981931698784463119417792452226667955x
106957416702359616794844470395410297067474379336183917441899059577039586635264700166259201350460898114443268581574321495977478664919064518686504560370492225 + 71897464142060450030931507674757648224533678212844544249277635207020356025278141154625175407148177163768807294250392284497092951194447997506132149678552714x
123275424588819998626435222785112162149769503581580007550650434387525567165495198037168879318101936672165633149135505165757847799832270858543973361820357245 + 59117066022973382699618728563111733089869773923249293176984415581027564901768740504406543631210788811136340586261158274694300791805522738795895971688280238x
63341630495795797332212197850122580710036950544082647555788844298412831996024231488964687061168546157283373961711964636045510876004675343449972440230316465 + 112585731153913932063162836579066385241845518424913751013043871121554112940229095197283536850495465591797614509743983956393383051685763059733183376048815124x
186728210054630030123954257128454460219908580197330657739943249315835989785844583725141425562044718725203936882712883949340142113334964166945341064459558271 + 60876517965846457983179101940302301528889925872532375001149237067283904574021039465900094936284484478229468766752470659281549946844504373490724208380403597x
99140531571271884859545654324786630837061714201360583296118192083407488219291627113492161824026747782899343818171287711090639904463459896102360641108213613 + 167327457578734220805264616363014055223891136745094941713074710793062374502861320792064736650852539205189147429862759906956667139329487504809678079664398255x
69718824430869421624048666651720723923912832179594072390321198837663068249741181123466104223262477188461231690586597100634019567588013134339643128409113556 + 59131644368552319847780699908874772101063559350860391205989075222394093198082077193243574986702587300347155499680747729971154973271917641303541701449222000x
----------------------------------------
SHOULD ALL BE 1:
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1

I also created a super secret repository to keep track of the code. Once I'm happy with the code and its performance, I will remove the repo and move the code to the IPv8 repository.

synctext commented 6 years ago

Please read IPv8 related work from IETF: https://tools.ietf.org/html/draft-padma-ideas-problem-statement-03 different storyline, but in-scope for IPv8. [updated] Wow, they are already chartering. An "IPv8"-ish workgroup exists at IETF: https://datatracker.ietf.org/doc/charter-ietf-ideas/ Plus another similar one: https://datatracker.ietf.org/doc/draft-ietf-lisp-introduction/?include_text=1

qstokkink commented 6 years ago

Status update: I optimized the code enough that I can generate a key using 1024 bit primes in about 20 seconds. It's still not fast, but its secure and usable. I'll move on to implementing the actual zero-out scheme on top of the cryptosystem now.

qstokkink commented 6 years ago

I have generic attribute attestation now, here is the output of a full proof of a 64 bit number:

Created attestation! 64 bit number took 1.95406484604 seconds
Expected:   {0: 20, 1: 9, 2: 3, 3: 0}
Got:        {0: 20, 1: 9, 2: 3, 3: 0} after 32 challenges
Proof took 0.333495140076 seconds

I also want to add support for proving ranges (which should be somewhat easy). After that I'm going to do a big code cleanup and then merge it into IPv8.

Also, some output of the certainty function:

Created attestation for 1152354643
=====================================
Certainty for: 1152354643 7 6545139 1152354642
-------------------------------------
Certainty: 0.287356321839 % 0.26455026455 % 0.282485875706 % 0.287356321839 %
Certainty: 0.574712643678 % 0.529100529101 % 0.564971751412 % 0.574712643678 %
Certainty: 0.862068965517 % 0.793650793651 % 0.847457627119 % 0.862068965517 %
Certainty: 1.14942528736 % 1.0582010582 % 1.12994350282 % 1.14942528736 %
Certainty: 1.4367816092 % 1.32275132275 % 1.41242937853 % 1.4367816092 %
Certainty: 1.72413793103 % 1.5873015873 % 1.69491525424 % 1.72413793103 %
Certainty: 2.01149425287 % 1.85185185185 % 1.97740112994 % 2.01149425287 %
Certainty: 2.29885057471 % 2.1164021164 % 2.25988700565 % 2.29885057471 %
Certainty: 2.58620689655 % 2.38095238095 % 2.54237288136 % 2.58620689655 %
Certainty: 2.87356321839 % 2.6455026455 % 2.82485875706 % 2.87356321839 %
Certainty: 3.16091954023 % 2.91005291005 % 3.10734463277 % 3.16091954023 %
Certainty: 3.44827586207 % 3.1746031746 % 3.38983050847 % 3.44827586207 %
Certainty: 3.73563218391 % 3.43915343915 % 3.67231638418 % 3.73563218391 %
Certainty: 4.02298850575 % 3.7037037037 % 3.95480225989 % 4.02298850575 %
Certainty: 4.31034482759 % 3.96825396825 % 4.23728813559 % 4.31034482759 %
Certainty: 4.59770114943 % 4.2328042328 % 4.5197740113 % 4.59770114943 %
Certainty: 4.88505747126 % 4.49735449735 % 4.80225988701 % 4.88505747126 %
Certainty: 8.58876117497 % 37.8306878307 % 13.1355932203 % 8.2183908046 %
Certainty: 12.2924648787 % 0.0 % 21.4689265537 % 11.5517241379 %
Certainty: 12.5798212005 % 0.0 % 21.7514124294 % 11.8390804598 %
Certainty: 23.6909323116 % 0.0 % 27.3069679849 % 28.5057471264 %
Certainty: 23.9782886335 % 0.0 % 27.5894538606 % 28.7931034483 %
Certainty: 24.2656449553 % 0.0 % 27.8719397363 % 29.0804597701 %
Certainty: 24.5530012771 % 0.0 % 28.1544256121 % 29.367816092 %
Certainty: 24.840357599 % 0.0 % 28.4369114878 % 29.6551724138 %
Certainty: 35.9514687101 % 0.0 % 33.9924670433 % 46.3218390805 %
Certainty: 36.2388250319 % 0.0 % 34.274952919 % 46.6091954023 %
Certainty: 36.5261813538 % 0.0 % 34.5574387947 % 46.8965517241 %
Certainty: 36.8135376756 % 0.0 % 34.8399246704 % 47.183908046 %
Certainty: 37.1008939974 % 0.0 % 35.1224105461 % 47.4712643678 %
Certainty: 37.3882503193 % 0.0 % 35.4048964218 % 47.7586206897 %
Certainty: 37.6756066411 % 0.0 % 35.6873822976 % 48.0459770115 %
Certainty: 41.3793103448 % 0.0 % 44.0207156309 % 51.3793103448 %
Certainty: 41.6666666667 % 0.0 % 44.3032015066 % 51.6666666667 %
Certainty: 41.9540229885 % 0.0 % 44.5856873823 % 51.9540229885 %
Certainty: 42.2413793103 % 0.0 % 44.868173258 % 52.2413793103 %
Certainty: 42.5287356322 % 0.0 % 45.1506591337 % 52.5287356322 %
Certainty: 42.816091954 % 0.0 % 45.4331450094 % 52.816091954 %
Certainty: 46.5197956577 % 0.0 % 53.7664783427 % 56.1494252874 %
Certainty: 46.8071519796 % 0.0 % 54.0489642185 % 56.4367816092 %
Certainty: 47.0945083014 % 0.0 % 54.3314500942 % 56.724137931 %
Certainty: 50.7982120051 % 0.0 % 0.0 % 60.0574712644 %
Certainty: 51.0855683269 % 0.0 % 0.0 % 60.3448275862 %
Certainty: 51.3729246488 % 0.0 % 0.0 % 60.632183908 %
Certainty: 51.6602809706 % 0.0 % 0.0 % 60.9195402299 %
Certainty: 51.9476372925 % 0.0 % 0.0 % 61.2068965517 %
Certainty: 52.2349936143 % 0.0 % 0.0 % 61.4942528736 %
Certainty: 55.938697318 % 0.0 % 0.0 % 64.8275862069 %
Certainty: 56.2260536398 % 0.0 % 0.0 % 65.1149425287 %
Certainty: 56.5134099617 % 0.0 % 0.0 % 65.4022988506 %
Certainty: 56.8007662835 % 0.0 % 0.0 % 65.6896551724 %
Certainty: 57.0881226054 % 0.0 % 0.0 % 65.9770114943 %
Certainty: 57.3754789272 % 0.0 % 0.0 % 66.2643678161 %
Certainty: 57.662835249 % 0.0 % 0.0 % 66.5517241379 %
Certainty: 57.9501915709 % 0.0 % 0.0 % 66.8390804598 %
Certainty: 58.2375478927 % 0.0 % 0.0 % 67.1264367816 %
Certainty: 58.5249042146 % 0.0 % 0.0 % 67.4137931034 %
Certainty: 58.8122605364 % 0.0 % 0.0 % 67.7011494253 %
Certainty: 59.0996168582 % 0.0 % 0.0 % 67.9885057471 %
Certainty: 59.3869731801 % 0.0 % 0.0 % 68.275862069 %
Certainty: 59.6743295019 % 0.0 % 0.0 % 68.5632183908 %
Certainty: 59.9616858238 % 0.0 % 0.0 % 68.8505747126 %
Certainty: 60.2490421456 % 0.0 % 0.0 % 69.1379310345 %
Certainty: 60.5363984674 % 0.0 % 0.0 % 69.4252873563 %
Certainty: 60.8237547893 % 0.0 % 0.0 % 69.7126436782 %
Certainty: 61.1111111111 % 0.0 % 0.0 % 70.0 %
Certainty: 61.398467433 % 0.0 % 0.0 % 70.2873563218 %
Certainty: 61.6858237548 % 0.0 % 0.0 % 70.5747126437 %
Certainty: 61.9731800766 % 0.0 % 0.0 % 70.8620689655 %
Certainty: 62.2605363985 % 0.0 % 0.0 % 71.1494252874 %
Certainty: 62.5478927203 % 0.0 % 0.0 % 71.4367816092 %
Certainty: 66.251596424 % 0.0 % 0.0 % 74.7701149425 %
Certainty: 66.5389527458 % 0.0 % 0.0 % 75.0574712644 %
Certainty: 66.8263090677 % 0.0 % 0.0 % 75.3448275862 %
Certainty: 67.1136653895 % 0.0 % 0.0 % 75.632183908 %
Certainty: 67.4010217114 % 0.0 % 0.0 % 75.9195402299 %
Certainty: 67.6883780332 % 0.0 % 0.0 % 76.2068965517 %
Certainty: 67.975734355 % 0.0 % 0.0 % 76.4942528736 %
Certainty: 68.2630906769 % 0.0 % 0.0 % 76.7816091954 %
Certainty: 68.5504469987 % 0.0 % 0.0 % 77.0689655172 %
Certainty: 68.8378033206 % 0.0 % 0.0 % 77.3563218391 %
Certainty: 69.1251596424 % 0.0 % 0.0 % 77.6436781609 %
Certainty: 69.4125159642 % 0.0 % 0.0 % 77.9310344828 %
Certainty: 69.6998722861 % 0.0 % 0.0 % 78.2183908046 %
Certainty: 69.9872286079 % 0.0 % 0.0 % 78.5057471264 %
Certainty: 70.2745849298 % 0.0 % 0.0 % 78.7931034483 %
Certainty: 70.5619412516 % 0.0 % 0.0 % 79.0804597701 %
Certainty: 70.8492975734 % 0.0 % 0.0 % 79.367816092 %
Certainty: 71.1366538953 % 0.0 % 0.0 % 79.6551724138 %
Certainty: 71.4240102171 % 0.0 % 0.0 % 79.9425287356 %
Certainty: 71.711366539 % 0.0 % 0.0 % 80.2298850575 %
Certainty: 71.9987228608 % 0.0 % 0.0 % 80.5172413793 %
Certainty: 72.2860791826 % 0.0 % 0.0 % 80.8045977011 %
Certainty: 72.5734355045 % 0.0 % 0.0 % 81.091954023 %
Certainty: 72.8607918263 % 0.0 % 0.0 % 81.3793103448 %
Certainty: 73.1481481481 % 0.0 % 0.0 % 81.6666666667 %
Certainty: 73.43550447 % 0.0 % 0.0 % 81.9540229885 %
Certainty: 73.7228607918 % 0.0 % 0.0 % 82.2413793103 %
Certainty: 84.8339719029 % 0.0 % 0.0 % 0.0 %
Certainty: 85.1213282248 % 0.0 % 0.0 % 0.0 %
Certainty: 85.4086845466 % 0.0 % 0.0 % 0.0 %
Certainty: 85.6960408685 % 0.0 % 0.0 % 0.0 %
Certainty: 85.9833971903 % 0.0 % 0.0 % 0.0 %
Certainty: 86.2707535121 % 0.0 % 0.0 % 0.0 %
Certainty: 86.558109834 % 0.0 % 0.0 % 0.0 %
Certainty: 86.8454661558 % 0.0 % 0.0 % 0.0 %
Certainty: 87.1328224777 % 0.0 % 0.0 % 0.0 %
Certainty: 87.4201787995 % 0.0 % 0.0 % 0.0 %
Certainty: 87.7075351213 % 0.0 % 0.0 % 0.0 %
Certainty: 87.9948914432 % 0.0 % 0.0 % 0.0 %
Certainty: 88.282247765 % 0.0 % 0.0 % 0.0 %
Certainty: 88.5696040868 % 0.0 % 0.0 % 0.0 %
Certainty: 88.8569604087 % 0.0 % 0.0 % 0.0 %
Certainty: 89.1443167305 % 0.0 % 0.0 % 0.0 %
Certainty: 92.8480204342 % 0.0 % 0.0 % 0.0 %
Certainty: 93.1353767561 % 0.0 % 0.0 % 0.0 %
Certainty: 96.8390804598 % 0.0 % 0.0 % 0.0 %
Certainty: 97.1264367816 % 0.0 % 0.0 % 0.0 %
Certainty: 97.4137931034 % 0.0 % 0.0 % 0.0 %
Certainty: 97.7011494253 % 0.0 % 0.0 % 0.0 %
Certainty: 97.9885057471 % 0.0 % 0.0 % 0.0 %
Certainty: 98.275862069 % 0.0 % 0.0 % 0.0 %
Certainty: 98.5632183908 % 0.0 % 0.0 % 0.0 %
Certainty: 98.8505747126 % 0.0 % 0.0 % 0.0 %
Certainty: 99.1379310345 % 0.0 % 0.0 % 0.0 %
Certainty: 99.4252873563 % 0.0 % 0.0 % 0.0 %
Certainty: 99.7126436782 % 0.0 % 0.0 % 0.0 %
Certainty: 100.0 % 0.0 % 0.0 % 0.0 %
synctext commented 6 years ago

Excellent progress..

btw fast prime generation bug http://www.reuters.com/article/us-infineon-cyber/infineon-says-has-fixed-encryption-flaw-found-by-researchers-idUSKBN1CL2KC?feedType=RSS&feedName=technologyNews

qstokkink commented 6 years ago

Slight change of plans: I did the code cleanup today and I'll integrate this into IPv8 next. I'll leave the range proofs for after the MVP has been released.

synctext commented 6 years ago

Cardinal requirement for Dispersy re-write: Both metadata and content will move as fast as the underlying network permits (see example requirement list https://github.com/ipfs/faq/issues/47)

qstokkink commented 6 years ago

Small milestone: code moved from the secret project repository to the IPv8 repository.

UPDATE: The community code is now done. Code-wise I only need to publish the attestation hash + metadata to the TrustChain to complete the MVP. After that I will write some tests and the MVP is done.

UPDATE2: The secret project repository is no more. In other news, the first unit tests have been created for the cryptographic primitives.

qstokkink commented 6 years ago

Larger milestone: unit tests are done for the basic attestation community, now I just need a way to advertise attestation hashes and metadata: I was thinking of using TrustChain for this.