search

Tribler / tribler

Privacy enhanced BitTorrent client with P2P content discovery
https://www.tribler.org
GNU General Public License v3.0
4.86k stars 450 forks source link

Universal communication using imperfect hardware #4827

Closed synctext closed 4 years ago

synctext commented 5 years ago

placeholder issue for thesis/survey

Skeleton GUI + 30-ish citation paper. Example from few years ago: "Survey of robust and resilient social media tools on Android" https://www.google.nl/search?q=friend+to+friend+torrent Switch between Libtorrent-broadcast and direct QUIC based transport to your friends Record protests, broadcast to the world. Please investigate: Hong Kong protesters using Bluetooth Bridgefy app. Plus Life in an Internet Shutdown: Crossing Borders for Email and Contraband SIM Cards. Our own proposal from pre-Snowden days, 7 years ago: https://tools.ietf.org/html/draft-pouwelse-censorfree-scenarios-02 Explicit offline/bluetooth,private and open Internet status:

synctext commented 5 years ago

Open questions:

Possible thesis scope:

synctext commented 5 years ago

https://github.com/MichaelMure/Arbore

synctext commented 5 years ago

proof-of-principle inside the survey with 30+ citations and screenshots (IEEE 2-column). Screenshots of prototype V0.013? science Can we build a store-and-forward based mesh network within the Android security framework? This would service as the principle building block for decentral friend-to-friend network. (Rust?, GPL-openness Bridgefy SDK?)

MattSkala commented 5 years ago

Current survey topic: Smartphone ad hoc networks (SPANs)

Up-to-date reading list, notes + latex draft: https://github.com/MattSkala/meshnet-survey Running code: https://github.com/MattSkala/meshnet-android

Current sprint goal:

Related papers:

synctext commented 5 years ago

Great progress! Screenshot?

MattSkala commented 5 years ago

UI not that interesting, but Nearby Connections and Bluetooth transports are working.

synctext commented 5 years ago
MattSkala commented 5 years ago

Last sprint:

Currently implemented communication protocols:

(Have not been able to test BLE L2CAP and WiFi Aware yet due to the lack of supporting devices)

Next sprint:

trustchain-android could be useful after some refactoring (the core logic should be extracted into a reusable Java library)

synctext commented 5 years ago

Emphasis on science: any-time any-place communication, No-Internet-Needed communication paradigm. Talked about for 40-years, no significant deployment yet. Finally the end-user has software access to the advanced mesh-networking capable devices on affordable devices. research question. For appealing writing: No-Internet-Needed science, state-of-the-art (screenshots of barely useful apps), underlying wireless technologies, Operating System Support, Our "MultiConnect" proof-of-principle, etc. Thesis, live mesh, like we did previously? img_20180123_114422

MattSkala commented 5 years ago

Testing setup. Each device connected to two other devices, all of them able to communicate with each other using a simple flooding-based mesh network.

IMG_20000101_150509

synctext commented 5 years ago

Impressive prototype, operational ad-hoc networking and broadcasting. No crypto yet in this early work. messaging works! Ad-hoc secure connection fabric

Discussed thesis direction potential: universal any-medium connector and bandwidth accounting (carrier-grade puncturing,freeriding,Trustchain recording; 10k open sockets to puncture symmetric NATs). Any phone as a Internet-relay. "collaborative wireless Internet". 'We are the Internetz" stripped IPv8 @ Kotlin? and/or Binary blob transfer on top of UDP in-scope?

devos50 commented 5 years ago

Quite related excerpt from the Redecentralized digest:

Meanwhile, in Catalonia, pro-independence group Tsunami Democràtic created an Android app based on Retroshare, the peer-to-peer communication and file sharing software that feels nearly old enough to deserve its name. It’s used in protest coordination, and is currently not even available for iOS nor published on Google Play: people have to download the app as an APK file from its website.

Ironically, GitHub then took the app down for visitors from Spain, after an order from Spanish law enforcement — which is novelty by itself, Spain now being the third country on GitHub’s takedown log, besides Russia and China. But at least, unlike the very centralised landscape of app stores, alternative web hosting options are plenty; and files can be passed around in many ways.

Retroshare seems to align with the goal of making a peer-to-peer social network overlay. It support chatting, mail, file sharing, forums, link sharing and even channel functionality. There even seems to be an experimental VOIP plugin for video and audio chatting.

synctext commented 5 years ago

in prison for using offline sharing app plus: https://www.bbc.com/news/world-asia-china-50511063

MattSkala commented 4 years ago

The Internet has been shut down for 120 days in Kashmir, WhatsApp deleting inactive accounts: https://www.buzzfeednews.com/article/pranavdixit/hundreds-of-kashmiris-are-disappearing-from-their-whatsapp

synctext commented 4 years ago

How Facebook Avoids Ad Blockers The next technique, this is the point where it gets particularly interesting, Facebook actually hides 'dummy' DOM nodes between the 'Sponsored' text. These values are entirely random characters, with a random number of DOM nodes between them. Invisible characters. image

MattSkala commented 4 years ago

Twitter is funding a small team of researchers to build an “open and decentralized standard for social media,” with the goal of making Twitter a client for that standard.

https://www.theverge.com/2019/12/11/21010856/twitter-jack-dorsey-bluesky-decentralized-social-network-research-moderation

MattSkala commented 4 years ago

Current thesis proposal:

Multi-transport Peer-to-peer Communication Protocol for Mobile Devices

synctext commented 4 years ago

Turkey: arrest of 75,000 suspects, primarily for downloading ByLock app. India Internet shutdowns: https://internetshutdowns.in/ + HN discussion

synctext commented 4 years ago

Always On Overlay: Universal Connectivity using imperfect hardware

synctext commented 4 years ago

Related work for decentral social network. You need to obtain the public key of your friends within a web-of-trust.

synctext commented 4 years ago

Remarks:

MattSkala commented 4 years ago

Progress:

synctext commented 4 years ago

Sprint feedback:

MattSkala commented 4 years ago

Update:

ProphetZarquon commented 4 years ago

Fascinating research here! I am convinced that censorship resistant internet-independent decentralized communication apps will never reach a geographic density of peers sufficient to ensure connectivity throughout urban (much less rural or wilderness) areas until cellular or other 1mi+ bands can be leveraged by said app. Local mesh connectivity is all well & good for research & in limited use cases where protests are localized or longer range centralized subscription service is available... but unfortunately the need for unlicensed 1mi+ ranges seems obvious, if free communication is to be assured.

I don't know what to suggest, as even LTE-Direct & LTE-V protocols do not permit connections without permission from a carrier tower, & cellular modems in commonly available end-user devices would require homebrew firmware to bypass such restrictions (or to work as SDR in xmit mode); I'm quite certain longer ranges are needed to make decentralized communication functional enough to achieve general viability though.

Please keep working on these projects (I hope my interjection is not unwelcome!) & know that I applaud your efforts... I am just tired of seeing one project after another languish in isolated deployments & low functional adoption. I believe that (whether legally or not) cellular or other bands with 1mi+ range must be leveraged in order to escape communications captivity.

Without specifically advocating such a course of action, I'd like to hear informed thoughts on this subject?

synctext commented 4 years ago

ports are assigned at random from the range of 45k-55k, port prediction not possible

devos50 commented 4 years ago

This hackaton project uses an Android app and Bluetooth features to check for people with COVID-19 in your vicinity. From the website:

WeTrace uses Bluetooth LE to keep track of people you were close to. It automatically collects the ids of other users while it shares its own ID to them. When a user is tested positive, he or she can report the status change. A silent notification is broadcasted to all the users. That then can locally compare the newly reported case with their database of collected user IDs. If the reported ID is found in the database, the user that he or she might be at risk - the app will inform the user.

synctext commented 4 years ago

@ProphetZarquon Yes, you indeed need long-hail links if you want to build a useful network. It seems already 8 years ago I did an IETF Internet Standard draft writeup on kill-switches from governments and what to do against them. Moving information around on smartphones, store-and-forward, seems like the only viable method. Directional antenna will only get you so far. Maybe in another 10 years we will have useful soft beamforming MIMO in our software radios in each phone...

ProphetZarquon commented 4 years ago

Without robust decentralized long-range communication options, I think ten years from now anyone knowledgeable enough to establish such solutions will be too scared to even discuss it. Already ~half~ almost all the projects I've seen showing promise have gone dark, & while I'd like to believe that's because the people involved went underground, instead I suspect they either took suppression-industry jobs or were put in the ground permanently.

MattSkala commented 4 years ago
synctext commented 4 years ago

sprint progress, latest .PDF thesis:

synctext commented 4 years ago

Title proposal: "Reliable Trustworthy Communication using faulty hardware", is more accurate and catchy. The harsh term fault is justified because IETF Internet Standards are violated by network operators. Trust-paradox is also linked in title now. Alternative to "Universal communication using imperfect hardware". Question: what scientific problem did the superapp solve?

MattSkala commented 4 years ago

image

synctext commented 4 years ago

:clap: With this milestone it becomes realistic to dream of adding address book functionality. Thus yielding a zero-server secure social network. Even your thesis title could reflect this. But your framework can also be extended to a MusicDAO, zero-server AI, and is thus a generic Big Tech alternative library.

The rate limiter forces you to use a certain UDP puncture method. Either detect using the behavioral pattern (switches in external IPv4) or hard-code the IPv4 blocks like AS33915 Vodafone Libertel B.V.. Advertise a signed block of your current NAT behavior and puncture effectiveness?

Final thesis sprints:

Thesis remarks:

synctext commented 4 years ago

Problem Description: The Internet is under continuous attack (Suggestion of alternative storyline)

Many Internet protocols, security companies, and security models have been compromised. Within this thesis we address a big open research question, can we make non-trivial software systems that are safe?

We identify three classes of attacks on The Internet: attacks on the protocol level, compromised central companies, and broken ecosystems. Many protocols such as email are inherently unsafe to use. However, we still rely daily on such fundamentally broken Internet protocols. Companies that are hires to secure other companies and banks have themselves been hacked. For instance, RSA using the SecureID two-factor authentication product (40 million customers) was infiltrated with a simple social engineering trick and a zero-day vulnerability in Adobe Flash. The digital sectificates we use to keep ourselves safe are compromised. This brings into question the integrity of the digital-certificate model itself. Can we rely on trusted third parties to be trustworthy? We have seen time and time again that companies are not incorruptible saints. The entire ecosystem of computer security is not provides users with safe computers. By not holding software vendors accountable for security problem they lack the direct incentive to address security. The security of endpoints is poor and financial damage done to users is never compensated. Software vulnerabilities in various machines has reached an significant level. To illustrate this point, we turn to botnets, which have automated the hacking of consumer and company computers. A study of Torpig explains that hijacked computers form a botnet, which itself may get hijacked. This is known as a botnet takeover.

These problems lead us to the following detailed problem description. Can we remove the reliance on any outside party, protect against malware, use open source software with proven software correctness, offer perfect forward secrecy for our communication, and protect against "Strong Existential Unforgeability under Chosen Message Attack"? Within this thesis we focus on the secure communication part within the grand question of how to make the Internet safe again. Other researchers within my research lab designed a system for trustworthy code module execution.

synctext commented 4 years ago

Latest .APK Replacing Big Tech storyline inspiration "3.4. P2P Communication Libraries" should be 3.1 and mention scuttlebutt.nz + briarproject.org 2004 related work, "DHT Routing Using Social Links" TUDelft related work 100 Million DHT replies Sub-second DHT paper

synctext commented 4 years ago

The scientific problem we focus on within this thesis is how can we bootstrap an ecosystem which will make The Internet safe and fair. The thesis approach to this ideological-driven scientific problem is to remove any central point from the ecosystem which can potentially be compromised or corrupted to act against the user interest.

We propose a zero-server architecture which removes structural bottlenecks for security, privacy, fairness and governance. This thesis provides irrefutable proof that our zero-server architecture is capable of solving wicked problems such as online voting, shared ownership of money, decentralised machine learning, bounty-based software engineering, fully automated decentralised marketplaces, and replace media industry intermediaries with code. We experimentally validated our zero-server architecture with software developers at master-level. A student development team with 17 members successfully created various non-trivial extensions to our foundational zero-server architecture within 10 weeks.

Our zero-server architecture offers various networking primitives. These serve as the basic building blocks for creating full fledged alternatives for the services of "trusted" third parties or Big Tech companies. They are:

  1. Identity layer with public key of your friends or businesses you interact with
  2. Secure end-to-end encrypted communication bypassing NAT boxes and other imperfect hardware
  3. Always connected to your friend using social-based zero-server overlay with automated Internet address discovery based on a UDP-puncturing capable DHT
  4. Secure messaging of any message size
  5. Distributed bookkeeping with tamper-resilience using Trustchain

This thesis builds upon the recent breakthrough of the superapp paradigm. We focus on the latest development in software engineering for smartphones because this is currently the dominant method for Internet access for a large part of the worlds population. In a very short period 1 million "mini-apps" have been developed for the popular Wechat ecosystem using the Superapp paradigm. [1, 2,3, 4, 5].

Our Superapp based on the novel zero-server architecture has several desirable properties. We believe that this thesis proves the first proof-of-principle of a disruptive open innovation ecosystem which is resilient against attacks by disrupted or disintermediated corporate entities. Especially legal intimidation, lawyer-based attacks and creation of chilling effects. Within our 10 week experiment several mini-apps where developed successfully.

devos50 commented 4 years ago

Related work for a potential storyline: An Economic Theory of Infrastructure and Commons Management.

This work contains a discussion on the management of the digital commons. Section IV: "At the heart of this debate is whether the Internet should retain its end-to-end architecture and continue to be managed in an openly accessible manner."

synctext commented 4 years ago

LATEST .pdf of thesis

synctext commented 4 years ago

Reclaiming The Internet for the social good

This master thesis empowers citizens to take back control of The Internet. More specifically, we present the self-organising technology stack to take back The Internet. Who owns The Internet? The Internet is essentially private property, with few exceptions. Google, Facebook, Amazon, Apple, Tencent, and others operate the central components of our daily digital lives. For instance, we require permission from Google and Apple to publish software for mobile devices. Their monopoly power means no other meaningful method exists to reach billions of smartphone users with newly created apps.

This thesis introduces a novel type of low-level network overlay and proof-of-principle zero-server social network. Our zero-server architecture offers various networking primitives. These serve as the basic building blocks for creating full fledged alternatives for the services of "trusted" third parties or Big Tech companies. These primitive increase in strength and cover identity management, communication, code execution and social software engineering.

Within the context of this thesis we conducted an experiment with computer science students. A student development team at master-level with 17 members successfully created various non-trivial extensions to our zero-server architecture within 10 weeks. With their efforts the "superapp" presented within this thesis provides operational features such as online voting, shared ownership of money, fully automated decentralised marketplaces, and a Spotify alternative.

The full list of Superapp features, at an early proof-of-principle stage of software development:

  1. Secure address book. Obtain the public key of your friends with protection against man-in-the-middle attacks. This forms the identity layer with public key of your friends or businesses you interact with
  2. Universal communication using imperfect hardware. Secure end-to-end encrypted communication bypassing NAT boxes and other imperfect hardware. Connect to any address book entry in a secure manner. Secure messaging of any message size.
  3. Robust social fabric. Discover the current Internet address of your friends and connect to them. When your friends or trusted businesses come online they will form an end-to-end encrypted connection with you. Address churn by creating a social-based zero-server overlay with automated Internet address discovery, based on a UDP-puncturing DHT.
  4. Community formation. Everybody can freely start a community which can grow to any size. This is the core of our Decentralized Autonomous Organization. Key features are discovery of DAO communities and joining them.
  5. Community accounting. For each address book entry we track their contributions to the common good. Distributed bookkeeping with tamper-resilience using Trustchain
  6. Democratic governance. Community members can propose votes with proposition for financial transactions.
  7. Shared ownership of money. These communities of any size can democratically control any amount of money. By utilising Bitcoin Multisig we provide a community of any size the democratic control for any amount of money.
  8. Distributed Apps. Freely executing code forms the beating heart of the DAO. Improvement of smart contract execution model, specifically optimised for smartphones usage with runtime code injection and bypass the Google security model.
  9. Distributed markets. Marketplace with bid/ask orderbook which is distributed.
  10. Bounty-based software engineering. Any developer can seed a distributed app in Bittorrent and a DAO can vote on payout of bounty.
  11. Decentralised machine learning. Without any central point of control or performance bottleneck perform multi-dimensional curve fitting.
  12. Distributed music industry. Support artists by making them self-published, self-promoted, and self-sufficient in general. Replace the music industry offerings with free software. Grown into a full master thesis aiming to remove all proprietary infrastructure between artists and ears of listeners with cost-free ecosystem.
synctext commented 4 years ago

The superapp is new paradigm in software engineering. Users have an app with numerous possibilities each of them integrated and deep understanding of your social graph and who you trust. No leakage of this vital information to others or central servers. It also serves as a social coding platform where each module contribution gets global visibility without the burden of marketing and (fake) app review hunting.

Progress: social network is operational in Beta with friend feeds :1st_place_medal: https://github.com/Tribler/trustchain-superapp/pull/41

Thesis writing first now, online profiles with photos are secondary. ToDo: 1) scientific paper by 31 Aug in 6-pages of your work 2) Google Play store listing of superapp 3) a few key screenshots with epic something..

Engineering first principle..: "5.1. Project Structure" and only then the scientific key insight of "5.2. System Architecture". Likewise "5.7. Library Usage", "5.7 creating a community"; we aim to germinate a community through software re-usage and defining an IETF Internet Standard around this work. Wild idea for results section, buy positive reviews and document results for a stand-alone app; stay legal. ToDo: graphs with experimental results in key "performance evaluation and experiments" chapter. Response time with 1000 friends in your address book, etc.

synctext commented 4 years ago

Supertrust: fixing the Facebook monopoly

Roughly 2 billion humans interact with Facebook every day, it has a 75% market share. Facebook has chosen not to compete on the merits of its products and services and instead has misled, deceived, and exploited consumers and publishers. Evidence discussed in an analysis paper shows that for 10 years Facebook has avoided competition in the social network market. Professor Fiona M. Scott Morton from Yale University states that: "Facebook has engaged in a long-term, integrated, anticompetitive strategy of half-truths about its privacy policies, exclusionary API manipulation, and anticompetitive acquisitions of nascent competitors that led to its current dominance of a market in which it now wields significant power over consumers, advertisers, and publishers". This master thesis is dedicated to the recommendation made within this study by economists of anti-competitive conduct: "remedial measures might need to include mandatory interoperability among competing platforms and complements". We successfully created the architecture plus implementation of our foundational layer for interoperability and data portability to all social network platforms. Our work consists of 3 parts: our zero-server architecture, fully operational network primitives, and extension of features using mini-apps.

We propose a zero-server architecture for social networking which removes the need for central control by Facebook. This thesis provides irrefutable proof that our zero-server architecture is capable of creating the first interoperable product between various social network platforms. No central element exists which can form a performance bottleneck or point-of-failure. Our zero-server architecture offers various networking primitives. These serve as the basic building blocks for creating full fledged Facebook alternative with full interoperability and data portability. We have not created a full alternative to Facebook within the context of a single master thesis. However, we provide the basis for it and claim that our infrastructure will scale to usage by the 2 billion existing Facebook users without fundamental alterations. Our Internet-deployed network primitives are:

  1. Identity layer with public key of your friends
  2. Secure end-to-end encrypted communication bypassing NAT boxes and other imperfect hardware
  3. Always connected to your friend using social-based zero-server overlay with automated Internet address discovery based on a UDP-puncturing capable DHT
  4. Secure messaging of any message size
  5. Distributed bookkeeping with tamper-resilience using Trustchain to prevent spam and abuse

Our zero-server architecture is also meticulously designed to offer another unique property: resilience against legal attacks. Legal action may be expected when a monopoly is attacked. We believe that this thesis proves the first proof-of-principle of a disruptive open innovation ecosystem which is resilient against attacks by disrupted or disintermediated corporate entities. Possible legal attacks includes: legal intimidation, lawyer-based attacks and creation of chilling effects. Our unique approach to governance gives ownership to both nobody and everybody. By creating an academically pure form of self-organisation we create a leaderless infrastructure in which even the original creators have no special powers. It is difficult to block, disrupt or shutdown our zero-server approach with either legal or technological means when used by citizens at scale. On top of our networking primitives we facilitate an open innovation ecosystem. Every user can extend the functionality of the platform freely, called "mini-apps". No gatekeepers exists to moderate mini-apps, this relies on self-moderation by the community. We conducted an experiment around the mini-apps to determine how easy they are to develop. A student development team with 17 members successfully created various mini-apps with our zero-server architecture. Developed mini-apps within a 10-week period include: group creation and shared ownership of money, secure online quorum-based voting, decentralised machine learning, and sharing of music recording.

MattSkala commented 4 years ago
synctext commented 4 years ago
MattSkala commented 4 years ago
synctext commented 4 years ago
synctext commented 4 years ago

Near final thesis .PDF of 60 pages

In general, this thesis does not claim any of the scientific breakthroughs that are achieved. Its really solid work and important step towards Big Tech alternatives.

Remarks:

ElBrujo commented 4 years ago

I'm pretty impressed, I'll love to follow your steps and checked all the code and notations.

This thesis has been deployed to make successful technology, I think, could implement a low-cost repeater, I figured that like a patrol of drones making surveillance in a way of "machine learning distributed network" like a microsatellite, I think we will put some low-cost hardware with comunicatión by laser to extend a network across larger distances... with all the respect what all of the people in this project deserves, I want to implement some of these big tech stuff you are making for real. Thanks and now you are making a decade of investment and deployment a successful protocol, library, and framework, congrats!

synctext commented 4 years ago

Technology Stack for Decentralized Mobile Services

THESIS FINISHED. Direct download of raw .PDF with 59 pages of text

btw Seems the idea itself of P2P apps got already introduced by Stanford in 2012, but never realised. Until Matt did it!

Musubi attempts to create a social mobile INTERNET where users can have a rich and carefree social
sharing experience with any group of friends, with the freedom to choose their own cloud storage provider
and apps on their mobile devices. This open architecture can disrupt the lock-in seen in existing social
INTRANETS which imposes rigid control over both the users and the developers.

https://mobisocial.stanford.edu/papers/musubi_disintermediated.pdf