synctext
commented
3 years ago As a general comment we need to talk together more in the Dev Meetings about what it means to focus on 1 million users. Are we missing features, stability tooling, polish, marketing, or user interface that we lack users after 15 years and 8 month of trying?
TCP-over-IPv8 I would like to push it to the main IPv8 repository to enable native transparent sending of messages of arbitrary length through IPv8.
This is a fascinating must-have feature in the long-term. Introducing and debugging several new components is hard. Lets focus first on arbitrary length transfers and thumbnails in channels 2.0. Please first finish exploring markdown/hypertext prototyping, then start releasing the first must-have component: arbitrary length transfers + thumbnails. Then focus on crowdsourcing and deploy a pull request mechanism in Tribler for thumbnails. Then subtitles.
Channels 3.0 community
Please keep this pull request and changes as small as possible. Exploratory understanding of markdown/hypertext possible feature is an isolated project. Please keep this separate from improving existing channels 2.0 in terms of scalability, speed and general usability.
By adopting this architecture, we will bring all the wealth of Web technologies into Tribler. It will be like jumping directly from times of Gopher, FTP and FidoNet straight into Web 2.0 territory.
Sorry, we are not a web project, apologies that this somehow got miscommunicated. The Tiddlywiki stuff is impressive, especially the dedicated community that makes plugins. However, that is not what Tribler is about. As a university we can sell DAO stuff we have operational, the time of web stuff has passed. Our engineering is about boring July 2001 stuff: finding and sharing swarms. Plus a ledger that scales. Please keep markdown in scope, as something more light. Just a gentle reminder that a majority of Tribler developers in 2020 discussion voted for markdown, instead of favouring an embedded web browser.
Let pick Friday 19 February as the focus point for a decision with the Tribler team. Try to have a Mac/Deb/Win installer by then on the Tribler discussion forum. Everybody can then test before our 15:00 meeting and give their opinion on next steps.
The process that has been followed is exploratory prototyping and zooming in on TiddlyWiki early on. Nothing wrong with that. Before we deploy anything in Tribler I think it would be wise to also spend an equal amount of time on an alternative markdown-based approach. Measure their CPU and memory usage, check code maturity, and quantify their invasiveness. Then we discuss the way forward.
Latest development: Solidity instead of Javascript? Solidity works now on top of IPv8 in experimental setting, using https://github.com/eth-brownie/brownie :medal_sports: Note: FBase Python based plugins with source code inspection and security rating is still in-scope for 2021+ https://repository.tudelft.nl/islandora/object/uuid%3Ad68197ec-50b9-4452-b9bf-e34a743f165f
Scalability: if hundreds of people view a markdown/hypertext page it should still work. What is their seeding incentive? This is always a primary design concern, but not yet mentioned in the initial design sketch above. This prototype work made me realise the implicit requirement that we have in Tribler about scalability. The latest buzzword for what we termed "unbounded scalability" is hyperscale architecture. One use-case which is now already implemented for our Superapp is around music and cover art. Lot of legal Creative Commons content exists, like a million songs. Key use-case for markdown/hypertext is having 200 cover art albums in a single page along with their magnet links for direct playback. Running code from our Superapp on Android Play Store:
ichorid
devos50
qstokkink
Successful experiments with TCP-over-IPv8, QWebView, TiddlyWiki synchronization allows us to build a prototype of the Channels 3.0 system.
The Goal
By adopting this architecture, we will bring all the wealth of Web technologies into Tribler. It will be like jumping directly from times of Gopher, FTP and FidoNet straight into Web 2.0 territory. :goat: :arrow_right: :rocket:
Architecture components
TiddlyWiki
TiddlyWiki is a one-of-a-kind wiki/application engine that runs entirely in the browser. Most important for us, TiddlyWiki is a microcontent system based on elements called "tiddlers". In TiddlyWiki everything is a tiddler: content entries, pictures, plugins and even the code itself. Every system tiddler can be "shadowed" by a user-provided tiddler, akin to CSS system. This makes TiddlyWiki almost infinitely malleable, enabling a vast library of plugins, CSS styles, etc. (People even create webshops with it). Also, TiddlyWiki has a clear system for backend synchronization.
These two architectural features (backend-free and microcontent) make TiddlyWiki a perfect choice for Channels 3.0 frontend.
TCP-over-IPv8
To enable sending tiddlers of arbitrary size, we have to employ some transport protocol over IPv8. To make stuff as simple as possible, I copied the code for a pedagogical implementation of TCP in pure Python. Initially, I hoped to use the
aioquicpackage, but this proved to be an overkill. I preferred the TCP thing over @qstokkink 's uTP implementation, because:The code is untested (except by manual test runs). I would like to push it to the main IPv8 repository to enable native transparent sending of messages of arbitrary length through IPv8. I would be glad if @qstokkink helps me with this :wink: UPDATE: done with EVA protocol
QWebView
PyQt allows us to use QWebView widget to, basically, create a platform-independent, Chrome-based browser window inside our GUI. The networking stuff is handled by QT. In regards to XSS attacks, this will be perfectly safe for the user, because it will not use the user's native system browser, hence perfect isolation. Each channel will be served by a separate instance of QWebView.
Libtorrent CDN
Common immutable data, such as the basic TiddlyWiki HTML page, common picture collections, etc, will be transported via torrents, the way we do in Channels 2.0. BitTorrent 2.0 torrent format assigns individual hashes to individual files. This feature allows us to use BitTorrent 2.0 as a CDN to search and host immutable common data. Mutable data will be replicated and gossiped directly with Channels 3.0.
Channels 3.0 community
C3 community purpose is to host and serve mutable data (e.g. forums, wiki pages), as indicated in https://github.com/Tribler/tribler/discussions/5721 . Initially, it will not feature any collective editing. In fact, it will be served as a special sub-type of Channels 2.0 channel, using the same top-level navigation elements in the GUI.
Persistence DB
C3 will use PonyORM as the ORM of choice. Though the semantics will be different, and it will only use disk-based DB as a persistent cache. The idea of C3 is to serve everything from memory, but cache everything and dump the cache to disk periodically. This way, we completely avoid our DB-locking problem.
The Roadmap
Guys! This stuff will require months of work if I do it alone. Fortunately, the work can be split up very nicely into smaller architectural parts. So, if anyone volunteers to help me with this - I can always find a part that will be fun enough for you to make :wink: