Documentation: Tribler, IPv8, and Trustchain developers portal

[synctext]

The Developers Doc Portal

Goal: enable starting developers an immediate broad grasp of Tribler Approach: single webpage with pointers to all documentation, low-maintenance, no sub-pages.

• Tribler basic functionality with screenshots: http://jenkins.tribler.org/jenkins/job/Test_tribler_full-ui-run_61x/lastBuild/
• Tribler architecture: modules and their functions
• Database structure: Tribler tables, Dispersy core and Dispersy communities
• Tribler API docs: Tribler download core, Libswift and Dispersy core
• Continous build system: Jenkins performance, unit test, and system tests
• Tribler, Dispersy and Libswift testing framework: DAS4 tests (Effort / Allchannels)
• Dispersy wire protocols: Walker, Allchannels, Search, BarterCast3 https://github.com/Tribler/dispersy/blob/master/doc/wireprotocol_2.x SHA: b8eab4fb501213ae151372cd83a6131582e2ad6b
• Dispersy access control, security and certificate validation
• Dispersy core: event handling, internal data structures and fault behavior

How much can we autogenerate? (zero-maintenance..)

[boudewijn-tribler]

As for the Database structure, we can easily create diagrams from the .schema data using:

sudo apt-get install sqlfairy
sqlite dispersy.db .schema > dispersy.sql
sqlt-diagram --color -f SQLite -o dispersy1.png dispersy.sql
sqlt-graph --color -f SQLite -t png -o dispersy2.png dispersy.sql --show-datatypes --show-constraints

Unfortunately this does not include comments. Nor is the parser able to understand the Tribler database, but I suspect a bit of database cleanup should be able to fix this. Regardless, dumping the .schema data every night can give us up-to-date database documentation.

[NielsZeilemaker]

@synctext seems like a great addition to the new website.

[synctext]

Historical Dispersy stuff:

 tribler.sql sadly does not parse. It gives this error: ``` Error: translate: Error with parser 'SQL::Translator::Parser::SQLite': Unable to parse line 32: '' 'CREATE TABLE MyInfo ( ' entry PRIMARY KEY, ' value text ');' ``` Full listing: ``` sql CREATE TABLE MetadataMessage ( message_id INTEGER PRIMARY KEY AUTOINCREMENT, dispersy_id INTEGER NOT NULL, this_global_time INTEGER NOT NULL, this_mid TEXT NOT NULL, infohash TEXT NOT NULL, roothash TEXT, previous_mid TEXT, previous_global_time INTEGER ); CREATE TABLE MetadataData ( message_id INTEGER, data_key TEXT NOT NULL, data_value INTEGER, FOREIGN KEY (message_id) REFERENCES MetadataMessage(message_id) ON DELETE CASCADE ); CREATE TABLE BarterCast ( peer_id_from integer, peer_id_to integer, downloaded numeric, uploaded numeric, last_seen numeric, value numeric ); CREATE UNIQUE INDEX bartercast_idx ON BarterCast (peer_id_from, peer_id_to); CREATE TABLE Category ( category_id integer PRIMARY KEY NOT NULL, name text NOT NULL, description text ); CREATE TABLE MyInfo ( entry PRIMARY KEY, value text ); CREATE TABLE MyPreference ( torrent_id integer PRIMARY KEY NOT NULL, destination_path text NOT NULL, progress numeric, creation_time integer NOT NULL, -- V2: Patch for BuddyCast 4 click_position INTEGER DEFAULT -1, reranking_strategy INTEGER DEFAULT -1 ); CREATE TABLE Peer ( peer_id integer PRIMARY KEY AUTOINCREMENT NOT NULL, permid text NOT NULL, name text, thumbnail text ); CREATE UNIQUE INDEX permid_idx ON Peer (permid); CREATE TABLE Torrent ( torrent_id integer PRIMARY KEY AUTOINCREMENT NOT NULL, infohash text NOT NULL, name text, torrent_file_name text, length integer, creation_date integer, num_files integer, thumbnail integer, insert_time numeric, secret integer, relevance numeric DEFAULT 0, source_id integer, category_id integer, status_id integer DEFAULT 0, num_seeders integer, num_leechers integer, comment text, dispersy_id integer, swift_hash text, swift_torrent_hash text, last_tracker_check integer DEFAULT 0, tracker_check_retries integer DEFAULT 0, next_tracker_check integer DEFAULT 0 ); CREATE UNIQUE INDEX infohash_idx ON Torrent (infohash); CREATE UNIQUE INDEX Torrent_swift_torrent_hash_idx ON Torrent (swift_torrent_hash); CREATE TABLE TorrentSource ( source_id integer PRIMARY KEY NOT NULL, name text NOT NULL, description text ); CREATE UNIQUE INDEX torrent_source_idx ON TorrentSource (name); CREATE TABLE TorrentStatus ( status_id integer PRIMARY KEY NOT NULL, name text NOT NULL, description text ); CREATE TABLE TrackerInfo ( tracker_id integer PRIMARY KEY AUTOINCREMENT, tracker text UNIQUE NOT NULL, last_check numeric DEFAULT 0, failures integer DEFAULT 0, is_alive integer DEFAULT 1 ); CREATE TABLE TorrentTrackerMapping ( torrent_id integer NOT NULL, tracker_id integer NOT NULL, FOREIGN KEY (torrent_id) REFERENCES Torrent(torrent_id), FOREIGN KEY (tracker_id) REFERENCES TrackerInfo(tracker_id), PRIMARY KEY (torrent_id, tracker_id) ); CREATE VIEW CollectedTorrent AS SELECT * FROM Torrent WHERE torrent_file_name IS NOT NULL; CREATE TABLE UserEventLog ( timestamp numeric, type integer, message text ); CREATE TABLE BundlerPreference ( query text PRIMARY KEY, bundle_mode integer ); CREATE TABLE _Channels ( id integer PRIMARY KEY ASC, dispersy_cid text, peer_id integer, name text NOT NULL, description text, modified integer DEFAULT (strftime('%s','now')), inserted integer DEFAULT (strftime('%s','now')), deleted_at integer, nr_torrents integer DEFAULT 0, nr_spam integer DEFAULT 0, nr_favorite integer DEFAULT 0 ); CREATE VIEW Channels AS SELECT * FROM _Channels WHERE deleted_at IS NULL; CREATE TABLE _ChannelTorrents ( id integer PRIMARY KEY ASC, dispersy_id integer, torrent_id integer NOT NULL, channel_id integer NOT NULL, peer_id integer, name text, description text, time_stamp integer, modified integer DEFAULT (strftime('%s','now')), inserted integer DEFAULT (strftime('%s','now')), deleted_at integer, FOREIGN KEY (channel_id) REFERENCES Channels(id) ON DELETE CASCADE ); CREATE VIEW ChannelTorrents AS SELECT * FROM _ChannelTorrents WHERE deleted_at IS NULL; CREATE INDEX TorChannelIndex ON _ChannelTorrents(channel_id); CREATE INDEX ChannelTorIndex ON _ChannelTorrents(torrent_id); CREATE INDEX ChannelTorChanIndex ON _ChannelTorrents(torrent_id, channel_id); CREATE TABLE _Playlists ( id integer PRIMARY KEY ASC, channel_id integer NOT NULL, dispersy_id integer NOT NULL, peer_id integer, playlist_id integer, name text NOT NULL, description text, modified integer DEFAULT (strftime('%s','now')), inserted integer DEFAULT (strftime('%s','now')), deleted_at integer, UNIQUE (dispersy_id), FOREIGN KEY (channel_id) REFERENCES Channels(id) ON DELETE CASCADE ); CREATE VIEW Playlists AS SELECT * FROM _Playlists WHERE deleted_at IS NULL; CREATE INDEX PlayChannelIndex ON _Playlists(channel_id); CREATE TABLE _PlaylistTorrents ( id integer PRIMARY KEY ASC, dispersy_id integer NOT NULL, peer_id integer, playlist_id integer, channeltorrent_id integer, deleted_at integer, FOREIGN KEY (playlist_id) REFERENCES Playlists(id) ON DELETE CASCADE, FOREIGN KEY (channeltorrent_id) REFERENCES ChannelTorrents(id) ON DELETE CASCADE ); CREATE VIEW PlaylistTorrents AS SELECT * FROM _PlaylistTorrents WHERE deleted_at IS NULL; CREATE INDEX PlayTorrentIndex ON _PlaylistTorrents(playlist_id); CREATE TABLE _Comments ( id integer PRIMARY KEY ASC, dispersy_id integer NOT NULL, peer_id integer, channel_id integer NOT NULL, comment text NOT NULL, reply_to_id integer, reply_after_id integer, time_stamp integer, inserted integer DEFAULT (strftime('%s','now')), deleted_at integer, UNIQUE (dispersy_id), FOREIGN KEY (channel_id) REFERENCES Channels(id) ON DELETE CASCADE ); CREATE VIEW Comments AS SELECT * FROM _Comments WHERE deleted_at IS NULL; CREATE INDEX ComChannelIndex ON _Comments(channel_id); CREATE TABLE CommentPlaylist ( comment_id integer, playlist_id integer, PRIMARY KEY (comment_id,playlist_id), FOREIGN KEY (playlist_id) REFERENCES Playlists(id) ON DELETE CASCADE FOREIGN KEY (comment_id) REFERENCES Comments(id) ON DELETE CASCADE ); CREATE INDEX CoPlaylistIndex ON CommentPlaylist(playlist_id); CREATE TABLE CommentTorrent ( comment_id integer, channeltorrent_id integer, PRIMARY KEY (comment_id, channeltorrent_id), FOREIGN KEY (comment_id) REFERENCES Comments(id) ON DELETE CASCADE FOREIGN KEY (channeltorrent_id) REFERENCES ChannelTorrents(id) ON DELETE CASCADE ); CREATE INDEX CoTorrentIndex ON CommentTorrent(channeltorrent_id); CREATE TABLE _Moderations ( id integer PRIMARY KEY ASC, dispersy_id integer NOT NULL, channel_id integer NOT NULL, peer_id integer, severity integer NOT NULL DEFAULT (0), message text NOT NULL, cause integer NOT NULL, by_peer_id integer, time_stamp integer NOT NULL, inserted integer DEFAULT (strftime('%s','now')), deleted_at integer, UNIQUE (dispersy_id), FOREIGN KEY (channel_id) REFERENCES Channels(id) ON DELETE CASCADE ); CREATE VIEW Moderations AS SELECT * FROM _Moderations WHERE deleted_at IS NULL; CREATE INDEX MoChannelIndex ON _Moderations(channel_id); CREATE TABLE _ChannelMetaData ( id integer PRIMARY KEY ASC, dispersy_id integer NOT NULL, channel_id integer NOT NULL, peer_id integer, type_id integer NOT NULL, value text NOT NULL, prev_modification integer, prev_global_time integer, time_stamp integer NOT NULL, inserted integer DEFAULT (strftime('%s','now')), deleted_at integer, UNIQUE (dispersy_id), FOREIGN KEY (type_id) REFERENCES MetaDataTypes(id) ON DELETE CASCADE ); CREATE VIEW ChannelMetaData AS SELECT * FROM _ChannelMetaData WHERE deleted_at IS NULL; CREATE TABLE MetaDataTypes ( id integer PRIMARY KEY ASC, name text NOT NULL, type text NOT NULL DEFAULT('text') ); CREATE TABLE MetaDataTorrent ( metadata_id integer, channeltorrent_id integer, PRIMARY KEY (metadata_id, channeltorrent_id), FOREIGN KEY (metadata_id) REFERENCES ChannelMetaData(id) ON DELETE CASCADE FOREIGN KEY (channeltorrent_id) REFERENCES ChannelTorrents(id) ON DELETE CASCADE ); CREATE INDEX MeTorrentIndex ON MetaDataTorrent(channeltorrent_id); CREATE TABLE MetaDataPlaylist ( metadata_id integer, playlist_id integer, PRIMARY KEY (metadata_id,playlist_id), FOREIGN KEY (playlist_id) REFERENCES Playlists(id) ON DELETE CASCADE FOREIGN KEY (metadata_id) REFERENCES ChannelMetaData(id) ON DELETE CASCADE ); CREATE INDEX MePlaylistIndex ON MetaDataPlaylist(playlist_id); CREATE TABLE _ChannelVotes ( channel_id integer, voter_id integer, dispersy_id integer, vote integer, time_stamp integer, deleted_at integer, PRIMARY KEY (channel_id, voter_id) ); CREATE VIEW ChannelVotes AS SELECT * FROM _ChannelVotes WHERE deleted_at IS NULL; CREATE INDEX ChaVotIndex ON _ChannelVotes(channel_id); CREATE INDEX VotChaIndex ON _ChannelVotes(voter_id); CREATE TABLE TorrentFiles ( torrent_id integer NOT NULL, path text NOT NULL, length integer NOT NULL, PRIMARY KEY (torrent_id, path) ); CREATE INDEX TorFileIndex ON TorrentFiles(torrent_id); CREATE TABLE TorrentCollecting ( torrent_id integer NOT NULL, source text NOT NULL, PRIMARY KEY (torrent_id, source) ); CREATE INDEX TorColIndex ON TorrentCollecting(torrent_id); CREATE TABLE _TorrentMarkings ( dispersy_id integer NOT NULL, channeltorrent_id integer NOT NULL, peer_id integer, global_time integer, type text NOT NULL, time_stamp integer NOT NULL, deleted_at integer, UNIQUE (dispersy_id), PRIMARY KEY (channeltorrent_id, peer_id) ); CREATE VIEW TorrentMarkings AS SELECT * FROM _TorrentMarkings WHERE deleted_at IS NULL; CREATE INDEX TorMarkIndex ON _TorrentMarkings(channeltorrent_id); CREATE VIRTUAL TABLE FullTextIndex USING fts3(swarmname, filenames, fileextensions); CREATE TABLE 'FullTextIndex_content'(docid INTEGER PRIMARY KEY, 'c0swarmname', 'c1filenames', 'c2fileextensions'); CREATE TABLE 'FullTextIndex_segments'(blockid INTEGER PRIMARY KEY, block BLOB); CREATE TABLE 'FullTextIndex_segdir'(level INTEGER,idx INTEGER,start_block INTEGER,leaves_end_block INTEGER,end_block INTEGER,root BLOB,PRIMARY KEY(level, idx)); ``` <\details>

[devos50]

Actually, this is partially implemented now: http://tribler.readthedocs.org and http://dispersy.readthedocs.org.

[synctext]

Great documentation example: https://www.toptal.com/python/beginners-guide-to-concurrency-and-parallelism-in-python Really cool stuff.. { Remember, please fix issue "1", The Token Economy in 2018, then this fancy stuff for 2019. But I agree we need to think and plan ahead; just no overengineering. }

[synctext]

The cool kids at the Javascript corner make quite nice documentation: https://github.com/amark/gun/blob/master/README.md

We need a GUI .gif demo and tutorial + Dev pointers.

[synctext]

We need to visualise the Trustchain.

Create a movie out of it growing. Cool tool to use: https://asciinema.org/a/35597

[synctext]

Revisiting #52 from 2003. Mostly duplicate issue from 2017: #2928. After over 14 years of effort, all of our initial research code has been replaced with improved code or even production-quality code. In the near future we need to start documenting {more}. Could we find somebody within this universe who wants to bug-hunt, code cover, and document full-time?

Another great example of online documentation: https://wyag.thb.lt/ This is a complete explanation of GIT by creating a fully compatible client. It consists of only 588 lines of very simple Python code. A IPv8 / Trustchain stack with minimal features in tutorial form would be amazing. For instance, a crawler written from scratch. Idea for a BEP project? Minimal viable Trustchain agent, fully stand-alone, without any non-default imports.

[synctext]

Documentation of related work. We need to be aware of the diversity of initiatives in our area. For instance, with Clio, you can develop modules and host them, so you can import them via network from another server. You can essentially build microservices as clio modules and build up decentralized systems. All NodeJS and JSON based, leaning firmly towards minimizing developers time, not raw performance. IPv8 communities need the same abstract API docs and tooling.
https://docs.clio-lang.org/learn/official-tutorial/hosting

[synctext]

Documentation of related IPv8 work. Content for Phd thesis intro? @qstokkink

Governance entity	Delft University	Protocol Labs	real-logic.github.io
Top layer	Tribler/VLC	Filecoin	Agrona data struct Lib for finance
Middle	Trustchain/Libtorrent	IPFS	Simple Binary Encoding for any format support in financial markets
Networking	IPv8 {modules}	Libp2p pub/sub, STUN/TURN, I2P,	Aeron efficient reliable UDP unicast, multicast with low latency and high throughput

[synctext]

blockchain-building framework

https://polkadot.network/technology/ Another full-stack, but almost really works now approach. Stunning documentation! Includes all our main ideas of parallel interlinked chains. But then explained clearly. Quite complex approach. Plus identity and file sharing incentives. Well know solution and used by several teams.

[synctext]

Related work: "Azimuth is a general-purpose public-key infrastructure (PKI) on the Ethereum blockchain, used as a decentralized ledger for what are known as Urbit identities, or simply identities."