Closed ichorid closed 9 months ago
Thank you for posting you overall design online, very helpful! This design is needs to be simplified. First deploy something which works. Only then can we take the next itteration. For instance, quantify performance bottlenecks and consider optimisations with "set reconciliation algorithm" or Bloom filters.
Channels Discovery Overlay bootstraps into the network by connecting to some random peers and continues to walk the network randomly. Whenever it meets a new peer, it queries it for channels that this peer serves/subscribes.
Good idea. However, make it simpler with merely 1 message: ChannelDiscovery
. Only purpose is to fill the channels viewing screen. There is no need for queries, no pub/sub. When doing random walk across this community every peer pushes a subset of their subscribed
channels blindly to others or random channels (50/50 probability). However, only promote a channel if they have at least 1 content item of this channel stored locally (non-emptiness protection at protocol level). Included fields: public key, channel name, collected channel items, total number of channel items, last update date, and signature. Anything missing?
Is there documentation of your deployed "channel popularity algorithm" with the 5-stars, please include.
Channels Availability Overlay uses channels availability data
Good idea. However, make it simpler with merely 1 message: RandomChannelContent
and complete randomness. If people leave Tribler on for 1 week it should have receive 1 message per second in this community. That is sufficient to start with for now. Everybody in 1 IPv8 community and everybody hears random stuff. Or are we missing functionality then? This is a deliberate conservative design. Only purpose is to randomly gossip the items within channels. Before blindly sending out random item content, do a 50/50 probability to either send subscribed channels content or non-subscribed content. Included fields: public key of channel, item (torrent hash, swarm-name, swarm size in Bytes, optional "directory" within channel, and signature of channel owner. For swarm with multiple files: something like the 25 filenames of the largest files inside the swarm are included {or less to fit inside UDP packet}. We used to have code for that.
The infinite sub-directories is not something our users are asking for, so it can be removed. This simplified the 3-layer design into a 2-layer design where each items comes with a single "directory" name optionally.
The Remote Query Community sends a request for the DB data to one or more peers known to serve the channel.
That might actually be tricky. How would that work, simple subset of 20 peers in a community? Or are we expanding the connected peers to 200 peers to have a decent probability for a hit? You seem to be thinking of doing an on-demand random walk and keeping the connection alive with peers of interest, peers with overlap in subscribed channels. We called those taste buddies 14 years ago. Are you planning to keep a list of Internet addresses of peers seen in last 10 minutes with channel they know about?
The request can have a TTL of 2 or 3
Query forwarding is something that invites abuse. How would you stop an amplification attack? 1-hop designs have proven sufficient.
Whenever it meets a new peer, it queries it for channels that this peer serves/subscribes. and (probably based on dissimilarity of their Bloom filters).
This is not a bad design, but it seems like you are proposing existing designs without being aware of them. Dispersy first was "extensive query-based", before the developer discovered that a simple blind random push is superior. There is a professor in Amsterdam who wrote publications for 12 years on how to design this gossip overlay stuff. It's advisable to know his findings, but that take several years of study. Protocol design is not something you learn in a few years. Additionally, this design is slowly drifting towards our failed Open2Edit crowdsourcing system.
The random walk and Bloom-filter design you specify there is almost an exact re-creation of a high-performance version of our prior deployed channel system? Please study that design first and indicate what you want to do different and why. We did Bloom filter usage calculations in the past and published it. Please document your design here in .MD format, we do a feedback round with the development team next week, and then implementation may start.
Expensive lessons learned by Dispersy team: Bloom filters have been proven to cut bandwidth usage in half at least. However gossip uses an insignificant amount of bandwidth: 1 Kbyte per second is already a lot. Bloom filters turned out to be fatal design mistake in our prior channel implementation: it introduced a lot of complex and impossible to debug code for a minor bandwidth saving.
Sadly these lessons of failure are not documented. I've started a scientific paper on "myths and misconceptions", design patterns to follow. This was written after 10 years of measurements and experiments with self-organising P2P systems. Period: :drum: :drum: :drum: 2001 - 2011. I'll try to finish these lessons when we have a million users, a single positive data-point.
There are two kinds of information exchange in any information network:
Examples of Type 1 (unknown) info exchange: telling fresh gossip to your friends; sending a message to your bank telling that you want to spend your account money on something, etc. Examples of Type 2 (known) info exchange: asking your friend to recite a poem they memorized earlier, asking your bank to send you details of your bank account state, etc.
Type 1 exchange is naturally served by push-based gossip. Type 2 exchange is naturally served by client-server or DHTs. It is infinitely dumb to try to serve Type 1 requests by client-server architecture, because, as mentioned by @synctext , you'll have to query hundreds and thousands of peers at regular intervals to get some new info. This creates a horrible network traffic amplification antipattern. See Kazaa and DC++ search request flood nightmare. The same is true for the reverse situation: trying to serve Type 2 requests by gossiping around everything and trying to store all possibly required info on every node. See Bitcoin log replay problem and our current Channels with 1M+ entries.
@synctext , to address your concerns:
How would that work, simple subset of 20 peers in a community?
The peer will only query those neighbours that it knows to serve the channel. It knows which node serves what from the exchange of subscription lists that happens on each peer introduction.
Are you planning to keep a list of Internet addresses of peers seen in last 10 minutes with channel they know about?
Not just that, but actively selecting for the most responsive and useful peers, trying to come up with a stable neighbourhood.
Good idea. However, make it simpler with merely 1 message:
ChannelDiscovery
. Good idea. However, make it simpler with merely 1 message:RandomChannelContent
It already works like this since the beginning. No one is going to touch that, I assure you :wink:
Please study that design first and indicate what you want to do different and why.
Dispersy tried to sync everything. It tried to optimize that task using Bloom filters. No amount of optimization will save a design that tries to copy everything around. It is akin to downloading the whole Google index and Wikipedia to your PC, "just in case". It does not matter if you use BitTorrent or direct download, your disk will blow up :boom: Instead, we're going to serve channel contents on-demand, just like the Web does with web sites. And the only thing that is going to be blindly gossiped around is the list of subscribed channels.
Every pushed list serves 3 purposes:
Bloom filters (or something like that) will possibly be used to only exchange subscribed channels lists. These will be very short (10-100 entries) and rarely change. Using knowledge of these subscription lists, every peer will try to stabilise to a neighbourhood that serves it the best.
That is interesting, you meaning building a semantic overlay! That would be quite a step forward for scalability and finding long-tail content. Many tried, everybody failed so far (including us). Please first finish your 4th development iteration your existing design, before diving into research on semantic overlay. Keep iterations small please, split repairs of prior work and new feature Pull Requests. Thus: progress bar, instant-download, faster insert time as requested in a comment above. {repeating}
Semantic overlays are really difficult and unsolved. You can't connect to peers of interest, a design which can cope with 90% blocking by NAT boxes is hard. Rich metadata is something our users seem to want more than long-tail. (btw every peer is already announcing each subscribed channel using the channel seeding+DHT, would be good to try to re-use that in the future semantic overlay iteration)
instant-download, faster insert time as requested in a comment above Please prioritise the above issues and put those in the release, before fixing the "SQLite record insert performance issue".
These two sentences contradict each other. I do not understand what you mean.
Subscribing to a channel works instantly
I do not understand what does it mean either. Subscription is already instant (a single click!), but the download/processing takes a long time and it is going to ever be that way until we switch to on-line channel contents fetching .
Each MByte downloaded on disk is both parsed, inserted and displayed in real-time.
I can't understand this either. What does it mean "displayed in real-time"? Like, showing thousands of entries flying through the Tribler interface each second? This is going to kill the GUI performance and daze the user. Maybe you meant showing the progress of channel download? But that is already in another point of your checklist. I am confused. :confused:
Stall the download until the block is processed
This is akin to using handbrake on your car if you need to go slower, instead of shifting the gear: dumb and wasteful way to do a simple thing. It will severely interfere with the pattern of channel torrent updates and basically kill the availability of newly-created big channels. I don't know why you keep insisting on this technical detail. If you wanted to show the progress of channel processing, that is solved by a progress bar. The only reason to stall the download is to synchronize the channel processing progress bar with the channel torrent progress bar (which the user is not supposed to see at all, and is for debugging purposes only). I don't see any other value in this solution.
you meaning building a semantic overlay!
What I propose has nothing to do with user's preferences, semantics, etc. It is just a simple and convenient scheme of predicting (or, in some sense, pre-caching) user preview queries to the most popular channels. And, coincidentally, improving the quality of remote search.
Using knowledge of these subscription lists, every peer will try to stabilise to a neighbourhood that serves it the best.
Since you build a structured overlay where peers with similar interests (taste?) cluster together, it sounds very similar to a semantic overlay. This concept has been extensively explored by one of our former PhD students.
I've read that work on semantic overlays. Again, I'm not going to connect to peers with similar preferences, I'm going to connect to people who, together, give the best coverage of the most popular channels. To serve previews of these channels instantly.
Recent comments by @drew2a and some cues from how users name their subchannels (e.g. with the same names as big popular channels) point us that the concept of the Channels GUI should be redesigned to resemble Spotify, etc. The whole idea of "Subscribing" to a channel is counterintuitive and should be abandoned. Instead, the user should add the desired channel to "My channels" list. This action will trigger the actual subscription procedure behind the scene.
This redesign idea is one of the reasons why I'm hesitant to do incremental changes to the current GUI Channels design.
the concept of the Channels GUI should be redesigned to resemble Spotify, etc. The whole idea of "Subscribing" to a channel is counterintuitive and should be abandoned.
OK, abandoning the current channel subscription paradigm is quite something. Please elaborate how "My Channels" new logic would work? What do you propose instead of "subscribe" or "like"? We need to have a clear design and end-goal before starting implementation work.
Subscribing to a channel works instantly
I mean no waiting time within the GUI before it's added. Now its takes a long time before a channel subscription is processing and added as a news swarm download within the downloads panel. Channel progress bar is something else then a download bar, it start instantly and is only finished when all information is inserted in local database.
Keyword searches usually takes 3 minutes to resolve. This issue was reported before (using cool IPv8 health monitor). It might be the another components blocking the main reactor and I've posted this to the wrong issue! Possible corner cases caught: typical loyal Tribler user database of 1.9 GByte metadata.db
, numerous Tribler database version upgrade for 2+ years, 9 subscribed channels, and Ubuntu 18.04.3 usage with this Jenkins build. We need to develop infrastructure to catch these bugs automatically. See below what I managed to catch (100% reproducible). First keyword search is fast (sometimes happens), then a 3 minute wait for search results to show (usual case). Suspecting something is blocking the reactor, might even be TorrentChecker. @devos50 has full debug .Tribler info and logs.
Health monitor (blib that show once in a while during keyword search):
This issue seems to be caused by an unresponsive Tribler core. This can be further debugged by closely monitoring the duration of HTTP requests to the core, as reported by the application tester:
There's also @egbertbouman's ipv8/asyncio/tasks
REST endpoint which shows long-running tasks. Maybe also nice to poll and integrate that in the application tester at some point?
Tested @synctext 's DB, was not able to reproduce the problem. It must be something regarding TorrentChecker (it is always TorrentChecker). The solution to this is a properly designed asynchronous threaded priority queue for DB requests.
See #4320
The distribution of the number of subscribed channels per-peer (logarithmic, slightly distorted for visibility, w/o personal channels).
X-axis - the number of channels subscribed by a peer.
Y-axis - the number of peers subscribing exactly X channels.
The data was collected by passively listening for Channels community since GigaChannels inception.
It is easy to see that most people subscribe to 3-15 channels max (1500 peers' data collected).
Human attention span and free time are limited. Each other subscribed channel requires clicking on it to check. These stats resemble the stats for Twitter, Telegram and other social networks: no one ever subscribes to more than a few dozen information sources.
This gives us an excellent opportunity to build an "inverted DHT" overlay to discover services/channels in close vicinity (1-2 hops) by using progressively compressed Bloom filters.
The idea for the new Channels design is described in #5721
There are two kinds of links to information:
The typical media collection contains human-links as intermediate nodes and hash-links as leaves. A good example is that of a file system: humans give names to directories and files, but the files are typically binary data. Also, the interpretation of human-links is subjective. For instance, in a typical "private data collection" scenario a folder with a boring name like work stuff / more work stuff / qweasd
can contain a collection of images not quite related to work...
A hash-link is easy to check and spread in a distributed system. The copies can be easily cross-verified between the peers (e.g. by using a Merkle tree). A human-link can only be verified by another human, and that is subject to human consensus.
Any interaction between a human and information aggregation system (such as Tribler, or Wikipedia, or Instagram, or Facebook, or torrent tracker, or Spotify, etc) boils down to the user finding their way from the root of the index to a satisfying leaf. The process is known in marketing as "the Sales Funnel".
Our "Discovered" tab and Search feature are the roots of two different kinds of indexes:
An unlinked hash-link is useless, as it is unreachable. However, hash-links are often reused (e.g. pictures or videos in a CDN). This means that we need to provide public key signatures only for human-links. Binary data, such as pictures, icons, sounds, etc., should be addressed and cached by hash-link only.
In short, hashes don't lie. People do.
In 2 months it will be the 3 year birthday of the AllChannels big rewrite plans. @ichorid :confetti_ball: :three: :cake: Please focus on including the following features in a Tribler release of these months.
Sprint | Milestone | Description |
---|---|---|
March 2021 | 1 million item | Markdown support and simple skeleton TFTP binary transport inside channels backend |
seedbox | @drew2a Fill this channels with nearly TeraByte of Creative Commons Music | |
Remote search | Determine the keyword search latency performance with lots of music tracks | |
April 2021 | instant usability | Partial seeding of a big channel with many mdblobs. Download 250MByte of random .mdblob files and seed them. This ensures channels can grow in number of items for a while, we nicely export this problem to Libtorrent. Random partial seeders ensure full availability of channel info. |
instant download | Gossip and instant Magnet link. When receiving the latest SHA1 hash of a channel inside the gossip network it is possible to instantly start a Bittorrent download. Currently another step is going on in the background which is unclear to the user. | |
May 2021 | Full markdown | Create a simple markdown viewer inside Tribler. Enable browsing markdown files inside a Tribler channel. |
Wikipedia import | Convert the Mediawiki content of wikipedia into markdown. Create a single static Wikipedia Markdown dump in example Tribler channel. See offline Wikipedia for inspiration. ![]() |
|
June 2021 | Magnet markdown | Support any magnet link with picture, audio or video content. External magnet links to content outside the channel are now supported. Markdown viewer initiated background loading and refreshes when download is complete. Any magnet link embedding and playback should be supported, just like we have Youtube .JPG preview in this example (however, it opens in an external player from Youtube). This step is like building HTML5 inside Tribler using markdown and VLC or some player. (Detailed design still to be discussed; content internal to channel versus external to channel. For now only support content internal to channel!) ![]() |
July 2021 | daily pinning | Every channel update changes the channel torrent's infohash. That will break down quickly :face_with_head_bandage:. Top-level swarm of channel is fixed daily. Real-time updates are send using gossip. All the gossip of a single day is bundled into a new series of .mdblob and everybody subscribed switches 1 time every day to single new top-level hash. This is to ensure we keep as close as possible to 1MByte .mdblobs ![]() |
2021-2026 | Global Brain | Incrementally grow content collection with static example channels. Let the community use our features, we showcase what is technically possible. Grow with more content and broader ecosystem. From importing all know Creative Commons content to all open-access medical and scientific based publishing ![]() |
@synctext , looks like a solid plan! However, I can't understand what you mean by:
Partial seeding of a big channel with many mdblobs. Download 250MByte of random .mdblob files and seed them. This ensures channels can grow in number of items for a while, we nicely export this problem to Libtorrent. Random partial seeders ensure full availability of channel info.
The "partial seeding" thing is especially unclear to me in the context of the "instant usability" milestone. If I understand correctly, you propose seeding some channel torrents speculatively to facilitate channel availability (1) and reduce the "waiting for full channel contents" time (2). Also, to solve the channel scaling problem (3) you propose seeding random subsets of mdblobs from big channels (4).
Now, let's analyse those ideas from the standpoint of the current architecture ("append-only log -> SQLite"):
To sum it up, while seeding top channels speculatively would enhance the user experience, partial seeding will not achieve any positive impact.
Also, to clarify, what did you mean by:
Gossip and instant Magnet link. When receiving the latest SHA1 hash of a channel inside the gossip network it is possible to instantly start a Bittorrent download. Currently another step is going on in the background which is unclear to the user.
Did you mean creating a feature that enables the user to "subscribe" to an actual download, for e.g. some TV series and receive the update automatically?
Sorry for being to short and cryptic. With partial seeding I mean to download a part of all .mdblob files of a single channel. Each file is still download 100%, but you don't have all of them. Libtorrent signals with want/have messages what blocks are available at which peer. Thus combined with PEX messages its clear who is around and who was what pieces.
Next step is inserting the information directly into the local database after download and processing. Each 1MByte .mdblob also can be inserted atomically into the local database. Such "incremental insert" is also a big refactoring I believe of the current code. Does it now make sense?
The problem is, every channel update changes the channel torrent's infohash. Pushing more than a couple of updates a day results in a trail of dead swarms.
Yes, that is an easy problem to solve. We can post-pone till after markdown integration. daily pinning Top-level swarm of channel is fixed daily. Real-time updates are send using gossip. All the gossip of a single day is bundled into a new series of .mdblob and everybody subscribed switches 1 time every day to single new top-level hash. [EDIT of roadmap above, new July and beyond task]
Yes, it makes sense now. Regarding the daily pinning, this will effectively make channel torrents a supplemental way of spreading the channels, targeted at new users. Also, it will still be compatible with Channels 3.0
Each 1MByte .mdblob also can be inserted atomically into the local database.
Well, mdblobs are inserted incrementally at the moment. But the hard requirement is they must be inserted sequentially. Otherwise, we'll have to use CRDTs, which seem to be overkill for the task.
Also, even if we manage to be able to insert the blobs non-sequentially, this would still give almost no speedup (or I don't understand what you propose again). If I understand correctly, you propose downloading a random subset of blobs (that is possible even now) and processing them (this will require something like CRDTs). Now, let's say processing a channel takes 1 hour. A user downloads 5% of the channel's mdblobs in the background and processes them. This takes about 3 minutes of Tribler working in the background, which is fine. However, if the user subscribes to the channel, they would still have to download and process the remaining 95% of the blobs, which will take 57 minutes. So, speculatively preprocessing a tiny fraction of mdblobs makes no sense to me.
ahh, nice! Sequential inserts should be no problem. Minimal changes, excellent. The problem is: how can we best package the entire markdown-based Wikipedia (English,Spanish,German, etc.) into a single channel? How much .mdblobs would we have?
The torrent for multistream (seekable) English Wikipedia articles archive is about 17GB of bz2-compressed data. It is assumed that the full unpacked size is about 80GB. Say, we store the stuff in DB, Brotli-compressed, indexing article titles only. That's about 20GB of mdblobs on disk and about 30GB in the database (the overhead of 10GB is from indexes, titles, signatures, etc). That is 10x our biggest channels experiments and 6x all the channels data out there, combined.
Now come the updates... It is hard to estimate how many pages change daily on English Wikipedia, but the number of daily edits is about 200k, and the page churn is about 1k/day. So, assuming an average of 50 edits per page per day, about 20k pages change daily. That's 600k pages changed per month. Again, I did not find any info on the distribution of page retention time, but from those numbers, we can roughly estimate a monthly change rate of 10% of 6M articles.
So, in the end, the whole English Wikipedia (text only) will take about 60GB on disk. It will take about 40 hours on a fast modern PC to upload it into the database in the current architecture. Wikipedia search results will be shown alongside other channels results. Updating Wikipedia will take about 1-2 hours per month on a fast PC.
ToDo note for future. Seedbox activity. Limit to Max. 2-3 weeks of effort. As a university we want to showcase the strengths of our technology, while minimising engineering time. static dumps only :exclamation: Dynamic updates are too difficult.
Downloading of Arxiv's archive will costs us for about 100$ (based on information from https://github.com/mattbierbaum/arxiv-public-datasets#pdfs-aws-download-only)
Seedbox activity is on pause now.
Channels are permissioned, that needs to change. Brainstorm target if tag experiment is successful: scientific journal without an editor-in-chief. Peer-review driven. Use markdown, picture and infinite amount of PDF attachment with crowd sourcing magic. The exact magic is our university goal: permissionless science. Leaderless science.
Tag terms of both journal and individual PDF files become the -exclusive- terms for discoverability. Human created tag terms dramatically enhance scalability versus all keyword from abstract as search space for all scientific papers. Stepping stone for 'decentral Google' ;-)
Permissionless crowdsourcing is incompatible with Channels 2.0 architecture that is based on torrents. Channels 3.0 architecture fits permissionless perfectly, though.
Early impression of content aggregation:
This is where my user journey died 😄
great stuff :clap: Suggestions for next iteration:
The next prototype we designed:
The number of snippets shown, and the number of torrents per snippets can be configured through global variables (default number of snippets shown: 1, default number of torrents per snippet: 4). We decided to keep the information in the snippet to a minimum. A torrent that is included in a snippet is not presented in another row as a "regular" torrent.
Now that the GUI part of the snippet is mostly figured out, we had a brief discussion on the next step(s).
The upcoming sprint is a good idea to do a deep dive into the Linux query only. So do something that does not scale, manually tune and tweet just a single query and make dedicated regular expressions for it. Work until it is really perfect. Then everybody learned something from actual swarms, turning dirty data into perfect search. Attempt at doing 3 iterations deployed in Oct, Nov and Dec?
PR #7070 contains the first version of the work described on both this ticket and #6214. This is also a key step towards #7064. Below are some of the insights/design decisions during this sprint.
Last sprint we decided to focus on snippets that bundle torrents that describe the same content. This sprint focussed on building the storage backend. We decided to keep our design as simple as possible while not trying to limit ourselves too much for further iterations. We will in Tribler maintain a knowledge graph with content items, torrents and tags. See the visualisation below.
We believe this structure is sufficient enough to improve search results by showing snippets, and for further iterations in which we can improve other aspects of Tribler. In the underlying database, this graph is stored as a collection of statements in the format (subject, predicate, object)
. We took inspiration from existing work in the semantic web, also to establish some language when building new algorithms on top of our infrastructure. For example, a content item - torrent relations is represented as a statement (“Ubuntu 22.04”, “TORRENT”, <infohash>)
. A torrent-tag relation is represented as a statement (<infohash>, “YEAR”, “2013”)
. We have updated various names in the Tribler code base to reflect these changes in knowledge representation.
We do not allow arbitrary predicates in the knowledge graph. To represent knowledge, we use the Dublin Core (DC) Metadata format and integrate it with the knowledge graph visualised above. This format gives us a set of attributes that are used to describe torrents, including year, data and format. The predicate field in each statement can take one of the 15 values given by DC. We also have two special predicate fields, namely TORRENT to describe content item - torrent relations, and TAG for backwards compatibility.
In addition to the above, the tag generator has been updated to automatically generate tags for Ubuntu-related content.
There are a few points we can focus on during the next sprint:
One of the design decisions is whether to add a FTS index to the strings in tuples. This would allow efficient searches in tuples (which I suspect will be a key aspect of querying the knowledge graph) but adds more complexity. We need an experiment to verify the speed-ups.
Automatic Content Item generation (Ubuntu, Debian, Linux Mint):
import re
from re import Pattern
from tribler.core.components.tag.rules.tag_rules_base import Rule, RulesList
space = r'[-\._\s]'
two_digit_version = r'(\d{1,2}(?:\.\d{1,2})?)'
def pattern(linux_distribution: str) -> Pattern:
return re.compile(f'{linux_distribution}{space}*{two_digit_version}', flags=re.IGNORECASE)
content_items_rules: RulesList = [
Rule(patterns=[pattern('ubuntu')],
actions=[lambda s: f'Ubuntu {s}']),
Rule(patterns=[pattern('debian')],
actions=[lambda s: f'Debian {s}']),
Rule(patterns=[re.compile(f'linux{space}*mint{space}*{two_digit_version}', flags=re.IGNORECASE)],
actions=[lambda s: f'Linux Mint {s}']),
]
Just a reminder to deeply understand what it means to be future proof
Date | Tribler version | Description of database of Tribler |
---|---|---|
20 Sep 2005 | initial import | Read the configuration file from disk |
29 June 2011 | v5.3.x | We ship with a full SQL database engine CURRENT_MAIN_DB_VERSION = 7 |
4 Dec 2012 | v6.0 | Lots of upgrades in 1.5 years CURRENT_MAIN_DB_VERSION = 17 |
25 june 2015 | v6.5.2 | complex DB versioning support TRIBLER_65PRE4_DB_VERSION = 28 |
11 June 2016 | v7.0.0 | Implemented upgrade process of FTS3 to FTS4 |
18 June 2020 | v7.5.0 | everything gets changed with gigachannels BETA_DB_VERSIONS = [0, 1, 2, 3, 4, 5] |
28 Oct 2021 | v7.11 | tagging added: 1st metadata crowdsourcing gui_test_mode else "tags.db" |
Lessons we never learned, but endured. 1) use mature libraries 2) REPEAT: avoid the latest cool immature tooling 3) never do a complete re-write 4) have an explicit roadmap with intermediaries goals 5) let the dev team make roadmap to ensure buy-in add realism. 6) what are you optimising exactly? 7) important final point: always remain stable; no SegFaults or technical debt
As we are getting #7070 ready for merge, we made a few additional design decisions:
TORRENT
resource type will be removed and we will use the TITLE
relation instead.Next steps:
Use cases.
We strive to provide the same kind of service that is provided by:
On fuzzy searches.
We do not try to become a distributed Google. Google is good at finding the knowledge that is relevant to words. It is used when the user does not know what he or she is looking for. Our user always knows what he or she wants. Therefore, we should disable metadata (or torrent contents) search by default, and only search by the torrent name. On the other hand, the user should be able to use metadata to refine the search results. Like, "sort by date of creation" or "sort by tag/channel", etc. This model already works perfectly for all use cases indicated above (except Youtube).
Data structurization.
We must be able to replicate the basic information organization structures that are used by our example use cases:
It is important to note that a single instance of the system is not required to serve all of these features at once. Instead, it should provide building blocks, or modules, to compose the use cases as necessary. For an average user a single database could be used to serve every type of content, but, e.g., users interested in building a huge scientific database should be able to set up a separate database instance optimized for scientific articles.
Constraints on design choices
The design should consist of several completely independent parts:
It is very important that we do not make any of these parts depend on specific implementation of one another. We must be able to design the system in such a way, that it would be trivial to exchange one implementation of some part for another one.
User experience requirements:
Implementation requirements: