d47081
commented
1 year ago Implemented, after creating separated SQLite branch
Initial database structure looks like this (InnoDB engine with foreign keys and transactions support)
upd
Realization has some new features:
- robots.txt files support (#2), related to each domain, where also could be applied default one, provided in config file
- page rank (coming soon)
- pages limit per domain to index
- crawl meta only for limited disk quota hostings
- domain status (enable/disable)
In total, this updates includes website settings customization, speed and data optimization.
More details in README.md or the release version soon.
Database project could be deployed from the MySQL Workbench project located here https://github.com/YGGverse/YGGo/tree/main/database
Just tried to make search request on 2.5M rows on SQLite / FTS5 and seems that we starting to have performance issue.
According to following conversation we need to rewrite current DB driver model. Suppose MySQL is the nice accessible alternative.
I have experience with Sphinx engine, it stores compiled data in RAM and able to process at least 8M rows in milliseconds with same server resources, comparing to the current result.
If some one have better ideas - you are welcome here.