Change internal architecture of storing data

[donhardman]

We should migrate to a system where we store each organization in a separate table. This approach will allow us to maintain the logic for searching one, two, or multiple repositories within a single organization. It will also help us narrow down query suggestions to be more relevant to each specific organization.

To start, we should update the code first. Once that's done, we can migrate the data by duplicating it and removing everything that doesn't belong to the organization.

[donhardman]

Implementing: https://github.com/manticoresoftware/manticore-github-issue-search/pull/26

Looking for the best way to migrate production database

[donhardman]

Fixes TODO:

• Update link to point to the correct repository in the "searching in ..." line
• Resolve issue with showcase repository
• Implement CPU allocation for indexation process
• Upgrade backup version in assets

[donhardman]

The problem with registryurl can't be fixed in our current setup. The reason? It's in a different repository, but we're searching in another one. We still suggest it because we now store data in the organization's repo.

[sanikolaev]

It's in a different repository, but we're searching in another one. We still suggest it because we now store data in the organization's repo

The context is that we now have a table per-organization to be able to do CALL KEYWORDS/SUGGEST easier. But what if we change the schema to per-repo tables, combine them into per-org distributed tables and then what will be left is to update select fuzzy=1 and call autocomplete, so that we run call suggest to each table separately since unlike call keywords, call suggest can't do it:

mysql> drop table if exists t; create table t(f text) min_infix_len='2'; insert into t values(1, 'abcdef'); drop table if exists t2; create table t2(f text) min_infix_len='2'; insert into t2 values(2, 'abcdefghi'); drop table if exists d; create table d type='distributed' local='t' local='t2'; call suggest('abdef', 'd');
--------------
drop table if exists t
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
create table t(f text) min_infix_len='2'
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into t values(1, 'abcdef')
--------------

Query OK, 1 row affected (0.00 sec)

--------------
drop table if exists t2
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
create table t2(f text) min_infix_len='2'
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into t2 values(2, 'abcdefghi')
--------------

Query OK, 1 row affected (0.00 sec)

--------------
drop table if exists d
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
create table d type='distributed' local='t' local='t2'
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
call suggest('abdef', 'd')
--------------

ERROR 1064 (42000): no such table d

[donhardman]

This task is done, merged into the main branch, and deployed.

As a result of splitting into table-per-repo, I created this task: https://github.com/manticoresoftware/manticore-github-issue-search/issues/27