mysociety / alaveteli

Provide a Freedom of Information request system for your jurisdiction
https://alaveteli.org
Other
385 stars 195 forks source link

Migrate legacy Chert Xapian DBs to Glass #6344

Open garethrees opened 3 years ago

garethrees commented 3 years ago

Potentially relevant here:

Lots of our older installs, including WDTK, use Chert.

Need to list the installs that need migration, and then do them.

sagepe commented 3 years ago

All of them will need migrating. If we're going to be deploying any new Alaveteli installations anytime soon let's get #6208 merged before then so they can start using the Glass backend.

gbp commented 2 years ago

While migrating vhosts to the new infra as part of https://github.com/mysociety/sysadmin/issues/1527

We have been rebuilding the Xapian indexes as glass where possible - eg where is didn't take long to generate from scratch and where there was no errors)

The current state of indexes are as follows:

Country Format
WDTK chert
EU chert
Belgium chert
Colombia chert
Czechia chert
Greece glass
Liberia glass
Nepal glass
Netherlands glass
Rwanda glass
Turkey glass
Uganda glass
sagepe commented 2 years ago

So the indexing process can take time and be quite memory intensive, so I think doing this at deploy during a migration is only really viable when the site is a small one.

We may need to look at building a new glass index offline on fast machine, then swapping into place and updating once done for the larger sites.