infolab-csail / wikipedia-mirror

Makefiles that will download and setup a local wikipedia instance.
1 stars 2 forks source link

** Building everything Basically all you want to do is load some sql files into an sql database and let the dependency tree find it's way into doing that. So if all goes well you should be able to have everything set up with:

+BEGIN_EXAMPLE

make sql-load-parts ROOT_DIR=/new/path/of/proj/root

+END_EXAMPLE

Make sure the media on which ROOT_DIR is mounted has more than 150G and is preferably not a network fs or it will be slow as hell.

*** Troubleshooting Wikipedia is known to put some strangeness into utf that mwdumper does not handle well. If the make process fails you can fix the file that failed with

#+BEGIN_EXAMPLE
make fix-<file-that-failed>
#+END_EXAMPLE

which is an alias to

#+BEGIN_EXAMPLE
make <filename that failed without .bz2 extension>.fix.xml
#+END_EXAMPLE

This will extract the xml and replace all invalid characters with
spaces. Then rerun =sql-load-ports=

*Note:* you should still define =ROOT_DIR= when running make in
this way if you are building outside of the project dir.

** Installing Bitnami MediaWiki Downloads and installs bitnami

+BEGIN_EXAMPLE

make bmw-install

+END_EXAMPLE

To have the downloaded stuff and the install dir in another path use

+BEGIN_EXAMPLE

make bmw-install ROOT_DIR=/new/path/of/proj/root

+END_EXAMPLE

NOTE: =ROOT_DIR= is not persistent. The default is the current dir and to run other commands based on an installed env keep redefining ROOT_DIR.

** Uninstalling Bitnami MediaWiki

+BEGIN_EXAMPLE

make bmw-uninstall

+END_EXAMPLE

** Running Bitnami This will run apache and sql etc. To do any db operation be sure that it is running.

+BEGIN_EXAMPLE

make bmw-run

+END_EXAMPLE

This kills (a bit blindly) anything that resembles bitnami if it fails to halt gracefully.

+BEGIN_EXAMPLE

make bmw-kill

+END_EXAMPLE

** Mysql A mysql shell interface to the database.

+BEGIN_EXAMPLE

make mysql-shell

+END_EXAMPLE