A repo meant to scrape, extract and publish education materials.
This project uses Docker
, docker-compose
, uwsgi
and (optionally) Conda
.
Make sure they are installed on your system before installing the project.
To start you'll need to first setup a local environment on a host machine with:
conda env create -f environment.yml
git rev-parse HEAD > harvester/.commit
Then copy the .env.development
file to .env
and update the variable values to fit your system.
When you're running the project locally you'll only need to provide your Elastic Search credentials.
Now you can activate the conda environment and load environment variables with:
source activate.sh
Inside of the root directory of this repo run the following command:
docker-compose up
This will build and run the backend software in the background.
After this basic installation you'll need to load the data. Get a Postgres dump file and load it with the following command from the root directory:
make import-db backup=<dump-file>
When running the project for the first time one more command is needed to complete the installation process. This command allows you to create your own local superuser for the administration interface.
docker-compose run harvester setup.sh
Now the administrator interface and API are available under:
http://localhost:8000/admin/
http://localhost:8000/api/v1/
Don't forget to stop your Docker containers when you're done working:
docker-compose down
The repo expects to be deployed on a Docker swarm. You'll need to init the swarm:
docker swarm init
Now make sure that the registry.surfedu.nl
repository is available.
Then run the following:
bash start.sh
To update the code it's sufficient to re-run the start command. It can be convenient to run Django commands outside of a Docker container on the server. You can create a venv on the server to achieve this:
python3 -m venv venv
pip install -r harvester/requirements.txt
Then before you run a command to load environment variables and activate the virtual environment run:
source activate.sh
How to harvest learning materials from repositories or other harvesters is described in detail in the harvester documentation.
If you want to know more about the available API please read the API documentation.
More information about the commands we use to analyse the system is in the analysis documentation
Datagrowth is a set of open source tools that helps to gather and organise data through Django. I've included the sources to this repo directly, but I mostly copy pasted it from another repo. When Datagrowth gets released as a package we should consider installing it as a dependency rather than using the copied sources.