search

FHNW-IVGI / Geoharvester

NDGI Project Geoharvester
10 stars 1 forks source link

readme

Scraper

NDGI Project Geoharvester

A scraper, search engine and data portal for Swiss Geoservices (WMS, WFS, WFTS).

Add YOUR data to the project.

[!Tip]

We scrape data based on the URLs in this source.csv file. If you have additional services that should be included or know of servers/urls that are no longer valid, please let us know. Either:

Stack & Deployment

Stack Diagram

Frontend:

Requirements:
Run:
  1. cd into frontend folder ("geoharvester_client")
  2. run npm i to install dependencies (from package.json)
  3. run npm start to start the fronted on localhost (npm start is defined in package.json)

Backend / Database:

Requirements:
Run:
  1. cd into server folder
  2. Run docker compose --env-file ./app/redis/redis.env up --build (this takes a while for the first build). Make sure to use docker compose not the (soon to be) depricated docker-compose
  3. Check localhost:8000/apiin your browser to verify that backend is running

Troubleshooting:

Expand ##### Cannot start Docker from terminal - Error `Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?` Start docker process with `sudo service docker start` ##### Cannot start application - Check that you are starting the backend from the `server` folder (not server/apps). Is Docker running? You might need to start the daemon (Ubuntu: `sudo service docker start`) or Docker Desktop (Windows) - Make sure that you have created the two files that contain the redis password, as described above. - Use `docker compose --env-file ./app/redis/redis.env up --build` (or another path to where the redis password is located). ##### NGINX not (re)starting This can happen if the server became unresponsive when building the frontend. The eror iss something like `Job for nginx.service failed because the control process exited with error code.` without any more info. Run `sudo killall apache2` to kill any process still running and blocking the restart of NGING, then run `sudo service nginx restart` to restart NGINX. ##### Error about reaching max_clients and no results in the frontend. Redis allows a limit of 10000 by default and will throw an error if the limit is set to a higher number in code. In order to allow a higher limit, this needs to be set in the redis .conf file and the modified config needs to be copied to the docker instance (which is handled by docker-compose.yml, "volumes" ). Redis v.7.0+ required. #### Development / VSCode Support: Docker is set up to automatically copy code changes into the container. However, when it comes to the Python interpreter and the management of dependencies on your local machine both (Docker and your local environment) are not in sync by default. VSCode might flag missing dependencies on your local environment, depending on which interpreter is selected. There are two approaches to solve this issue for development: a) You can either set up a venv and install the dependencies from requirements.txt (cd into server/app, then run `python -m venv env && source ./env/bin/activate && pip install -r requirements.txt`), then point the Python interpreter of VSCode to it. Make sure to rerun `pip install` if you make changes to the requirements file. b) For a "single source of truth" approach, install the "dev containers" extension for VSCode (https://code.visualstudio.com/docs/devcontainers/containers), then attach to the container (https://code.visualstudio.com/docs/devcontainers/attach-container). Windows user: This requires Docker Desktop with WSL 2 set up.

API Documentation

SwaggerUI

Fast API comes with Swagger UI preinstalled. If you have the backend running (see steps above), Swagger UI is available on http://localhost:8000/api/docs. See the wiki pages of this repo for the documentation of this project.