Copyright Notice: This repository is licensed under the GNU AGPLv3 license. The basic structure is generated using the FastAPI project generator (MIT license). If you want to do a similar project, you might want to start there.
The HeiGIT disaster portal API was developed to allow the accessing, adding, deleting and editing of features that can be used in requests to HeiGIT services in the form of a portal system between applications or users and services provided by HeiGIT. This portal should act as a single entry place to the various services (openrouteservice, ohsome etc.) as well as allowing additional processing to be performed on either the requests made to these services, or the responses returned from them.
In the initial version, the portal will only communicate with openrouteservice, and will be used for adding avoid areas to requests which will be provided by an external source.
To accomplish this, there will need to be two APIs - one for maintaining the database of features, and one for making requests against the HeiGIT services.
The API for communicating with services should be kept as close to the original requests as possible with minimal alterations and additional parameters.
The avoid-areas functionality should be able to identify features from the database that would be avoid-areas for the request without the user having to explicitly pass these areas in with their request. It should also be able to strip these derived areas from the meta information in the response, where normally the full query passed to openrouteservice is added. In the case of the portal, the request to openrouteservice will contain information not directly specified by the user, and it may be desired (e.g. to reduce network transmission) to remove this information from the response.
Requirements:
# copy .env-sample to .env and adjust settings
cp .env-sample .env
# builds the docker disaster-area-portal image and dap-api container
docker-compose build
# deploy dap-api and dap-db in background
docker-compose up -d
# check api logs
docker logs -f dap-api
docker logs --tail 200 dap-api
# grep docker logs using `2>&1`
docker logs dap-api 2>&1 | grep WARNING
The interactive api documentation can be accessed at http://localhost:8080/api/v1/docs or alternative documentation on http://localhost:8080/api/v1/redoc
Requirements:
# Install poetry
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -
# enable poetry command (check [poetry documentation](https://python-poetry.org/docs/#enable-tab-completion-for-bash-fish-or-zsh) for shell completion setup)
source $HOME/.poetry/env
On some systems where python2 and python3 are installed in parallel (e.g. Ubuntu), above installation method for poetry may lead to problems due to poetry trying to use python 2. In case you see such an error message, try installing poetry via pip:
# Install poetry
pip3 install poetry
Now we can install all dependencies locally.
# install dependencies
poetry install --no-root
When using PyCharm you can use the Poetry plugin to create an interpreter with a python3.10 base or higher. It should use the virtual environment poetry has created. After creating it, you might need to close and open PyCharm again, so the Project uses now the correct interpreter and env. In case pycharm creates a separate virtual env, you could prevent poetry from creating further virtual envs by running:
poetry config virtualenvs.create false
Copy .env-sample
to .env
and adjust settings
cp .env-sample .env
Build the development image
docker-compose -f docker-compose.reload.yml build
This will build an image for 3 containers:
Additionally, you can create a docker-compose run configuration with PyCharm. The advantage of this:
+
(add new configuration) and choose Docker Compose./docker-compose.reload.yml;
dap-api-reload, dap-db, dap-test-db
Create and run development containers (or run the configuration)
docker-compose -f docker-compose.reload.yml up -d
Note: The first time it will take some time until the db containers are configured. The dap-api-reload container won't be able to connect right away.
You can test the service through the interactive docs on http://localhost:8081/docs
To be able to add breakpoints you need to add a python run configuration for the debug.py script:
+
(add new configuration) and choose Python<project root>/dap_api/app/debug.py
<project root>/dap_api/app
.env
& .env-dev
files located in the project root (in this order !). If the .env
& .env-dev
files do not show up, enable showing hidden files. (Mac: Command+Shift+Dot) Click the debug button on the previously created run configuration.
To test if the breakpoints are working, add one in the return line of the root path function (@app.get("/")
decorator)
and open http://localhost:8083 in your browser.
Make sure the dap-test-db
container is running
(If you always forget to do so, you can create a python run config for the prestart_test_db.py
and let it run before the tests.)
To quickly run tests you can create a pytest run configuration:
+
(add new configuration) and choose pytest<project_root>/dap_api/app/app/test
.env
& .env-dev
files located in the project root (in this order !)<project_root>/dap_api/app
and select Mark Directory as ->
Sources Root. (NOT the db_api/app/app
folder)Will open a user-friendly test interface, instead of scrolling through console :+1: (Also having the option to only rerun failed tests)
You will also be able to debug the tests if you run this config in debug mode.
Alternatively you could run the tests within the dap-api-reload
docker container:
# access the reload container
docker exec -i -t dap-api-reload bash
# run the tests (you should be in /app already)
pytest
For displaying test coverage during development there are 2 viable options:
Use the Run pytest with coverage button (next to the debug button) for the previous pytest configuration. This is a bit slower and creates coverage for the whole project. Also, it is only available in the professional PyCharm Edition. It will visually highlight all parts in the code that are not covered by tests yet.
Add --cov=app --cov-report=term-missing
as Additional Arguments to the pytest configuration.
With this the pytest output will also create a coverage report including the lines "missed", blocks that were not
executed with the test.
important: PyCharm somehow uses the same helpers for coverage & debugging, so when using this approach,
your breakpoints will not be hit. You can either add the coverage as a second pytest config, or add --no-cov
as
Additional Arguments to the pytest configuration, and switch back and forth between debug and coverage mode by
adding and removing --no-cov
from the config.
For the second option you need to check
Pytest: don't add "--no-header --no-summary -q"
inPreferences
>Advanced Settings
in PyCharm
Of course, you can also run pytest with coverage in the reload-container with
pytest --cov=app --cov-report=term-missing
.
A html coverage report can be created to access coverage on an interactive website.
# run the tests with coverage (or add `--cov-report=html` to _Additional Arguments_)
pytest --cov=app --cov-report=html
Open the created dap-api/app/app/tests/htmlcov/index.html
in any browser.
If this is working correctly the code coverage can be displayed e.g. with codecov as a badge on the repository.
As during local development your app directory is mounted as a volume inside the container,
you can also run the migrations with alembic
commands inside the container, and the migration
code will be in your app directory (instead of being only inside the container).
So you can add it to your git repository.
Make sure you create a "revision" of your models and that you "upgrade" your database with that revision every time you change them. This is what will update the tables in your database. Otherwise, your application will have errors.
$ docker exec -it dap-api-reload bash
If you created a new model in ./app/app/models/
, make sure to import it in ./app/app/db/import_models.py
,
that imports all the models that will be used by Alembic.
After changing a model (for example, adding a column) inside the container, create a revision, e.g.:
$ alembic revision --autogenerate -m "Add column last_name to User model"
Commit to the git repository the files generated in the alembic directory.
After creating the revision, run the migration in the database (this is what will actually change the database):
$ alembic upgrade head
If you don't want to start with the default models and want to remove them / modify them, from the beginning,
without having any previous revision, you can remove the revision files (.py
Python files) under ./backend/app/alembic/versions/
.
Then create a first migration as described above.
See Readme in alembic for further information.