FixMyBerlin / fixmy.platform

Backend providing (geo)data and content for multiple applications
GNU Affero General Public License v3.0
11 stars 3 forks source link
django python

fixmy.platform

CI

Development environment

In order to use the provided development environment install Docker Desktop for Mac or Windows. On Linux, make sure you have the latest version of Compose.

This project has a docker-compose.yml file, which will start the Django application on your local machine.

1. Setup Docker

Clone the repository and start the development environment in the created directory:

docker compose up -d

(Should you recieve errors like docker/transport/unixconn.py, try starting docker desktop first, then run the compose command.)

2. Test API

Now, the API is available at http://localhost:8000/api/.

curl http://localhost:8000/api/
# => {"users":"http://localhost:8000/api/users/"}

3. Configure Django

Access the console of the Docker container running the backend app:

docker compose exec app bash

3.1 Create a new user for Django's admin console

http://localhost:8000/admin/

python manage.py createsuperuser

3.2 Seed data

TODO

Code Editor VSCode

"black" code formatter

Please use black as code formatter. https://black.readthedocs.io/en/stable/

We use # fmt: off whenever we want to specify our own formatting style.

VSCode hints:

VSCode Docker support

Using the Docker extension is recommended to develop in VSCode. It requires special ./.devcontainer setting files hat a fellow developer can provide.

Configuration

Configuration options are set through environment variables, all of which are listed in docker-compose.yml.

Products

/reports: Radbügel Meldedialog

/fixmyapp: Xhain-Terrassen / Gastro app

/fixmyapp: Planungskarte, Happy-Bike-Index (HBI)

Environment variables

/fixmyapp: Spielstraßen / Playstreets app

Note there is also spielstraßen.de which might be using the static geojson extract of this from github.com/FixMyBerlin/fixmy.spielstrassen

/fahrradparken: radparken.info

/survey: Studie zur subjektiven Sicherheit im Radverkehr

Components

E-Mail Service

TODO

Mapbox service

TODO

Newsletter service

TODO

Django commands

Debugging

Using debugpy:

You can enable interactive debugging through debugpy by setting the environment variable DEBUGPY=1 (e.g. through a .env file).

Using pdb:

Custom Commands: fixmyapp

anonymizedata

Removes personal information from the database. By default it preserves data of staff.

python manage.py anonymizedata

downloadfiles

Downloads content of S3 bucket to /tmp, filtered by path prefix. This command requires the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_S3_REGION_NAME and AWS_STORAGE_BUCKET_NAME to be set correctly.

python manage.py downloaddata Data/

exportprojects

Exports projects as GeoJSON intended for Mapbox.

python manage.py exportprojects > /tmp/projects.json

exportsections

Exports sections as GeoJSON intended for Mapbox.

python manage.py exportsections > /tmp/sections.json

exportplaystreets

Similar to exportreports, this command exports play street signups to a csv file.

python manage.py exportplaystreets spielstrassen.csv

exportgastrosignups

This command exports GastroSignup entries, either in CSV or GeoJSON format. The GeoJSON format has an additional switch that allows exporting usage areas instead of the geolocation of the entry's shop.

The CSV format doesn't include usage areas and uploaded certificates.

The GeoJSON format doesn't include any personal information.

In order to export the requested usage areas in GeoJSON format into a file gastrosignup_area.geojson:

python manage.py exportgastrosignups --format geojson --area gastrosignup_area.geojson

updatehbi

Bootstrap the database for HBI. Downloads and imports road and intersection data, section details and section accidents, also applying migrations.

python manage.py updatehbi

importsections

Imports sections from shape file. The file is usually downloaded from S3 with downloadfiles.

python manage.py importsections /tmp/road_sections-v1.1.shp
python manage.py importsections /tmp/intersections-v1.1.shp

importsectiondetails

Imports section details including pictures and traffic information from CSV file. The file is usually downloaded from S3 with downloadfiles.

python manage.py importsectiondetails /tmp/section_details-v1.1.csv

importsectionaccidents

Import section accident data set, which references previously imported sections.

python manage.py importsectionaccidents /tmp/section_accidents-v1.1.csv

updateprojectgeometries

Updates project geometries from a shape file. The file is usually downloaded from S3 with downloadfiles.

python manage.py updateprojectgeometries /tmp/projects.shp linestring

uploadtileset

Uploads GeoJSON export of projects or sections (see exportprojects and exportsetcions) to Mapbox. This command requires the environment variables MAPBOX_UPLOAD_NAME_SECTIONS, MAPBOX_UPLOAD_TILESET_SECTIONS, MAPBOX_UPLOAD_NAME_PROJECTS and MAPBOX_UPLOAD_TILESET_PROJECTS to be set correctly.

python manage.py uploadtileset --dataset projects /tmp/projects.json

Custom Commands: reports

exportreports

Export reports about bike stands in either CSV or GeoJSON format

python manage.py exportreports --format csv /tmp/reports.csv

Notes:

importreports

Import or update entries in the reports app from a CSV file.

The imported CSV file is required to have the columns;

python manage.py importreports reports.csv

sendnotifications

Batch send notifications, which are enqueued when a reports status is changed

python manage.py sendnotifications

This command requires the environment variables REPORTS_NOTIFICATION_CAMPAIGN and REPORTS_NOTIFICATION_SENDER to be set. You can send sample emails containing all variations of the text templates using

python manage.py sendnotifications --send-samples your@email.org

Other scripts

fetch_database

Creates a backup using a given Heroku app, downloads it and uses it to overwrite the local database schema and contents. User data is anonymized in the process, except for staff users, who are preserved.

todo: Requires heroku tools? Only for heroku based installations.

./scripts/fetch_database.sh fixmyplatform