search

maproulette / maproulette-backend

MapRoulette back-end / API
Apache License 2.0
50 stars 38 forks source link
maproulette openstreetmap osm

readme

MapRoulette API

Build Status Quality Gate Status GitHub release (latest by date)

Welcome to the repository for the MapRoulette back-end server code. The MapRoulette back-end exposes the MapRoulette API, which the MapRoulette front-end web application depends on. The source code for the web application is in a separate repository.

If you just want to deploy the MapRoulette back-end, we have a 🚢 Docker image 🚢 for that. This is especially useful if you want to contribute to the MapRoulette front-end and don't intend to touch the back-end.

Requirements

MapRoulette depends on several technologies for building and running the project:

Setup

MapRoulette is a complex tool and depends on several other tools (Mapillary, Overpass API, OpenStreetMap, PostgreSQL, and others) that need to be correctly configured within the application's HOCON configuration files.

The initial setup is not trivial, so the documentation is split to specific steps where each step can be easily validated. The validation at each step is a helpful sanity check so please, regardless of your experience, take the time to verify along the way.

Step 1: Installing Tools

To get started you'll need to install Docker, JDK 17, and sbt.

Docker

Docker is a virtualization tool and feel free to use Docker Desktop, or Podman, or even Rancher Desktop.

JDK 17 and sbt

sdkman is a great tool to install a specific build of the JDK to keep your environment as similar to production as possible. It also handles fetching x8664 and aarch64 builds automatically. Follow the installation steps and install the JDK and sbt using a command similar to:

Validation

Within a terminal check that docker, javac, and sbt are working.

Step 2: Setup PostGIS and MapRoulette DB Configuration

PostGIS Container

MapRoulette development assumes a database is running on the local system within a docker container.

Below is a sample command to run a PostGIS database within a container and sets necessary ports/credentials.

docker run \
    -d \
    -p 5432:5432 \
    --name maproulette-postgis \
    --restart unless-stopped \
    --shm-size=512MB \
    -e POSTGRES_DB=maproulette-db \
    -e POSTGRES_USER=maproulette-db-user \
    -e POSTGRES_PASSWORD=maproulette-db-pass \
    postgis/postgis:13-3.3

Configuration

Clone the maproulette-backend repository and cd to that directory, and create conf/dev.conf using the example file:

    cp conf/dev.conf.example conf/dev.conf

Edit conf/dev.conf and set db.default based on the previous step's POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD values:

db.default {
  url="jdbc:postgresql://localhost:5432/maproulette-db"
  username="maproulette-db-user"
  password="maproulette-db-pass"
}

Alternatively, you can configure the backend using environment variables. The variables MR_DATABASE_URL, MR_DATABASE_USERNAME and MR_DATABASE_PASSWORD control the same values as the config parameters shown above. Look in conf/ application.conf for what else can be overridden via environment variables. Any pattern like ${?FOO} in that file will be replaced with the value of the environment variable FOO at runtime.

Now start the MapRoulette server! Run this command in a terminal, not within Intellij/vscode:

sbt -J-Xms4G -J-Xmx4G -J-Dconfig.file=./conf/dev.conf -J-Dlogger.resource=logback-dev.xml run

There should be some output that looks like this:

--- (Running the application, auto-reloading is enabled) ---
[info] p.c.s.AkkaHttpServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
(Server started, use Enter to stop and go back to the console...)

That's the expected output. And when you need, like it says, use Enter to stop the server.

Step 2 - Validation

Open a new terminal so that the MapRoulette server is not stopped. Verify:

If there's a failure at this point READ THE LOG MESSAGES, checking for simple setup issues.

A few known setup misconfigurations could be:

Step 3: Setup OAuth 2.0 Login and Run the front-end GUI

With the database and MapRoulette server working together, it's now time to get the front-end login workflow working!

High level the user visits to MapRoulette, clicks 'login' and is sent to OpenStreepMap to login then, after authentication, MapRoulette is able to make OSM changes of behalf of the user. This is done by registering the local dev instance of MapRoulette as an OSM OAuth 2.0 Application to create keys, and then the those keys are pasted into the MapRoulette server configuration. More details on how OSM uses OAuth is on the OpenStreetMap OAuth documentation.

Register localhost as an OpenStreetMap OAuth 2.0 Application

Within OpenStreetMap, MapRoulette needs to be setup as an "OAuth 2 Application" so that changes in MapRoulette are associated with a user's OSM account.

  1. Login to the development OpenStreetMap server (make an account if needed)
  2. In the top right, click the user and go to My Settings
  3. Then click OAuth 2 applications tab
  4. At the bottom of the OAuth2 settings page, click Register new application
  5. Register using these settings:
    • Name: maprouletteDevLocalhost (any name is fine)
    • Redirect URIs: http://127.0.0.1:9000
    • Confidential application: Keep this checked
    • Permissions: Select
      • Read user preferences
      • Modify user preferences (MapRoulette saves the user's MR API key in OSM)
      • Modify the map (MapRoulette tasks edit the map)

Click Register and note the displayed Client ID and the Client Secret. Copy and paste those keys into the MapRoulette conf/dev.conf file's osm section where it has "CHANGE_ME":

osm {
  # The OSM OAuth 2.0 Client ID and Client Secret, respectively
  consumerKey="CHANGE_ME"
  consumerSecret="CHANGE_ME"
}

Stop and start the MapRoulette server to load the updated configuration.

Run the MapRoulette front-end

Clone the maproulette3 repository and follow the steps in the DEVELOPMENT.md "Run the UI from Docker" section. Be sure to execute the docker run step that starts the UI.

Step 3 - Validation

Known setup issues:

Step 4: Intelllij Configuration

The server is working end-to-end on command line! Time to import into IntelliJ. Stop the running MapRoulette server from the previous step, if it's still running.

Step 4 - Validation

Open the front-end UI http://localhost:3000/ and attempt to log in. It should function.

Additional (Optional) Server Configuration

Notes

Logging Levels

During development, please set -Dlogger.resource=logback-dev.xml to have the best experience. The logback dev file sets many items to the devel level and logs all HTTP request paths and response times.

Any changes to the conf/logback-dev.xml will be loaded within about 5 seconds and a service restart is not needed.

To have all request headers sent by the client logged, update the conf/logback-dev.xml "org.maproulette.filters" entry to TRACE.

Deploying on Windows

Windows is not officially supported. However, there is an unofficial setup guide.

SMTP (email) configuration

MapRoulette now supports transmission of emails, for example to inform users when they receive new in-app notifications. You will need access to an SMTP server to send emails, and will also need to add SMTP configuration settings to your configuration file (or whatever configuration mechanism you're using). play.mailer.host is the only required SMTP setting, but most SMTP servers will also want a username and password.

play.mailer.host = "smtp.server.com"
play.mailer.user = "smtpusername"
play.mailer.password = "secret"

Many additional SMTP configuration options are available -- see play-mailer for a full list.

Note: If you're doing development, you can set play.mailer.mock = yes to simply log emails instead of transmitting them

It's also important that you set the emailFrom setting to the email address you wish emails to come from, and that you ensure publicOrigin is set so that links in emails will properly point to your server.

emailFrom = "maproulette@yourserver.com"
publicOrigin = "https://www.yourserver.com"

By default, notification emails that are to be sent immediately are processed by a background job every 1 minute. Both the frequency and max number of emails to process in a single run can be controlled.

notifications.immediateEmail.interval = "1 minute"
notifications.immediateEmail.batchSize = 10         # max emails per run

Notification emails that are to be sent as a digest are initially processed at 8pm local server time by default, and then every 24 hours thereafter (i.e. daily digests). Both of these settings can be customized. There is not currently a maximum limit to the number of emails for digest emails.

notifications.digestEmail.startTime = "20:00:00"    # 8pm local server time
notifications.digestEmail.interval = "24 hours"     # once daily

Creating new Challenges

The wiki for this repo has some information on creating challenges.

Challenge API has further information about creating challenges through the API.

See also the Swagger API documentation. You can view the documentation by going to the URL /docs/swagger-ui/index.html on any MapRoulette instance.

Dev Docs

Contributing

Please fork the project and submit a pull request. See Postman Docs for information on API Testing. The project is integrated with Travis-CI, so PR's will only be accepted once the build compiles successfully. MapRoulette also uses Scalafmt as it's code formatter. This is too keep the code style consistent across all developers. The check will be run first in Travis for the build, so if there are any code style issues it will fail the build immediately. IntelliJ should pick up the formatter and use Scalafmt automatically, however you can also use sbt scalafmt to format any and all code for you.

Contact

Bug and feature requests are best left as an issue right here on Github. For other things, contact maproulette@maproulette.org

MapRoulette now also has a channel #maproulette on the OSM US Slack community.