[DEVOPS] Setup fresh stage server on Azure

kshitijrajsharma commented 1 year ago

Stage server should be pointed to this URL : https://export-staging.hotosm.org/ , Currently it is broken and we are trying to setup fresh with raw-data-api

eternaltyro commented 1 year ago

Standard instance on Azure with the following configuration.

Instance capacity:

Root disk: 32GB (mounted on /) Data disk: 200 GB (mounted to /data)

Applications to be installed:

Ubuntu 22.04 LTS
PostgreSQL server (in-situ)
Redis server (in-situ)
Python == 3.6.10 (installed using conda or PyPy)
Node.js == 16.14

Monitoring:

Jumpcloud agent
Newrelic infra agent

Configuration

RAW_DATA_API_URL = https://raw-data-api-demo.hotosm.org/
HDX_Site = demo

Sensitive strings:

Mailgun SMTP credentials
PostgreSQL credentials
OAuth1 client ID and secret

Database configuration

CREATE DATABASE exports
CREATE EXTENSION postgis
Add exports specific database user

Configuring OAuth app

Create OAuth1 application in OSM with Read preferences in scope. Redirect URI = https://export-stage.hotosm.org/authorized

Other packages

Some java packages are required for Garmin exports.
osmium - for updating the planet file

Scheduled jobs

cleanup.py and schedule.py must be running as cron services (or systemd.timer) . Configuration and schedule to be copied from production

Building frontend

cd ui
npm run dist
cd ..
python manage.py collectstatic

eternaltyro commented 1 year ago

Added supporting systemd files and corresponding timers and env files in this PR: https://github.com/hotosm/osm-export-tool/pull/495

eternaltyro commented 1 year ago

Export tool requires a specific (and unfortunately unsupported) version of Python to function. Old Python versions can be managed using miniconda.

$ curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o miniconda-setup.sh
$ chmod +x miniconda-setup.sh && ./miniconda-setup.sh
$ export PATH=$PATH:/home/user/miniconda3/bin
$ conda config --set auto_activate_base false
$ conda create --name exports python=3.6.10 rasterio gdal rtree
$ conda activate exports 
(exports) $ python --version
Python 3.6.10 :: Anaconda, Inc.

eternaltyro commented 1 year ago

Suggested changes to env vars:

Via review to PR: https://github.com/hotosm/osm-export-tool/pull/498 And by issue: https://github.com/hotosm/osm-export-tool/issues/499

eternaltyro commented 1 year ago

Discovered that data is being shipped to Sentry using an old outdated python library called Raven. Requested removal: https://github.com/hotosm/osm-export-tool/issues/500

hotosm / osm-export-tool