Create production infrastructure on Azure

[bartjkdp]

VNG / service provider

• [x] Setup with service provider
• [x] Generate access tokens for every system connection at 's-Hertogenbosch
• [x] Set security headers
• [x] Configure monitoring -> requested at True
• [x] Configure task signals.apps.signals.tasks.anonymize_reporters on 0 4 *
• [x] Copy selective data dump from testing environment to acceptance and production -> waiting for finalizing at 's-Hertogenbosch
• [x] Move images to media persistent volume
• [x] Configure backups -> requested at True
• [x] Perform loadtests -> will be performed on Thursday 25th by @bartjkdp

's-Hertogenbosch

• [x] Configure DNS -> requested at 's-Hertogenbosch
  • [x] meldingen.s-hertogenbosch.nl
  • [x] meldingen.afvalstoffendienst.nl
• [x] Create web certificates -> requested at 's-Hertogenbosch
  • [x] meldingen.s-hertogenbosch.nl
  • [x] meldingen.afvalstoffendienst.nl -> waiting for configuration by @bartjkdp
• [x] Create system-to-system certificate -> requested at 's-Hertogenbosch
• [x] Configure OpenID Connect (Single Sign On) -> requested at 's-Hertogenbosch
• [x] Configure API gateway -> requested at 's-Hertogenbosch
• [x] Check settings in application -> requested at 's-Hertogenbosch

All requests go through @erikveerm

[bartjkdp]

We generate a public / private key pair for service accounts with jwcrypto:

import uuid
from jwcrypto import jwt, jwk

key = jwk.JWK.generate(kty='EC', crv='P-256', kid=str(uuid.uuid4()))

keyset = jwk.JWKSet()
keyset.add(key)

keyset.export() # includes private key, remove "d" for public key

Load the key with:

from jwcrypto import jwt, jwk

data = <key>
key = jwk.JWK(**data) # load specific key, not keyset

Generate specific tokens for service accounts with:

token = jwt.JWT(header={'alg': 'ES256', 'kid': key.key_id}, claims={'aud': 'signalen', 'email': 'moon@s-hertogenbosch.signalen.dev', 'name': 'MOON systeem'})
token.make_signed_token(key)
token.serialize()

[bartjkdp]

Created Signalen/backend#122 for improved tracing.

[bartjkdp]

Steps I followed to configure the new acceptance environment:

Automated steps

Dump data from old testing environment with:

python manage.py dumpdata \
--exclude admin \
--exclude sessions.session \
--exclude sites.site \
--exclude django_celery_beat \
--exclude django_celery_results \
--exclude users.profile \
--exclude feedback \
--exclude reporting \
--exclude signals.signal \
--exclude signals.attachment \
--exclude signals.buurt \
--exclude signals.categoryassignment \
--exclude signals.history \
--exclude signals.location \
--exclude signals.note \
--exclude signals.priority \
--exclude signals.reporter \
--exclude signals.signaldepartments \
--exclude signals.signaluser \
--exclude signals.status \
--exclude signals.type > /tmp/data.json

Remove all Amsterdam-specific data and initially created testdata with:

from django.contrib.auth.models import User, Group, Permission
from django.contrib.contenttypes.models import ContentType
from signals.apps.users.models import Profile

User.objects.all().delete()
Group.objects.all().delete()
Permission.objects.all().delete()
ContentType.objects.all().delete()
Profile.objects.all().delete()

from signals.apps.signals.models import Department, Category, CategoryTranslation

Department.objects.all().delete()
CategoryTranslation.objects.all().delete()
Category.objects.filter(parent__isnull=False).delete()
Category.objects.filter(parent__isnull=True).delete()

Then restore the dump on the new environment with:

python manage.py loaddata /tmp/data.json

Remove old users with:

User.objects.get(email="admin@signalen.s-hertogenbosch.dev").delete()
User.objects.get(email="gebruiker1@signalen.s-hertogenbosch.dev").delete()
User.objects.get(email="gebruiker2@signalen.s-hertogenbosch.dev").delete()
User.objects.get(email="gebruiker3@signalen.s-hertogenbosch.dev").delete()
User.objects.get(email="moon@signalen.s-hertogenbosch.dev").delete()

Create system users again with:

User.objects.create(username="moon@s-hertogenbosch.signalen.dev", email="moon@s-hertogenbosch.signalen.dev", first_name="Moon", last_name="Systeem")
User.objects.create(username="cognos@s-hertogenbosch.signalen.dev", email="cognos@s-hertogenbosch.signalen.dev", first_name="Cognos", last_name="Systeem")
User.objects.create(username="esb@s-hertogenbosch.signalen.dev", email="esb@s-hertogenbosch.signalen.dev", first_name="ESB", last_name="Systeem")

And recreate the superuser with:

python manage.py createsuperuser --username {email} --email {email}

Finally initialize the Elasticsearch index with:

python manage.py elastic_index --init
python manage.py elastic_index --index-all

Manual steps

• Check sources for duplicates
• Assign the correct roles to the system users
• Configure site
• Copy newest version of machine learning model
• Copy UserProfile and UserProfileDepartments manually