This repo is now archived, development has moved here

CriMS

Policing and Crime supply-demand modelling. CriMS is an evolution of crime-sim-toolkit ^[6] and forms the microsynthesis and microsimulation components of this workflow:

Population Data

Uses the ukcensusapi ^[1] and ukpopulation ^[2] packages to generate MSOA-level population data derived from the 2011 census and scaled to 2020 subnational population projections.

Crime Data

Uses the police-api-client ^[3] and the police open data portal ^[4] directly to get open data on crime occurrences.

Model

Uses the neworder ^[5] microsimulation framework to run the model. It uses historical data to determine counts of crimes as a function of location (MSOA), time (month), and (broad) type, so can capture seasonal fluctuations in crime frequency. It then imposes further weekly and daily periodicity to the crime rate, and this to sample crime incidences from a non-homogeneous Poisson process. More detailed crime types, and whether a suspect has been identified, are also sampled at force area resolution. This synthetic crime data can be fed into an agent-based model of Police operations which can alter its policies, potentially feeding back changes to crime rates that may result.

Data sources

• Bulk crime and outcome data, force boundaries: data.police.uk
• Mapping between Home Office Offence Codes and the data.police.uk categories. : data.police.uk

• MSOA (2011) boundaries: geoportal.statistics.gov.uk
• Detailed crime counts by ONS classification
• Crime severity scores

NB Although none of the data here contains any personally identifiable information, some is encrypted as a precaution and the encryption key is NOT provided in the repo, but may be provided on request.

In order to run the model, you will need to be able to access encrypted data. See below for further instructions

Mapping crime types to counts and severity weights

Crimes are sampled only by the 14 broad Police.UK categories, since this is the only categorisation in the bulk data. Police.UK also provide a breakdown of finer crime types by category.

ONS provide annual counts of crimes by force area in a more detailed categorisation with 134 distinct types, including the home office code for each type

However, the resourcing impact on the police of different crimes varyies enormously and "severity scores" are given by crime type for 246 detailed crime descriptions, including the home office code.

The first two datasets are not consistent, and require manual (i.e. insightful) matching of the finer/detailed crime type across the two datasets, which in turn allows weighted sampling of the ONS finer crime categorisation from the Police.UK broad category.

The home office codes can then be used to join the crime counts and their severity scores, so that the data supplied to the upstream (agent-based) model includes a measure of the police resourcing required to deal with each simulated crime. This step is can be done programmatically, and the code is given in weights.py

The mapping dataset is in data/policeuk-ons-code-join.csv

Usage

Dependencies

First install dependencies, either

pip install -r requirements.txt

or for conda environments:

conda env create -f conda-env.yml

(NB the above file is created using conda env export > conda-env.yml)

Encryption

Some of the input data is encrypted and the model requires a key to decrypt it. The key should be stored in the environment variable CRIMS_ENCRYPTION_KEY. The best practice for managing this is to store the key in a local .env file (which will be loaded by python automatically), with the following content:

CRIMS_ENCRYPTION_KEY=<insert key here>

Standalone Run

The script run-model.py can be used to run the model on a single force area and plot some output. Change the force area by editing the script. Run it like so:

python run_model.py "West Yorkshire" 2020 1 2021 1

which will simulate crime occurrences for West Yorkshire Police for the calendar year 2020. Note that not all the crime locations will be within the force area.

Output

The model produces simulated crime data in four variables:

• spatial: MSOA in which the crime occurred
• temporal: the time at which the crime occurred/was reported/was responded to. (TODO which?)
• categorical:
  • the type of the crime
  • whether a suspect has been identified

Model Stack

Interoperation of Police supply-demand ABM and crime microsimulation: see stack

App Service

NB this has dependencies (e.g. Flask) not specified in requirements.txt/conda-env.yml. If running outside docker, you will need to install manually.

Building

The app's data is in a separate image due to its size and infrequent changes. Build and push this image only as necessary:

docker build -t mopd/crims-data -f Dockerfile.data .
docker push mopd/crims-data

The app itself it more lightweight, and uses the data as a base image

docker build -t mopd/crims -f Dockerfile.app .
docker push mopd/crims

Running

To run locally

FLASK_APP=server.py flask run

which exposes an API at port 5000 with two endpoints:

/data

Takes 2 query params, force and month plus an optional param format (which defaults to json), and returns one month's simulated crime data for a given force area, e.g.

http://localhost:5000/data?force=Durham&month=7

http://localhost:5000/data?force=City%20of%20London&month=2&format=csv

/map

Takes 2 query params, force and month, and returns crime density by MSOA plotted on a map.

http://localhost:5000/map?force=Devon%20and%20Cornwall&month=12

Docker

App Service Container

This service is available as a docker image (due to its size and relatively infrequent changes, the data is in a separate image - which will take a while to initially download):

docker pull mopd/crims
docker run --rm -d  -p 80:5000/tcp mopd/crims

which runs it locally, listening for requests on the default http port. You can then request data from the container, e.g. in python/pandas:

>>> import pandas as pd
>>> df1 = pd.read_csv("http://localhost/data?force=City%20of%20London&month=3&format=csv")
>>> df1.head()
        MSOA                    crime_type                                  description                 time  suspect
0  E02006924  violence and sexual offences  Sexual assault on a female aged 13 and over  2020-03-01 00:35:00    False
1  E02000001  violence and sexual offences                       Assault without injury  2020-03-01 02:09:00     True
2  E02000001                   other theft                                  Other theft  2020-03-01 02:37:00    False
3  E02000001                 vehicle crime             Interfering with a motor vehicle  2020-03-01 03:33:00    False
4  E02000001                   other theft                                  Other theft  2020-03-01 03:56:00    False
>>> df2 = pd.read_json("http://localhost/data?force=City%20of%20London&month=3", orient="table")
>>> df2.head()
        MSOA             crime_type                                description                time  suspect
0  E02000001  possession of weapons  Possession of article with blade or point 2020-03-01 00:09:00    False
1  E02000001            other theft                                Other theft 2020-03-01 00:44:00     True
2  E02000001          vehicle crime                         Theft from vehicle 2020-03-01 02:38:00    False
3  E02000001          vehicle crime                         Theft from vehicle 2020-03-01 05:28:00    False
4  E02000001               burglary            Burglary Business and Community 2020-03-01 06:39:00    False
>>>

Integrated Models Container

Police-Supply-Demand (crims-integration branch) is a submodule, added like so

git submodule add -b crims-integration git@github.com:danbirks/Police-Supply-Demand

to update,

git submodule update [--init]

using the --init flag if the content of the submodule is empty.

Better still,

git config --global submodule.recurse true

then git pull will update the submodule too

An interactive GUI-based demo of the model integration between the crims microsimulation and an agent-based netlogo model can be found in the mopd/crims-int container. The ABM currently performs the Schelling segregation model whilst also sampling crimes from crims and feeding back a (random) loading factor that increases or decreases the overall crime rate.

This is built with

docker build -t mopd/crims-int -f Dockerfile.int .

and requires permission to connect to the host's graphical display when it runs. On ubuntu, this works:

xhost +
docker run --rm -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY mopd/crims-int

on other platforms YMMV. Google is your friend.

References

1. ukcensusapi: UK census data query automation
2. ukpopulation: UK Demographic Projections
3. police-api-client: Python client library for the Police API
4. Police Open Data Portal
5. neworder: A dynamic microsimulation framework
6. crime-sim-toolkit