Census Atlas geodata API and database.
⚠️ This repository will be archived in September 2024 as it is no longer in development. ⚠️
Environment Variables
Database connection details and feature flags are controlled with environment variables. Almost all processes depend on these variables being set correctly, so be careful to verify before running anything risky.
Postgres/PostGIS
The system is designed around Postgres with PostGIS extensions. The database may be run as a native local process, within a container, or as an RDS instance.
Data Ingest
The "ingest" process takes source data files and loads them into the database. For best results, the ingest should be applied to a local native postgres instance. Running against RDS directly or against a containerised postgres instance is too slow.
Source data files should be in place, and the postgres environment variables must be exported before running the ingest script (indigestion.sh).
Export/Import
The export/import processes copies a database. Our main application is to export a locally ingested database for import into RDS.
API
The API presents a specific interface for querying the database. It is mainly used by the front end application.
The API may be run as a local process, within a container, or on some temporary EC2 instances we have in place for the moment. The plan is that instance of the API will run within the standard develop and prod environments.
CLIs
These clis are mostly for developers, and may not be fully caught up with features of the API.
geoLookup.json file)Testing
Terraform
How-Tos
make debugTo enable postgres and census queries, set ENABLE_DATABASE, and the postgres environment variables.
To lookup the postgres password in Secrets Manager, set FI_PG_SECRET_ID instead of PGPASSWORD,
and make sure to set AWS_REGION and any other AWS environment variables.
swagger.yaml is used to generate code via make generate
Particularly api/api.go (and similar files) shouldn't be directly edited.
| Environment variable | Default | Description |
|---|---|---|
| BIND_ADDR | :25252 | The host and port to bind to |
| GRACEFUL_SHUTDOWN_TIMEOUT | 5s | The graceful shutdown timeout in seconds (time.Duration format) |
| HEALTHCHECK_INTERVAL | 30s | Time between self-healthchecks (time.Duration format) |
| HEALTHCHECK_CRITICAL_TIMEOUT | 90s | Time to wait until an unhealthy dependent propagates its state to make this app unhealthy (time.Duration format) |
| ENABLE_DATABASE | false | Enable postgres and census query functionality |
| AWS_REGION | used by AWS SDK when ENABLE_DATABASE is true and PGPASSWORD is empty | |
| PGHOST | postgres host when ENABLE_DATABASE is true | |
| PGPORT | postgres port when ENABLE_DATABASE is true | |
| PGUSER | postgres user when ENABLE_DATABASE is true | |
| PGPASSWORD | postgres password when ENABLE_DATABASE is true (also see FI_PG_SECRET_ID) | |
| PGDATABASE | postgres database when ENABLE_DATABASE is true | |
| FI_PG_SECRET_ID | ARN of key holding postgres password if PGPASSWORD is empty |
See CONTRIBUTING for details.
Copyright © 2021, Office for National Statistics (https://www.ons.gov.uk)
Released under MIT license, see LICENSE for details.