NISMOD (National Infrastructure Systems Model) is an integrated model of infrastructure systems, developed as part of the ITRC/MISTRAL project.
NISMOD v2 has several components:
This repository contains only:
In particular:
To set up NISMOD, first clone this repository, or download a release. The directories in this folder contain or will contain everything required for a model run:
config
: model configurationdata
: (initially incomplete) model datamodels
: model wrappersnotebooks
: jupyter notebooks for exploring model resultsplanning
: decision modelsprovision
: scripts to install models and get dataresults
: (initially empty) for model run resultstest
: integration testsutilities
: helpful scripts to create or migrate configurationThen in outline (more details further below) either install everything directly to run nismod on a desktop, server or cluster:
conda
)s3://nismod2-data
(copy
provision/template.s3.ini
to provision/s3.ini
and edit)provision/template.dbconfig.ini
to
provision/dbconfig.ini
and edit for your database)provision/install_*
scripts)provision/get_data_*
scripts)Or set up a virtual machine for testing:
template.s3.ini
to s3.ini
and edit)template.dbconfig.ini
to
dbconfig.ini
)template.xpauth.xpr
or your own license to xpauth.xpr
)vagrant up
then connect with vagrant ssh
)Download the latest release of NISMOD v2.0 from GitHub as a .zip archive and unzip it into a directory.
Alternatively, clone the NISMOD v2.0 repository from https://github.com/nismod/nismod2.
cd path/to/git/projects # somewhere to keep the nismod project
git clone https://github.com/nismod/nismod2.git
cd nismod2
To update NISMOD to the latest development version:
cd path/to/nismod2 # wherever you cloned the nismod folder
git checkout master # to make sure you’re on the master branch
git pull # pull changes from Github to your local machine
Within the NISMOD folder, copy provision/template.s3.ini
to provision/s3.ini
and add your
credentials to access the NISMOD AWS S3 bucket:
[profile nismod2-s3]
aws_access_key_id = <your-access-key-id>
aws_secret_access_key = <your-secret-access-key>
region = eu-west-2
output = json
Contact Tom Russell or Craig Robson to request credentials.
This assumes the awscli
command line client is installed and
you have credentials.
pip install awscli # install
aws configure # configure (this will ask for access key and secret key)
To check contents of the bucket:
aws s3 ls s3://nismod2-data
To upload a zip:
aws s3 cp mydata_v1.1.zip s3://nismod2-data/model_name/mydata_v1.1.zip
Further usage and documentation: AWS Command Line Interface docs
Within the NISMOD folder, copy provision/template.dbconfig.ini
to provision/dbconfig.ini
and edit to connect to your database.
If you intend to use the vagrant virtual machine with the database as provisioned by default, leave the details unchanged.
[energy-supply]
dbname=vagrant
host=localhost
user=vagrant
password=vagrant
port=5432
If you will install FICO XPRESS manually, skip this step.
Within the NISMOD folder, copy provision/template.xpauth.xpr
to provision/xpauth.xpr
.
If you are running within the University of Oxford OUCE network, the license server should be available and no changes are necessary. Otherwise, set the value to connect to your local license server or replace this file with a license file provided by FICO support (contact via https://www.fico.com)
use_server server="ouce-license.ouce.ox.ac.uk"
To run NISMOD directly, you will need to install smif, model dependencies and models.
The provision/vm_provision.sh
script may be a useful point of reference, as it is run by the
root user within the virtual machine (on Ubuntu 16.04) to automate the same process.
See installation instructions for full details.
The recommended method uses conda
):
conda config --add channels conda-forge # enable the conda-forge channel, required for smif
conda create --name nismod python=3.6 smif # create a conda environment for nismod
conda activate nismod # activate the nismod conda environment
See energy supply documentation for full details.
On Linux (x64) it should be possible download and install FICO by running:
mkdir xpress_setup
wget -nc -qO- \
"https://clientarea.xpress.fico.com/downloads/8.5.10/xp8.5.10_linux_x86_64_setup.tar" \
--no-check-certificate | tar -C xpress_setup -xv
./xpress_setup/install.sh
See transport model documentation for full details.
To run the transport model, it should be sufficient to have the Java Runtime Environment (JRE) version 8 installed.
On Ubuntu, run:
sudo apt update && sudo apt install default-jre
See energy supply documentation for full details.
The energy supply model connects to a Postgres database via ODBC, so requires access to a Postgres server, an ODBC installation and the Postgres ODBC drivers.
On Ubuntu, install Postgres, libpq (shared library for postgres clients), ODBC and postgres connector:
sudo apt update && sudo apt install \
postgresql postgresql-contrib \
libpq-dev \
unixodbc-dev odbc-postgresql
Install boto3
: pip install boto3
.
Run each of the provision/get_data_*
scripts:
bash ./provision/get_data_digital_comms.sh .
bash ./provision/get_data_energy_demand.sh .
bash ./provision/get_data_energy_supply.sh .
bash ./provision/get_data_transport.sh .
bash ./provision/get_data_water_supply.sh .
bash ./provision/get_data_water_demand.sh .
Run each of the provision/install_*
scripts:
bash ./provision/install_digital_comms.sh .
bash ./provision/install_energy_demand.sh .
bash ./provision/install_energy_supply.sh . ./path/to/xpress_install_dir
bash ./provision/install_transport.sh .
bash ./provision/install_water_supply.sh .
bash ./provision/install_water_demand.sh .
Run post-install scripts:
energy_demand setup -f ./models/energy_demand/wrapperconfig.ini
Note that install_water_supply.sh
clones a repository that is currently private: setting up the water supply model may require requesting access to the repository.
The Vagrantfile defines the automated setup for a virtual machine, which will provide a reproducible development environment.
In virtual machine terminology, the virtual machine (vm for short) is sometimes called the ‘guest’ machine and the physical computer is called the ‘host’ machine.
To use the NISMOD virtual machine, first install:
If you see only 32-bit options in Virtualbox, please ensure that:
Note for Ubuntu 17.10 users: If you are experiencing the issue The box ‘bento/ubuntu-16.04’ could not be found or could not be accessed in the remote catalog. Make sure that you have the latest version of Vagrant (>v2) installed. This version is currently not in the standard package archive (PPA) but can be downloaded from the vagrant website.
Create and configure the guest machine, running this command from the NISMOD directory:
vagrant up
This will download a virtual machine image, install all the packages and software which are required to test and run NISMOD onto that virtual machine and download the data and model releases from the AWS S3 bucket.
Once it has run, you should be able to:
vagrant ssh # log in to the virtual machine on the command line
cd /vagrant # move to the folder that’s shared between the host and guest machines
ls # list files and folders
smif list # list available model runs
smif run energy_supply_toy # to run the energy_supply_toy model run
logout # log out of the virtual machine (or shortcut: CTRL+D)
Then results are written to subfolders within the results
directory.
Parquet files are smaller, typed and faster to read and write than CSV.
Run smif csv2parquet
converter after getting data and installing models, or after updates.
smif csv2parquet --noclobber data
Then run model-runs using the local_binary
interface. smif will read and write parquet
files.
smif run et_module_test --interface local_binary
To reload and re-provision the virtual machine after updating NISMOD:
vagrant reload --provision
It is possible to connect to some services running in the guest virtual machine directly from the host. Vagrant is set to automatically forward some ports.
NISMOD includes some experimental jupyter notebooks to help access and view results.
First, start the notebook server in the background with the command:
vagrant ssh
cd /vagrant
jupyter notebook &> /dev/null &
Then navigate in a browser to
localhost:8910
Some sector models use a database which runs within the virtual machine and is set up when the virtual machine is provisioned.
The virtual machine exposes the database on port 6543 on the host machine, so you should be able to connect with the following details:
key | value |
---|---|
Host | localhost |
Database | vagrant |
User | vagrant |
Password | vagrant |
Port | 6543 |
For example, on the command line, assuming the psql
postgres command line client is installed
on the host machine, you should be able to connect with:
psql -d vagrant -h localhost -U vagrant -p 6543
This database setup is intended for testing and development only and assumes that high-numbered ports are only accessible to the local user. Strong passwords and secure login methods should be used if the host machine is expected to be accessed by multiple users.
master
branch is protected against push and force-push.master
branch should be treated as live and is used for major releases onlydevelop
branchFor further detail, see this article
git checkout -b feature/<feature_name> develop
Update the changelog under the heading Version X.Y.Z
Check versions of models and data specified in provision/config.ini
Now, create a branch from develop e.g. checkout -b release-x.y develop
Tag this as a release candidate git tag -a vx.y.z-rc1
Any bugfixes are committed to this branch and should be tagged with incremented
release candidate tags e.g. git tag -a vx.y.z-rc2
Once the release candidate is stable, submit a pull request to the master
branch
Merge the pull request and tag git tag -a vx.y.z
git checkout -b hotfix/<bug_name> master
git tag -a vx.y.z
incrementing the minor z
digitThe tests in this repository are intended to test the integration of the various
NISMOD sector models with smif
, the simulation modelling integration
framework. In outline, tests here should:
smif
with only scenario dependencies (not
coupling with any other models)
Continuous integration testing is currently available through a GitLab server based in the School of Geography in Oxford. You can access the project on OUCE GitLab and monitor jobs through the web interface.
To add the remote and push a branch for automated testing, run:
git remote add ouce https://gitlab.ouce.ox.ac.uk/NISMOD/nismod2.git
git checkout feature/branch
git push -u ouce
There is a pool of gitlab-runner workers active that pick up and run jobs on each commit/push. A local gitlab-runner can be setup using the following configuration.
concurrent = 1
check_interval = 0
[[runners]]
name = "ubuntu-nismod2"
url = "https://gitlab.ouce.ox.ac.uk/"
token = "12964bfcd398c966ebf128f1b5b34f"
executor = "docker"
[runners.docker]
tls_verify = false
image = "ubuntu:xenial"
privileged = false
disable_cache = false
volumes = ["/cache"]
shm_size = 0
[runners.cache]
NISMOD was written and developed by researchers in the Infrastructure Transition Research Consortium as part of the EPSRC sponsored MISTRAL programme.