search

ECMWFCode4Earth / wildfire-forecasting

Forecasting wildfire danger using deep learning.
GNU General Public License v3.0
50 stars 11 forks source link
deep-learning earth-observation gis remote-sensing wildfire-forecasting

readme

Forecasting Wildfire Danger Using Deep Learning

Documentation Status Binder Code style: black

Introduction

The Global ECMWF Fire Forecasting (GEFF) system, implemented in Fortran 90, is based on empirical models conceptualised several decades back. Recent GIS & Machine Learning advances could, theoretically, be used to boost these models' performance or completely replace the current forecasting system. However thorough benchmarking is needed to compare GEFF to Deep Learning based prediction techniques.

The project intends to reproduce the Fire Forecasting capabilities of GEFF using Deep Learning and develop further improvements in accuracy, geography and time scale through inclusion of additional variables or optimisation of model architecture & hyperparameters. Finally, a preliminary fire spread prediction tool is proposed to allow monitoring activities.

TL; DR

This codebase (and this README) is a work-in-progress. The master is a stable release and we aim to address issues and introduce enhancements on a rolling basis. If you encounter a bug, please file an issue. Here are a quick few pointers that just work to get you going with the project:

Next:

The work-in-progress documentation can be viewed online on wildfire-forecasting.readthedocs.io.

Getting Started

Using Binder

While we have included support for launching the repository in Binder, the limited memory offered by Binder means that you might end up with crashed/dead kernels while trying to test the Inference or the Forecast notebooks. At this point, we don't have a workaround for this issue.

Clone this repo

git clone https://github.com/esowc/wildfire-forecasting.git
cd wildfire-forecasting

Once you have cloned and navigated into the repository, you can set up a development environment using either conda or docker. Refer to the relevant instructions below and then skip to the next section on Running Inference

Using conda

To create the environment, run:

conda env create -f environment.yml
conda clean -a
conda activate wildfire-dl

The setup is tested on Ubuntu 18.04, 20.04 and Windows 10 only. On systems with CUDA supported GPU and CUDA drivers set up, the conda environment and the code ensure that GPUs are used by default for training and inference. If there isn't sufficient GPU memory, this will typically lead to Out of Memory Runtime Errors. As a rule of thumb, around 4 GiB GPU memory is needed for inference and around 12 GiB for training.

Using Docker

We include a Dockerfile & docker-compose.yml and provide detailed instructions for setting up your development environment using Docker for training on both CPUs and GPUs. Please head over to the Docker README for more details.

Running Inference

Implementation overview

deep-learning-network-architecture We implement a modified U-Net style Deep Learning architecture using PyTorch 1.6. We use PyTorch Lightning for code organisation and reducing boilerplate. The mammoth size of the total original dataset (~1TB) means we use extensive GPU acceleration in the code using NVIDIA CUDA Toolkit. For a GeForce RTX 2080 with 12GB memory and 40 vCPUs with 110 GB RAM, this translates to a 25x speedup over using only 8 vCPUs with 52GB RAM.

For reading geospatial datasets, we use xarray and netcdf4. The imbalanced-learn library is useful for Undersampling to tackle the high data skew. Code-linting and formatting is done using black and flake8.


Optional arguments (default values indicated below):

`  -h, --help show this help message and exit`
    -init-features 16                       Architecture complexity [int]
    -in-days 4                              Number of input days [int]
    -out-days 1                             Number of output days [int]
    -epochs 100                             Number of training epochs [int]
    -learning-rate 0.001                    Maximum learning rate [float]
    -batch-size 1                           Batch size of the input [int]
    -split 0.2                              Test split fraction [float]
    -use-16bit True                         Use 16-bit precision for training (train only) [Bool]
    -gpus 1                                 Number of GPUs to use [int]
    -optim one_cycle                        Learning rate optimizer: one_cycle or cosine (train only) [str]
    -dry-run False                          Use small amount of data for sanity check [Bool]
    -case-study False                       The case-study region to use for inference: australia,california, portugal, siberia, chile, uk [Bool/str]
    -clip-output False                      Limit the inference to the output values within supplied range (e.g. 0.5,60) [Bool/list]
    -boxcox 0.1182                          Apply boxcox transformation with specified lambda while training and the inverse boxcox transformation during the inference. [Bool/float]
    -binned "0,5.2,11.2,21.3,38.0,50"       Show the extended metrics for supplied comma separated binned FWI value range [Bool/list]
    -undersample False                      Undersample the datapoints having smaller than specified FWI (e.g. -undersample=10) [Bool/float]
    -round-to-zero False                    Round off the target values below the specified threshold to zero [Bool/float]
    -date-range 2019-04-01,2019-05-01       Limit prediction to a smaller subset of dates than available in the data directories [Bool/float]
    -cb_loss False                          Use Class-Balanced loss with the supplied beta parameter [Bool/float]
    -chronological_split False              Do chronological train-test split in the specified ratio [Bool/float]
    -model unet_tapered                     Model to use: unet, unet_downsampled, unet_snipped, unet_tapered, unet_interpolated [str]
    -out fwi_reanalysis                     Output data for training: gfas_frp or fwi_reanalysis [str]
    -smos_input False                       Use soil-moisture input data [Bool]
    -forecast-dir ${FORECAST_DIR}           Directory containing forecast data. Alternatively set $FORECAST_DIR [str]
    -forcings-dir ${FORCINGS_DIR}           Directory containing forcings data. Alternatively set $FORCINGS_DIR [str]
    -reanalysis-dir ${REANALYSIS_DIR}       Directory containing reanalysis data. Alternatively set $REANALYSIS_DIR [str]
    -smos-dir ${SMOS_DIR}                   Directory containing soil moisture data. Alternatively set $SMOS_DIR [str]
    -mask src/dataloader/mask.npy           File containing the mask stored as the numpy array [str]
    -benchmark False                        Benchmark the FWI-Forecast data against FWI-Reanalysis [Bool]
    -comment Comment of choice!             Used for logging [str]
    -checkpoint-file                        Path to the test model checkpoint [Bool/str]

Documentation

We use Sphinx for building our docs and host them on Readthedocs. The latest build of the docs can be accessed online here. In order to build the docs from source, you will need sphinx and sphinx-autoapi. Follow the instructions below:

cd docs
make html

Once the docs get built, you can access them inside docs/build/html/.

Acknowledgements

This project tackles Challenge #26 from Stream 2: Machine Learning and Artificial Intelligence, as part of the ECMWF Summer of Weather Code 2020 Program.

Team: Roshni Biswas, Anurag Saha Roy, Tejasvi S Tomar.