search

VSainteuf / pytorch-psetae

PyTorch implementation of the model presented in "Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention"
MIT License
184 stars 40 forks source link
agriculture computer-vision cvpr cvpr2020 deep-learning earth-observation machine-learning pytorch remote-sensing satellite-image self-attention spatio-temporal time-series-classification transformer-architecture

readme

Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention (CVPR 2020, Oral)

PyTorch implementation of the model presented in "Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention" published ar CVPR 2020.

Paper abstract:

Satellite image time series, bolstered by their growing availability, are at the forefront of an extensive effort towards automated Earth monitoring by international institutions. In particular, large-scale control of agricultural parcels is an issue of major political and economic importance. In this regard, hybrid convolutional-recurrent neural architectures have shown promising results for the automated classification of satellite image time series. We propose an alternative approach in which the convolutional layers are advantageously replaced with encoders operating on unordered sets of pixels to exploit the typically coarse resolution of publicly available satellite images. We also propose to extract temporal features using a bespoke neural architecture based on self-attention instead of recurrent networks. We demonstrate experimentally that our method not only outperforms previous state-of-the-art approaches in terms of precision, but also significantly decreases processing time and memory requirements. Lastly, we release a large open-access annotated dataset as a benchmark for future work on satellite image time series.

[UPDATES]

Requirements

The code has been tested in the following environment:

Ubuntu 18.04.1 LTS, python 3.6.6, pytorch 1.1.0, CUDA 10.0

Downloads

Datasets

A toy version of the Pixel-set dataset can be directly downloaded here, to get an idea of the dataset structure.

The complete Pixel-set and Pixel-patch datasets are accessible on Zenodo at the following links:

Pre-trained weights

We also provide the pre-trained weights for inference.

Code

Code structure

Code Usage

Reproduce

Run the train.py script to reproduce the results of the PSE+TAE architecture presented in the paper. You will just need to specify the path to the Pixel-Set dataset (link above) with the --dataset_folder agrument.

Experiment

The default settings of the train.py script are those used to produce the results in the paper. Yet, some options are already implemented to play around with the model's hyperparameters and other training settings. These options are accessible through an argparse menu (see directly inside the script).

Re-use

Data format

In order to use the PixelSetData dataset classs with other data than those provided in the link above, the data folder should be structured in the following fashion:

Data structure

Samples

Each dataset sample consits in the different observations for a single parcel. The observations are aggregated in a single array of shape TxCxS with T the number of temporal observations, C the number of channels, and S the number of pixels in the parcel (different for each data sample). Each of these arrays should be stored separately in a numpy file: unique_id_of_the_sample.npy

All the individual .npy files are stored in the same sub-directory DATA.

Normalisation values

The normalisation values should be computed beforehand and stored in the form of a tuple of arrays (means, stds) in a pickle file in the main folder. The PixelSetData dataset class can adapt to different normalisation strategies depending on the shape of the arrays:

Labels

The labels should be stored in the META/labels.json file. This file has a nested dictionary like structure and can contain multiple nomenclatures:

labels.json = {
  "Name_of_nomenclature1": {
    "unique_id_0": label_0,
    ...,
    "unique_id_N": label_N,
    }, 
  "Name_of_nomenclature2": {
    "unique_id_0": label_0,
    ...,
    "unique_id_N": label_N,
    }
}

Dates and pre-computed features

The dates of the observations, if they are going to be used for the positional encoding, should be stored in YYYYMMDD format in the META/dates.json file:

dates.json = {
    1: date_0,
    ...,
    T: date_T,
}

If some pre-computed static parcel features are to be used between the two MLPs of the PSE, they should be stored in another json file META/name_of_features.json:

name_of_features.json = {
    "unique_id_0": features_0,
    ...,
    "unique_id_N": features_N,
}

Folder structure

The dataset folder should thus have the follwoing structure:

Dataset_folder
│ normalisation_values.pkl
└─DATA
│    │ sample0.npy
│    │ . . .
│    │ sampleN.npy
└─META
     │ labels.json
     │ dates.json
     │ geomfeat.json

Credits

Reference

In case you use part of the present code, please include a citation to the following paper:

@article{garnot2019psetae,
  title={Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention},
  author={Sainte Fare Garnot, Vivien  and Landrieu, Loic and Giordano, Sebastien and Chehata, Nesrine},
  journal={CVPR},
  year={2020}
}