By Dr Daniel Buscombe
daniel@mardascience.com
Deep learning framework for optical granulometry (estimation of sedimentological variables from sediment imagery).
A configurable machine-learning framework for estimating either (or both) continuous and categorical variables from a photographic image of clastic sediment. It has wide potential application, even to subpixel imagery and complex mixtures, because the dimensions of the grains aren't being measured directly or indirectly, but using a mapping from image to requested output using a machine learning algorithm that you have to train using examples of your data.
For more details, please see the paper:
Buscombe, D. (2019). SediNet: a configurable deep learning model for mixed qualitative and quantitative optical granulometry. Earth Surface Processes and Landforms 45 (3), 638-651. https://onlinelibrary.wiley.com/doi/abs/10.1002/esp.4760
Free Earth ArXiv preprint here
This repository contains code and data to reproduce the above paper, as well as additional examples and jupyter notebooks that you can run on the cloud and use as examples to build your own Sedinet sediment descriptor
The algorithm implementation has changed since the paper, so the results are slightly different but the concepts and data, and most of everything, have not changed.
SediNet can be configured and trained to estimate:
The motivating idea behind SediNet is community development of tools for information extraction from images of sediment. You can use SediNet "off-the-shelf", or other people's models, or configure it for your own purposes.
Within this package there are several examples of different ways it can be configured for estimating categorical variables and various numbers of continuous variables
You can use the models in this repository for your purposes (and you might find them useful because they have been trained on large numbers of images). If that doesn't work for you, you can train SediNet for your own purposes even on small datasets.
The examples have been curated with the following hardware specification in mind: 16 GB RAM, and Nvidia GPU with 11 GB of DDR4 or DDR6 memory (e.g. RTX 2080 Ti). If you have access to larger GPU memory, you can use larger imagery and larger batch sizes and you should achieve better accuracy.
Sedinet is a deep learning model, which is a type of machine learning model that uses very large neural networks to automatically extract features from data to make predictions. For imagery, network layers typically use convolutions therefore the models are called Convolutional Neural Networks or CNNs for short.
CNNs have multiple processing layers (called convolutional layers or blocks) and nonlinear transformations (that include batch normalization, activation, and dropout), with the outputs from each layer passed as inputs to the next. The model architecture is summarised below:
SediNet is very configurable, and is designed primarily to be a research tool. There are two in-built model sizes (shallow and false), and numerous options for how to train and treat the data. For example, data inputs can optionally be scaled. Various image sizes can be used. A single batch size may be chosen, or a model might be constructed using multiple batch sizes. Therefore it might take some experimentation to achieve optimal results for a particular dataset. Hopefully, this toolbox makes such experimentation straightforward. It isn't always obvious what combinations of settings to use, so be prepared to construct models using a variety of settings, then using the model with the best validation scores.
You must have python 3, pip for python 3, git and conda. On Windows I recommend the latest Anaconda release.
Windows:
git clone --depth 1 https://github.com/MARDAScience/SediNet.git
Linux/Mac:
git clone --depth 1 git@github.com:MARDAScience/SediNet.git
Anaconda/miniconda:
If you do NOT want to use your GPU for computations with tensorflow, edit the conda_env/sedinet.yml
replacing tensorflow-gpu
with tensorflow
. This is NOT recommended for training models, only using them for prediction.
(if you are a regular or long-term conda user, perhaps this is a good time to conda clean --packages
and conda update -n base conda
?)
conda env create -f conda_env/sedinet.yml
conda activate sedinet
(Later, when you're done ... conda deactivate sedinet
)
The following examples have been selected to demonstrate the range of options you can choose when optimizing a SediNet model for a particular dataset. It therefore serves as a guide, rather than a gallery of best possible model outcomes. I encourage you to experiment with a few sets of options before deciding on a final optimal configuration and defaults file. Sometimes, using multiple batch sizes can be advantageous.
python sedinet_train.py -c config/config_9percentiles.json
Subsequently predict using:
python sedinet_predict.py -c config/config_9percentiles.json -1 grain_size_global/res/global_9prcs_simo_batch12_im768_768_9vars_pinball_noaug.hdf5 -2 grain_size_global/res/global_9prcs_simo_batch13_im768_768_9vars_pinball_noaug.hdf5 -3 grain_size_global/res/global_9prcs_simo_batch14_im768_768_9vars_pinball_noaug.hdf5
The above model has been trained with multiple batch size of 12, 13 and 14, with 768x768 pixel imagery, no augmentation, and no variable scaling
To use the model to predict on a single image:
python sedinet_predict1image.py -c config/config_9percentiles.json -i images/Cal_16.tif -1 grain_size_global/res/global_9prcs_simo_batch12_im768_768_9vars_pinball_noaug.hdf5 -2 grain_size_global/res/global_9prcs_simo_batch13_im768_768_9vars_pinball_noaug.hdf5 -3 grain_size_global/res/global_9prcs_simo_batch14_im768_768_9vars_pinball_noaug.hdf5
To use the model to predict on all images in a folder:
python sedinet_predictfolder.py -c config/config_9percentiles.json -w grain_size_global/res/global_9prcs_simo_batch14_im768_768_9vars_pinball_noaug.hdf5 -i images/
python sedinet_train.py -c config/config_sievedsand_sieve_plus.json
Subsequently predict using:
python sedinet_predict.py -c config/config_sievedsand_sieve_plus.json -w grain_size_sieved_sands/res_sieve_plus/sievesand_sieve_plus_simo_batch8_im512_512_6vars_pinball_aug_scale.hdf5
The above model has been trained with a single batch size of 8, with 768x768 pixel imagery, augmentation, and scaling
To use the model to predict on a single image:
python sedinet_predict1image.py -c config/config_sievedsand_sieve_plus.json -w grain_size_sieved_sands/res_sieve_plus/sievesand_sieve_plus_simo_batch8_im512_512_6vars_pinball_aug_scale.hdf5 -i images/IMG_0214.JPG
python sedinet_train.py -c config/config_sievedsand_sieve.json
Subsequently predict using:
python sedinet_predict.py -c config/config_sievedsand_sieve.json -w grain_size_sieved_sands/res_sieve/sievesand_sieve_siso_batch7_im512_512_1vars_pinball_aug_scale.hdf5
The above model has been trained with a single batch size of 8, with 768x768 pixel imagery, augmentation, and scaling
To use the model to predict on a single image:
python sedinet_predict1image.py -c config/config_sievedsand_sieve.json -w grain_size_sieved_sands/res_sieve/sievesand_sieve_siso_batch7_im512_512_1vars_pinball_aug_scale.hdf5 -i images/IMG_0214.JPG
python sedinet_train.py -c config/config_pop.json
Subsequently predict using:
python sedinet_predict.py -c config/config_pop.json -1 grain_population/res/grain_population_siso_batch3_im768_768_pop_focal_noaug.hdf5 -2 grain_population/res/grain_population_siso_batch4_im768_768_pop_focal_noaug.hdf5 -3 grain_population/res/grain_population_siso_batch6_im768_768_pop_focal_noaug.hdf5 -4 grain_population/res/grain_population_siso_batch8_im768_768_pop_focal_noaug.hdf5
The above model has been trained with multiple batch size of 3, 4, and 6, with 768x768 pixel imagery, no augmentation, and no variable scaling (by default for categorical variables)
To use the model to predict on a single image:
python sedinet_predict1image.py -i images/um125_179_1.jpg -c config/config_pop.json -w grain_population/res/grain_population_siso_batch3_im768_768_pop_focal_noaug.hdf5
python sedinet_train.py -c config/config_shape.json
Subsequently predict using:
python sedinet_predict.py -c config/config_shape.json -1 grain_shape/res/grain_shape_siso_batch6_im768_768_shape_focal_noaug.hdf5 -2 grain_shape/res/grain_shape_siso_batch8_im768_768_shape_focal_noaug.hdf5 -3 grain_shape/res/grain_shape_siso_batch10_im768_768_shape_focal_noaug.hdf5
The above model has been trained with multiple batch size of 6, 8 and 10, with 768x768 pixel imagery, no augmentation, and no variable scaling (by default for categorical variables)
To use the model to predict on a single image (remember to change the BATCH_SIZE
to a list):
python sedinet_predict1image.py -c config/config_shape.json -i images/Cal_16.tif -1 grain_shape/res/grain_shape_siso_batch6_im768_768_shape_focal_noaug.hdf5 -2 grain_shape/res/grain_shape_siso_batch8_im768_768_shape_focal_noaug.hdf5 -3 grain_shape/res/grain_shape_siso_batch10_im768_768_shape_focal_noaug.hdf5
python sedinet_train.py -c config/config_gravel.json
Subsequently predict using:
python sedinet_predict.py -c config/config_gravel.json -w grain_size_gravel_generic/res/gravel_generic_9prcs_simo_batch6_im768_768_9vars_pinball_aug.hdf5
The above model has been trained with a batch size of 6, with 768x768 pixel imagery, augmentation, and no variable scaling
To use the model to predict on a single image:
python sedinet_predict1image.py -c config/config_gravel.json -i images/Cal_16.tif -w grain_size_gravel_generic/res/gravel_generic_9prcs_simo_batch6_im768_768_9vars_pinball_aug.hdf5
python sedinet_train.py -c config/config_sand.json
Subsequently predict using:
python sedinet_predict.py -c config/config_sand.json -w grain_size_sand_generic/res_9prcs/sand_generic_9prcs_simo_batch12_im768_768_9vars_pinball_noaug_scale.hdf5
The above model has been trained with a batch size of 12, with 768x768 pixel imagery, no augmentation, and variable scaling
To use the model to predict on a single image:
python sedinet_predict1image.py -c config/config_sand.json -i images/IMG_1591_1400microns.JPG -w grain_size_sand_generic/res_9prcs/sand_generic_9prcs_simo_batch12_im768_768_9vars_pinball_noaug_scale.hdf5
python sedinet_train.py -c config/config_sand_3prcs.json
Subsequently predict using:
python sedinet_predict.py -c config/config_sand_3prcs.json -w grain_size_sand_generic/res_3prcs/sand_generic_3prcs_simo_batch12_im768_768_3vars_pinball_noaug_scale.hdf5
The above model has been trained with a batch size of 12, with 768x768 pixel imagery, no augmentation, and variable scaling
To use the model to predict on a single image:
python sedinet_predict1image.py -c config/config_sand_3prcs.json -i images/IMG_1591_1400microns.JPG -w grain_size_sand_generic/res_3prcs/sand_generic_3prcs_simo_batch12_im768_768_3vars_pinball_noaug_scale.hdf5
python sedinet_train.py -c config/mattole.json
Subsequently predict using:
python sedinet_predict.py -c config/config_mattole.json -w mattole/res/mattole_simo_batch7_im512_512_2vars_pinball_aug.hdf5
The above model has been trained with a batch size of 7, with 768x768 pixel imagery, augmentation, and no variable scaling
To use the model to predict on a single image:
python sedinet_predict1image.py -c config/config_mattole.json -i images/mattole_images/all/DSCN3521c.JPG -w mattole/res/mattole_simo_batch7_im512_512_2vars_pinball_aug.hdf5
A typical SediNet model configuration for predicting categorical variables is:
A typical SediNet model configuration for predicting continuous variables is:
Contains values for defaults that you may change. They are listed in order of likelihood that you might change them:
# size of image in pixels. keep this consistent in training and application
# suggestd: 512 -- 1024 (larger = larger GPU required)
# integer
IM_HEIGHT = 768 #1024
IM_WIDTH = IM_HEIGHT #1024 #IM_HEIGHT
# number of images to feed the network per step in epoch #suggested: 4 --16
# integer
#BATCH_SIZE = 7
#use an ensemble of batch sizes like this
BATCH_SIZE = [4,6,8]
# if True, use a smaller (shallower) network architecture
##True or False ##False=larger network
SHALLOW = False #True
## if True, carry out data augmentation. 2 x number of images used in training
##True or False
DO_AUG = False #True
# maximum learning rate ##1e-1 -- 1e-4
MAX_LR = 1e-3
# max. number of training epics (20 -100)
# integer
NUM_EPOCHS = 100
## loss function for continuous models (2 choices)
CONT_LOSS = 'pinball'
#CONT_LOSS = 'mse'
## loss function for categorical (disrete) models (2 choices)
CAT_LOSS = 'focal'
#CAT_LOSS = 'categorical_crossentropy'
# optimizer (gradient descent solver) good alternative == 'rmsprop'
OPT = 'rmsprop' #'adam'
# base number of conv2d filters in categorical models
# integer
BASE_CAT = 30
# base number of conv2d filters in continuous models
# integer
BASE_CONT = 30
# number of Dense units for continuous prediction
# integer
CONT_DENSE_UNITS = 1024 #512
# number of Dense units for categorical prediction
# integer
CAT_DENSE_UNITS = 128
For continuously distributed variables, file names are constructed according to the following convention
name "_" mode "_batch" batch_size "_im" IM_HEIGHT "_shallow_" varstring "_" CONT_LOSS "_aug_scale.hdf5"
if imagery is not augmented, aug
in the above is replaced with noaug
. If variables are not scaled, _scale
is missing from the end
For categorical variables, we use
name "_" mode "_batch" batch_size "_im" IM_HEIGHT "_shallow_" varstring "_" CAT_LOSS "_aug.hdf5"
if imagery is not augmented, aug
in the above is replaced with noaug
. Categorical variables are never scaled
SediNet is very configurable. You can specify many variables in the config file, from the size of the imagery to use, to the number of models to ensemble and their respective batch sizes.
The SediNet training function train_sedinet_continuous.py
is set up to predict arbitrary numbers of continuous variables. All your specific information (what data set to use, what to predict, etc) is contained in the config file and called the same way as above. For example:
python train_sedinet_continuous.py -c config/config_custom_4prcs.json
where config/config_custom_4prcs.json
has ben put together by you in the config folder like the following example that would estimate the mean grain size and 4 arbitrary percentiles:
{
"train_csvfile" : "your_train_dataset.csv",
"test_csvfile" : "your_test_dataset.csv",
"mean" : "mean",
"P20": "P20",
"P40": "P46",
"P60": "P60",
"P80": "P80",
"res_folder": "my_custom_model",
"name" : "custom_4prcs",
"dropout": 0.5,
"scale": false
}
The program will still expect your images to reside inside the 'images' folder
You must label the file names in your csv file the same way as in the examples, i.e. "images/yourfilename.ext" and that column must be labeled 'files'
Put together a config file in the config folder (called, say config_custom_colour.json
) and populate it like this example:
{
"csvfile" : "dataset_colour.csv",
"var" : "colour",
"numclass" : 6,
"res_folder": "grain_colour",
"name": "grain_colour",
"dropout": 0.5,
}
Notes:
Release v1.0 (Sep 30 2019): initial submission of SediNet paper to journal
Release v1.1 (Nov 5 2019): upgrade from keras with Tensorflow 1.X backend to Tensorflow 2.0 native keras. Enforce TF==2.0 in conda yml file
Release v1.2 (Feb 4 2020): major upgrade with the following improvements: 1) Additional dataset (Mattole); a mixed sand-gravel beach data set collected in summer 2019 by Sarah Joerger as part of her MS in Geology at Northern Arizona University 2) Robust continuous variable scaling; all response variables now get scaled using scikit-learn's RobustScaler, which removes the median and scales the data according to the quantile range. See https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html. . This information is contained in the scaler.pkl file. Categorical model training is unchanged 3) Removed user requirement in previous versions for specifying base model size; previously, the overall size of the model was controlled by a variable
base
, specifying the minimum number of filters in the first convolutional block. Now, that number is fixed (30) and the user has an option to make the model more or less shallow, which has the effect of adding an additional convolutional block. This is controlled usingshallow = True/False
indefaults.py
. Applies to both categorical and continuous variables 4) Removed bias in continuous model predictions; the final step is to fit a linear model to (x,y-predicted) to remove any small biases left in predictions. This information is contained in the bias.pkl file. Categorical model training is unchanged 5) 1024x1024 image size used by default (larger than previous). For color imagery on smaller GPUs, you may have to compromise between batch size and image size. I did the latter, using 768x768x3 pixels 6) The user now creates two csv files -- one for training files and associated values, and one for testing files and their associated variables 7) Sedinet no longer creates (uses) ensembles of models (model predictions), which speeds up model training and execution. Applies to both categorical and continuous variables 8) Set things up so multiple inputs can be used to predict outputs (i.e. imagery plus an attribute) -- however this is not yet and may not ever be implemented 9) Default now uses rmsprop optimizer (instead of adam) 10) Color as well as greyscale imagery (user optional) 11) Simpler workflow oftrain
followed bypredict
- no longer any separate scripts for continous and categorical variables. Addingnumclasses
to the config file tells the project to use a categorical variable workflow 12) Use of GPU is controlled byuse_gpu
= True \ False in thedefaults.py
scriptUpdate 26 June 2020: 1) updated yml file and tested env on linux (pop!OS, ubuntu) and windows (10) 2) updated README 3) switched from deprecated
from sklearn.externals import joblib
tojoblib
4) added a folder of updated jupyter notebooksRelease v1.3 Update July 2020: 1) fixed generator error by adding exception and print statement to
get_data_generator_Nvars_siso_simo
,get_data_generator_Nvars_miso_mimo
, andget_data_generator_1image
2) new optimiseddefaults.py
values (image size = 768, batch_size = 4, shallow=True) 3) addedBASE_CAT
andBASE_CONT
options todefaults.py
4) added image size, model depth, to output file names 5) addedCAT_DENSE_UNITS
andCONT_DENSE_UNITS
options todefaults.py
6) addedCONT_LOSS
andCAT_LOSS
options todefaults.py
, with defaults fromtensorflow_addons
(conda env yml updated). loss for categorical models can now befocal
(default) orcategorical_crossentropy
. Models for continuous variables can now bepinball
(default) 7) all global variables are now capitalized, for readability/etc 8) general tidy up of code for readability 9) fixed bug in categorical model training (expected ndarray, not list, for batch labels) 10) fixed bug in categorical model plotting 11) added LICENSE 12) now can take multiple batch sizes and build an ensemble model. This generally results in higher accuracy but more models = more model training time 13) response variables can be scaled using a robust scaler, or not. Usescale=True
in a config file to use scaling 14) now checks for estimating weights path in root andres_folder
directory and, if present, uses it. This can be used to add batch size combinations sequentially 15) optionally, training imagery is now augmented ifDO_AUG=True
(in the defaults file). This doubles the training set, by augmenting each image (random horizontal shift, followed by a vertical flip) 16) file names shorter (number of variables enumerated rather than each listed) 17) improved/rewritten README 18) more consistent and descriptive filenaming convention 19) simpler structure:train
only does training (no prediction). Usepredict
to get train and test sets evaluation. This also allows defaulting to CPU for prediction, to avoid OOM errors that are more likely using GPU for prediction 20) no separate config file for prediction. One config file for both training and prediction 21) fixed many bugs, including one that was using 3-band greyscale imagery (doh!) 22) uses an exponentially decreasing learning rate scheduler rather than adaptive (because validation loss can be erratic) 23) uses depthwise separable 2d convolutions rather than trad 2d convs. see here 24) variables indefaults.py
based on consideration of accuracy across many datasets, both included and not included as pat of the SediNet package 25) categorical models also have a shallow and false option 26)predict_all.sh
is a fully worked example of using the framework to predict on all continuous datasets 27) simplified yml conda env, and a requirements.txt 28) addedsedinet_predict1image.py
for making predictions on a single image 28) addedsedinet_predictfolder.py
for making predictions on a folder of imagesThe most important changes area
- depthwise separable convolution layers
- exponentially decreasing learning rate scheduler
- pinball loss for continuous variables
- focal loss and "shallow=False" for categorical variables
- training and prediction using model ensembles trained with up to 4 different batch sizes
Note that you will see different results than in the paper because
After long training periods, especially with multiple batch sizes, the train
script gets killed at the end when it tries to use the model(s) in prediction mode. It is unclear why this happens. However, if you run the script again, with the same everything, this time it will skip the model training (assuming the hdf5
files are still in the root directory or in res_folder
- you'll see a Loading weights that already exist:
message) and use the model weights to predict.
Please use the 'issues' tab so everyone can see the question and answer. Please do not email me directly. Thanks
If you find this useful for your research please cite this paper:
Buscombe, D. (2019). SediNet: a configurable deep learning model for mixed qualitative and quantitative optical granulometry. Earth Surface Processes and Landforms 45 (3), 638-651. https://onlinelibrary.wiley.com/doi/abs/10.1002/esp.4760
Thanks to the following individuals for donating imagery:
In order of trial:
BATCH_SIZE
in defaults.py
)DO_AUG = False
in defaults.py
)IM_WIDTH
and IM_HEIGHT
in defaults.py
)SediNet is organized as follows:
Model training
sedinet_train.py
is called, it first sets an operating system environmental variable that controls the use or otherwise of the GPU. It uses GPU 0 if use_GPU=True, otherwise GPU -1 (shorthand for CPU)sedinet_infer
which import everything in sedinet_models
, so on for sedinet_utils
, and finally imports
imports
sets global variables and reads the defaults.py
sedinet_train.py
reads the specified (at the command line) config
file, organizes the config variables, and finally calls run_training_siso_simo
from sedinet_infer
run_training_siso_simo
runs either make_cat_sedinet
to make a categorical model, or make_sedinet_siso_simo
to make a continuous model (both called from sedinet_models
)train_sedinet_siso_simo
for continuous model training, or train_sedinet_cat
for categorical (both called from sedinet_infer
)predict_test_train_siso_simo
or predict_test_train_cat
for cont/cat variables (both called from sedinet_utils
) and tidy
moves the files into the res_folder
, specified in the config
fileModel prediction
config
file, csv_file
and weights_file
(or up to 4 weights files), use the model defined in the config file, load the weights, and estimate the variables on the images listed in the csv filesedinet_predict.py
is called, it first sets an operating system environmental variable that controls the use or otherwise of the GPU. It uses GPU 0 if use_GPU=True, otherwise GPU -1 (shorthand for CPU)sedinet_eval
which import everything in sedinet_models
, so on for sedinet_utils
, and finally imports
imports
sets global variables and reads the defaults.py
sedinet_predict.py
reads the specified (at the command line) config
file, and weights file, organizes the config variables, and finally calls estimate_siso_simo
or estimate_categorical
from sedinet_eval
estimate_siso_simo
runs make_sedinet_siso_simo
to make a continuous model. estimate_categorical
runs make_cat_sedinet
to make a categorical model (both called from sedinet_models
)predict_test_train_siso_simo
for continuous model training, or predict_test_train_cat
for categorical (both called from sedinet_utils
) and finally tidy
moves the files into the res_folder
, specified in the config
fileModel prediction on sample imagery
sedinet_predict_1image.py
and sedinet_predictfolder.py
are now provided. These scripts allow you to point a model to an image or image folder, resulting in a csv file of requested outputs per imageIf you wish to contribute to the development of this project (yes please!) it is better that you first fork this repository to your own github, then work on changes, and submit a pull request. Before submitting, please test your code changes by running a full set of tests in predict_all.sh
, then verifying they all executed without error.
You can also contribute imagery this way, but if you do so, also please provide a dataset (csv file) that goes along with the imagery, a file that describes the data with your name and contact details, (and you should also thank yourself in this README!)
First, follow instructions here for how to set up an instance to run in GCP. Make sure to set a static IP address, as per the instructions, and make a note of that because you'll need it later
Then open a shell into the VM and set it up to
ssh-keygen -t rsa -b 4096 -C "yourname@youremail.com"
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub
Then copy the key into your github profile keys. For more information about how to do that, see here. xclip likely won't work, but you can simply copy (Ctrl-C) the text printed to screen
You will be cloning your fork of the main repo, so replace YOURUSERNAME
in the below code to clone the repo and set up a conda environment to run in
git clone --depth 1 git@github.com:YOURUSERNAME/SediNet.git
cd SediNet
pip install --upgrade pip
conda env create -f conda_env/sedinet.yml
source activate sedinet
Now you can run sedinet on the cloud.
To run the jupyter notebooks, run the following command to run the jupyter notebook server
python -m ipykernel install --user
jupyter notebook --NotebookApp.iopub_data_rate_limit=10000000
The jupyterlab server will be displayed at
http://IP:8888
where IP
is the static IP of the VM that you noted earlier.