remicres
commented
1 year ago Hi Brayden,
Sorry for the long reply! This is basically part of the new doc, updated very soon.
TL;DR
The PatchesExtraction application performs the extraction of patches in one or multiple sources, that cover the same area.
When you have different sites, then you have to run PatchesExtraction over each site.
Then you can use the OTBTF Python API. It is quite new, and we made a huge step toward a very easy use with tf.keras. The documentation is incoming, you can check the work in progress here (it's nearly finished, maybe just a few refactoring without any impact on the things)
We should publish the new release (3.3.0) before wednesday! In the mean time, you can give a try on the development feature branch docker image: docker pull gitlab-registry.irstea.fr/remi.cresson/otbtf:29-add-a-model-class-in-otbtf (CPU only... the GPU flavored docker images will be publish on dockerhub with the 3.3.0 release next week)
Details
Here is what you can do.
Say you have two XS images acquired over A and B sites, with some terrain truth.
First you have extracted your patches, and the output of PatchesExtraction results in the following files:
- xs_A.tif
- xs_B.tif
- labels_A.tif
- labels_B.tif
Then you can use the OTBTF Python API:
from otbtf import DatasetFromPatchesImages
ds = DatasetFromPatchesImages(filenames_dict={"xs_patches": ["xs_A", "xs_B"],
"labels_patches": ["labels_A", "labels_B"]})
tf_ds = ds.get_tf_dataset(batch_size=8) # This is a TensorFlow dataset !Such TensorFlow dataset can be used using the TensorFlow v1 or v2 API.
If you want to use keras, it's more convenient that your dataset splits its samples in two (inputs_dict, target_dict). You just have to pass the target name to the targets_keys parameter of get_tf_dataset to get the samples in the right format for keras:
# This is a TensorFlow dataset, ready to be used with keras !
tf_ds = ds.get_tf_dataset(batch_size=8, targets_keys="predictions")From here, you can build and train your model with Keras.
But I strongly encourage you to use the new otbtf.model.ModelBase, it really helps to build a model which has the good settings to be used later in TensorflowModelServe for inference in production.
For instance, here is a small model implementation:
from otbtf.model import ModelBase
import tensorflow as tf
class MyModel(ModelBase):
def normalize_inputs(self, inputs):
""" Normalize the remote sensing image into [0, 1] range prior to convolutions, etc.
Optional, but quite convenient in production. Return dict of normalized inputs """
return {"xs": tf.cast(inputs["xs"], tf.float32) * 0.0001} # Say input is uint16
def get_outputs(self, normalized_inputs):
""" Build the model. Return the dict of outputs (must include target!) """
inp = normalized_inputs["xs"]
net = tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation="relu")(inp)
net = tf.keras.layers.Conv2D(filters=64, kernel_size=3, activation="relu")(net)
net = tf.keras.layers.Conv2D(filters=N_CLASSES, kernel_size=3)(net)
predictions = tf.keras.layers.Softmax(name="predictions_softmax")(net)
return {"predictions": predictions}**One last thing is to prepare the labels in one hot encoding, in the case you have integer values in your labels. For instance, you can do:
def dataset_preprocessing_fn(examples):
def _to_categorical(x):
return tf.one_hot(tf.squeeze(tf.cast(x, tf.int32), axis=-1), depth=N_CLASSES)
return {"xs": examples["xs_patches"], "predictions": _to_categorical(examples["labels_patches"])}Then you ask the dataset to (1) use this function to prepare the input patches so they match the model input and target names, (2) split the dataset examples in (inputs_dict, target_dict) so it can be used in keras logic.
tf_ds = ds.get_tf_dataset(batch_size=batch_size,
preprocessing_fn=dataset_preprocessing_fn,
targets_keys="predictions")Then you just have to train the model like any other keras model.
strategy = tf.distribute.MirroredStrategy() # For single or multi-GPUs
with strategy.scope():
model = MyModel(dataset_element_spec=tf_ds.element_spec)
model.compile(loss=tf.keras.losses.CategoricalCrossentropy(),
optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4))
model.fit(ds_train, epochs=params.nb_epochs, validation_data=ds_valid)
model.save("/path/to/savedmodel")In production you want to use the cropped outputs (named <node_name>_crop<int_crop_value> (in doubt, you can retrieve their names using saved_model_cli show --dir /path/to/savedmodel --all) to avoid blocking artifacts of your fully convolutional model (that's another story, we won't cover it here, but you know that convolutions have side effects. The otbtf.model.ModelBase comes with automatically added extra outputs to avoid blocking artifacts in inference production pipelines).
In CLI you will typically do:
otbcli_TensorflowModelServe \
-source1.il "fullsize_xs_image.tif" \
-source1.rfieldx 64 \
-source1.rfieldy 64 \
-source1.placeholder "xs" \
-model.dir "/path/to/savedmodel" \
-model.fullyconv on \
-output.names "prediction_softmax_crop16" \
-output.efieldx 32 \
-output.efieldy 32 \
-out "output.tif"Hope that helps, and that you will enjoy the new API. Also do not hesitate to tell us if something isn't going as expected, or if you found some nasty bug :+1:
Regards,
Rémi
bjyberg
Hi Remi (or whomever else knows), I have data from two different sites which I could like to use to create a single training dataset for a CNN. Is this possible using the patch extraction tool? If not, is it possible to input two sets of patches to train a CNN within OTBTF? Cheers, Brayden