image classifier improvements

[sdtaylor]

use some sort of time series segmentation net.

some potential useful things:

timeseries + machine learning package https://dmbee.github.io/seglearn/index.html
several img segmentation models https://github.com/divamgupta/image-segmentation-keras
the penguin video CNN paper https://www.biorxiv.org/content/10.1101/2020.06.29.177261v1.full in depth segmentation explainer: https://divamgupta.com/image-segmentation/2019/06/06/deep-learning-semantic-segmentation-keras.html

[sdtaylor]

Very good walk thru of a paper on video classifications (paper from google/stanford people).

https://towardsdatascience.com/introduction-to-video-classification-6c6acbc57356

Tests several strategies of combining frames. But each replicate still has a single classification, thus transitions between video segments are still not modeled.

paper (from 2014) https://ieeexplore.ieee.org/document/6909619

---

This one seems to emphasis the temporal nature of video classification. Long-term Recurrent Convolutional Networks for Visual Recognition and Description: https://arxiv.org/abs/1411.4389

[sdtaylor]

Conditional random fields look on the surface to be what I need. Generally applied to NLP

A very mathy tutorial on CRFs: http://www.davidsbatista.net/blog/2017/11/13/Conditional_Random_Fields/

[sdtaylor]

Ding ding ding. temporal segmentation and classification

paper1:http://arxiv.org/abs/1602.02995 paper1 code: https://github.com/colincsl/LCTM paper2: https://arxiv.org/abs/1611.05267 paper2 code:https://github.com/colincsl/TemporalConvolutionalNetworks

Note paper2 utilizes the LCTM package in ppaer1

[sdtaylor]

Also ding ding ding. A super simple timeseries classifier with LSTM & TimeDistributed.

https://stackabuse.com/solving-sequence-problems-with-lstm-in-keras-part-2/

The many-to-many one is very straightforward, assuming it can be softmax activation at the end

[sdtaylor]

two papers using markov random fields to constrain change detection to only probable transitions

A Spatial-Temporal Modeling Approach to Reconstructing Land-Cover Change Trajectories from Multi-temporal Satellite Imagery https://doi.org/10.1080/00045608.2011.596357

A spatial–temporal contextual Markovian kernel method for multi-temporal land cover mapping https://doi.org/10.1016/j.isprsjprs.2015.04.009

[sdtaylor]

Muli-label classification example
from: https://vijayabhaskar96.medium.com/multi-label-image-classification-tutorial-with-keras-imagedatagenerator-cd541f8eaf24

inp = Input(shape = (100,100,3))
x = Conv2D(32, (3, 3), padding = 'same')(inp)
x = Activation('relu')(x)
x = Conv2D(32, (3, 3))(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size = (2, 2))(x)
x = Dropout(0.25)(x)
x = Conv2D(64, (3, 3), padding = 'same')(x)
x = Activation('relu')(x)
x = Conv2D(64, (3, 3))(x)
x = Activation('relu')(x)
x = MaxPooling2D(pool_size = (2, 2))(x)
x = Dropout(0.25)(x)
x = Flatten()(x)
x = Dense(512)(x)
x = Activation('relu')(x)
x = Dropout(0.5)(x)
output1 = Dense(1, activation = 'sigmoid')(x)
output2 = Dense(1, activation = 'sigmoid')(x)
output3 = Dense(1, activation = 'sigmoid')(x)
output4 = Dense(1, activation = 'sigmoid')(x)
output5 = Dense(1, activation = 'sigmoid')(x)
model = Model(inp,[output1,output2,output3,output4,output5])
model.compile(optimizers.rmsprop(lr = 0.0001, decay = 1e-6),
loss = ["binary_crossentropy","binary_crossentropy",                                                              "binary_crossentropy","binary_crossentropy",                "binary_crossentropy"],metrics = ["accuracy"])

generator for multiple outputs
via https://classifai.net/blog/multiple-outputs-keras/

train_datagen = ImageDataGenerator(rescale=1./255,
    shear_range=0.2,
    zoom_range=0.2,
    horizontal_flip=True,
    validation_split=0.2) # set train/validation split

# custom generator
def multiple_outputs(generator, image_dir, batch_size, image_size, subset):
    gen = generator.flow_from_directory(
        image_dir,
        target_size=(image_size, image_size),
        batch_size=batch_size,
        class_mode='categorical',
        subset=subset)

    while True:
        gnext = gen.next()
        # return image batch and 3 sets of lables
        yield gnext[0], [gnext[1], gnext[1], gnext[1]]

[sdtaylor]

Can the dominant cover and crop status HMM's be done jointly? Maybe with multi-level HMM's:

nice looking tutorial & R package: https://cran.r-project.org/web/packages/mHMMbayes/vignettes/tutorial-mhmm.html

[sdtaylor]

recent paper doing sentinal2 crop classification with heavy emphasis on time series Crop Classification under Varying Cloud Cover with Neural Ordinary Differential Equations (accepted TGRS)

uses this as a baseline Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series

from the same group as the cloud cover/timeseries paper: Crop mapping from image time series: Deep learning with multi-scale label hierarchies.
multi-scale label hierarchies are the following. very cool: