Re-train PLAsTiCC model with 'weighted' Log-Loss metric and loss for optimisation

[tallamjr]

Previous models have been optimised using the following line in astronet/t2/opt/hypertrain.py:118

        # Evaluate the model accuracy on the validation set.
        score = model.evaluate(X_val, y_val, verbose=0)
        return score[1]

However, it would be desirable to optimise this using the metric defined by AI. Malz et. al of a weighted Log-Loss

An example implementation of this can be found in astrorapids or perhaps with sklearn.metrics.log_loss akin to:

def plasticc_log_loss(y_true, probs):
    """Implementation of weighted log loss used for the Kaggle challenge.

    Parameters
    ----------
    y_true: np.array of shape (# samples,)
        Array of the true classes
    probs : np.array of shape (# samples, # features)
        Class probabilities for each sample. The order of the classes corresponds to
        that in the attribute `classes_` of the classifier used.

    Returns
    -------
    float
        Weighted log loss used for the Kaggle challenge
    """
    predictions = probs.copy()
    labels = np.unique(y_true) # assumes the probabilities are also ordered in the same way
    weights_dict = {6:1/18, 15:1/9, 16:1/18, 42:1/18, 52:1/18, 53:1/18, 62:1/18, 64:1/9,
                    65:1/18, 67:1/18, 88:1/18, 90:1/18, 92:1/18, 95:1/18, 99:1/19,
                    1:1/18, 2:1/18, 3:1/18}

    # sanitize predictions
    epsilon = sys.float_info.epsilon  # this is machine dependent but essentially prevents log(0)
    predictions = np.clip(predictions, epsilon, 1.0 - epsilon)
    predictions = predictions / np.sum(predictions, axis=1)[:, np.newaxis]

    predictions = np.log(predictions) # logarithm because we want a log loss
    class_logloss, weights = [], [] # initialize the classes logloss and weights
    for i in range(np.shape(predictions)[1]): # run for each class
        current_label = labels[i]
        result = np.average(predictions[y_true==current_label, i]) # only those events are from that class
        class_logloss.append(result)
        weights.append(weights_dict[current_label])
    return -1 * np.average(class_logloss, weights=weights)

There, the above code in astronet/t2/opt/hypertrain.py:118 would be something like follows:

        # Evaluate weighted Log Loss on the validation set.
        probs = model.predict(X_val)
        logloss = plasticc_log_loss(y_val, probs)
        return -logloss  # Minus since we want to maximise the objective

And then in astronet/t2/train.py the model.compile( .. ) would need something like:

        model.compile(
            loss="categorical_crossentropy", optimizer=optimizers.Adam(lr=lr), metrics=[plasticc_log_loss]
        )

[tallamjr]

And then in astronet/t2/train.py the model.compile( .. ) would need something like:

    model.compile(
        loss="categorical_crossentropy", optimizer=optimizers.Adam(lr=lr), metrics=[plasticc_log_loss]
    )

Should really be:

loss = plasticc_log_loss
...
        model.compile(
            loss=loss, optimizer=optimizers.Adam(lr=lr), metrics=['auc']
        )

[tallamjr]

Whilst it is now optimising using the loss function of "categorical_crossentropy" with 3b7abb85787574ee0f9, a "plasticc" implementation is not used just yet as it requires a tensorflow mathematical operation implementation as opposed to a numpy one due to this error:

    Error:
        NotImplementedError: Cannot convert a symbolic Tensor (IteratorGetNext:1) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

See 7312f7849ffb264b94eb2.

Whilst this is possible with tf-nightly, currently there is a bug that is preventing using it at the moment. Leaving this issue open until this is fixed and a full plasticc-tensorflow version of LogLoss implemented