Logistic regression : Accuracy of 1.0 (is there an error in my code ?)

[MohamedLEGH]

Description

Hello. I'm trying to build a simple logistic model with DJL. I have an accuracy of 1 with my logistic regression model.

Expected Behavior

I expected to have around 0.92 (as with PyTorch).

Below you will find my Java code :

package centralized;
import java.io.IOException;

import tech.tablesaw.api.Table;
import ai.djl.Model;
import ai.djl.ndarray.NDManager;
import ai.djl.ndarray.NDArray;
import ai.djl.nn.SequentialBlock;
import ai.djl.nn.core.Linear;
import ai.djl.training.dataset.ArrayDataset;
import ai.djl.training.dataset.RandomAccessDataset;
import ai.djl.training.loss.Loss;
import ai.djl.training.loss.SigmoidBinaryCrossEntropyLoss;
import ai.djl.training.tracker.Tracker;
import ai.djl.training.optimizer.Optimizer;
import ai.djl.training.TrainingConfig;
import ai.djl.training.DefaultTrainingConfig;
import ai.djl.training.Trainer;
import ai.djl.training.evaluator.Accuracy;
import ai.djl.training.TrainingResult;
import ai.djl.training.listener.TrainingListener;
import ai.djl.metric.Metrics;
import ai.djl.training.EasyTrain;
import ai.djl.ndarray.types.Shape;
import ai.djl.translate.TranslateException;

public class Centralized {
    public static ArrayDataset loadArray(NDArray features, NDArray labels, int batchSize, boolean shuffle) {
        return new ArrayDataset.Builder()
                    .setData(features) // set the features
                    .optLabels(labels) // set the labels
                    .setSampling(batchSize, shuffle) // set the batch size and random sampling
                    .build();
    }

    public static void main(String[] args) throws IOException,TranslateException {
        Table spambase = Table.read().csv("spambase.csv");
        Table inputs = spambase.copy().removeColumns("is_spam");
        Table outputs = spambase.copy().retainColumns("is_spam");
        NDManager manager = NDManager.newBaseManager();
        NDArray x = manager.create(inputs.as().floatMatrix());
        NDArray y = manager.create(outputs.as().intMatrix());
        int batchSize = 10;
        ArrayDataset dataset = loadArray(x, y, batchSize, true);
        RandomAccessDataset[] datasets_split = dataset.randomSplit(80, 20);
        RandomAccessDataset trainingSet = datasets_split[0];
        RandomAccessDataset validationSet = datasets_split[1];
        Model model = Model.newInstance("logistic-regression");
        SequentialBlock net = new SequentialBlock();
        Linear linearBlock = Linear.builder().optBias(true).setUnits(1).build();
        net.add(linearBlock);
        model.setBlock(net);
        Loss loss = new SigmoidBinaryCrossEntropyLoss();

        Tracker lrt = Tracker.fixed(0.1f);
        Optimizer sgd = Optimizer.sgd().setLearningRateTracker(lrt).build();

        TrainingConfig config = new DefaultTrainingConfig(loss)
            .optOptimizer(sgd) // Optimizer
            .optDevices(manager.getEngine().getDevices(0)) // CPU
            .addEvaluator(new Accuracy()) // Model Accuracy
            .addTrainingListeners(TrainingListener.Defaults.logging()); // Logging

        Trainer trainer = model.newTrainer(config);

        trainer.initialize(new Shape(1, inputs.columnCount())); 
        Metrics metrics = new Metrics();
        trainer.setMetrics(metrics);

        int numEpochs = 1;

        EasyTrain.fit(trainer, numEpochs, trainingSet, validationSet);
        // System.out.println(metrics.getMetricNames());
        double accuracy = metrics.mean("validate_epoch_Accuracy");
        System.out.println(accuracy);

        TrainingResult result = trainer.getTrainingResult();
        System.out.println(result);
    }
}

This is the output:

> Task :app:runCentralized
SLF4J(I): Connected with provider of type [org.slf4j.jul.JULServiceProvider]
avr. 24, 2024 4:19:39 AM ai.djl.pytorch.engine.PtEngine newInstance
INFOS: PyTorch graph executor optimizer is enabled, this may impact your inference latency and throughput. See: https://docs.djl.ai/docs/development/inference_performance_optimization.html#graph-executor-optimization
avr. 24, 2024 4:19:39 AM ai.djl.pytorch.engine.PtEngine newInstance
INFOS: Number of inter-op threads is 8
avr. 24, 2024 4:19:39 AM ai.djl.pytorch.engine.PtEngine newInstance
INFOS: Number of intra-op threads is 12
avr. 24, 2024 4:19:39 AM ai.djl.training.listener.LoggingTrainingListener onTrainingBegin
INFOS: Training on: cpu().
avr. 24, 2024 4:19:39 AM ai.djl.training.listener.LoggingTrainingListener onTrainingBegin
INFOS: Load PyTorch Engine Version 2.1.1 in 0,031 ms.
Training:    100% |████████████████████████████████████████| Accuracy: 1,00, SigmoidBinaryCrossEntropyLoss: 1155,97
Validating:  100% |████████████████████████████████████████|
avr. 24, 2024 4:19:39 AM ai.djl.training.listener.LoggingTrainingListener onEpoch
INFOS: Epoch 1 finished.
avr. 24, 2024 4:19:39 AM ai.djl.training.listener.LoggingTrainingListener onEpoch
INFOS: Train: Accuracy: 1,00, SigmoidBinaryCrossEntropyLoss: 1148,56
avr. 24, 2024 4:19:39 AM ai.djl.training.listener.LoggingTrainingListener onEpoch
INFOS: Validate: Accuracy: 1,00, SigmoidBinaryCrossEntropyLoss: 1097,44
1.0
{
  "epoch": 1,
  "evaluations": {
    "validate_Accuracy": 1.0,
    "train_SigmoidBinaryCrossEntropyLoss": 1148.5554,
    "validate_SigmoidBinaryCrossEntropyLoss": 1097.441,
    "train_Accuracy": 1.0,
    "train_loss": 1148.5554,
    "validate_loss": 1097.441
  }
}

I created a similar model in PyTorch:

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder

df = pd.read_csv("spambase.csv")
data = df.to_numpy()
y = LabelEncoder().fit_transform(data[:, 57])
X = np.delete(data, [57], axis=1).astype('float64')
X = StandardScaler().fit_transform(X)
X = torch.tensor(X).float()
y = torch.tensor(y).float().reshape(-1, 1)
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, shuffle=True)
loader = DataLoader(list(zip(X_train, y_train)), shuffle=True, batch_size=10)

n_inputs = 57
n_outputs = 1

model = nn.Sequential(
    nn.Linear(n_inputs, n_outputs),
    nn.Sigmoid()
)

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
loss_fn = nn.BCELoss()
n_epochs = 1

model.train()
for epoch in range(n_epochs):
    for X_batch, y_batch in loader:
        y_pred = model(X_batch)
        loss = loss_fn(y_pred, y_batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

# evaluate accuracy after training
model.eval()
y_pred = model(X_test)
acc = (y_pred.round() == y_test).float().mean()
acc = float(acc)
print("Model accuracy: %.2f" % acc)

And the output:

Model accuracy: 0.92

Is there an error with my Java/DJL code ? I tried to follow the "Dive into Deep Learning" book and the documentation but I'm not sure of my code. Are my Java and Python codes equivalent?

ps: The dataset is from https://archive.ics.uci.edu/dataset/94/spambase, and I transformed it into a csv.

Thanks

[MohamedLEGH]

For information this is the model parameters before learning (if you know how to write all the values instead of "37 more" please tell me):

weight: (1, 57) cpu() float32 hasGradient
[[-0.0309, -0.0797,  0.3092, -0.1011,  0.2231,  0.118 ,  0.1488,  0.0543, -0.5098, -0.4196, -0.1896,  0.0717,  0.0586, -0.0586,  0.0395,  0.0914,  0.3283, -0.1463, -0.3702,  0.1664, ... 37 more],
]

bias: (1) cpu() float32 hasGradient
[0.]

And this is after training:

weight: (1, 57) cpu() float32 hasGradient
[[ -0.3021,  -3.2209,  -0.8564,   0.5633,   0.0698,   0.1584,   1.1403,   0.4022,  -0.3137,  -1.5199,  -0.0627,  -5.3316,  -0.1716,  -0.3107,   0.3686,   1.119 ,   0.8939,  -0.0675,  -8.1434,   0.9069, ... 37 more],
]

bias: (1) cpu() float32 hasGradient
[-9.5678]

I tried to use Predictor to check the values predicted, with this code

Translator translator = new NoopTranslator();
Predictor predictor = model.newPredictor(translator);

for(Batch b: validationSet.getData(manager)) {
    NDList data = b.getData();
    NDArray prediction = ((NDList) predictor.predict(data)).singletonOrThrow();
    NDArray truelabel = b.getLabels().singletonOrThrow();

    System.out.println("Predicted is: " + prediction.toString());
    System.out.println("True value is: " + truelabel.toString());
}

And this is the result (one batch but it's similar for all batch):

Predicted is: ND: (10, 1) cpu() float32
[[-10483.418 ],
 [ -1452.9528],
 [ -7738.9502],
 [  -709.6656],
 [ -1241.8141],
 [ -3426.0046],
 [  -188.6846],
 [  -580.9608],
 [   -98.5921],
 [ -1608.1211],
]

True value is: ND: (10, 1) cpu() int32
[[ 1],
 [ 0],
 [ 1],
 [ 0],
 [ 0],
 [ 1],
 [ 1],
 [ 0],
 [ 0],
 [ 1],
]

Then I tried to use a sigmoid operations on the predicted values, like this:

for(Batch b: validationSet.getData(manager)) {
    NDList data = b.getData();
    NDArray prediction = ((NDList) predictor.predict(data)).singletonOrThrow();
    NDArray prediction_binary = Activation.sigmoid(prediction);
    NDArray truelabel = b.getLabels().singletonOrThrow();

    System.out.println("Predicted is: " + prediction_binary.toString());
    System.out.println("True value is: " + truelabel.toString());
}

and results are like this :

Predicted is: ND: (10, 1) cpu() float32
[[1.    ],
 [1.    ],
 [1.    ],
 [1.    ],
 [1.    ],
 [1.    ],
 [1.    ],
 [0.0012],
 [1.    ],
 [1.    ],
]

True value is: ND: (10, 1) cpu() int32
[[ 0],
 [ 0],
 [ 0],
 [ 1],
 [ 0],
 [ 0],
 [ 0],
 [ 0],
 [ 1],
 [ 1],
]

Any idea where the problem is?

[adinath1233]

Hey @MohamedLEGH, I have question is NoopTranslator() is used for only for classification or can used for the Regression?

as well this is because weights I guess try to StandardScale or min max scaling on the input data.

[MohamedLEGH]

Ok, I have updated the code, I made the following modifications:

• The data are normalized (remove mean and divide by standard deviation)
• The BinaryAccuracy is used instead of Accuracy

Below you find the code updated :

package machine_learning;

import java.io.IOException;
import java.util.Random;

import tech.tablesaw.api.Table;
import ai.djl.Model;
import ai.djl.ndarray.NDManager;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDList;
import ai.djl.ndarray.types.DataType;
import ai.djl.nn.SequentialBlock;
import ai.djl.nn.core.Linear;
import ai.djl.nn.Parameter;
import ai.djl.nn.Activation;
import ai.djl.training.dataset.ArrayDataset;
import ai.djl.training.dataset.RandomAccessDataset;
import ai.djl.training.loss.Loss;
import ai.djl.training.loss.SigmoidBinaryCrossEntropyLoss;
import ai.djl.training.tracker.Tracker;
import ai.djl.training.optimizer.Optimizer;
import ai.djl.training.TrainingConfig;
import ai.djl.training.DefaultTrainingConfig;
import ai.djl.training.Trainer;
import ai.djl.training.evaluator.Accuracy;
import ai.djl.training.evaluator.BinaryAccuracy;
import ai.djl.training.TrainingResult;
import ai.djl.training.listener.TrainingListener;
import ai.djl.training.dataset.Batch;
import ai.djl.training.EasyTrain;
import ai.djl.training.initializer.ConstantInitializer;
import ai.djl.metric.Metrics;
import ai.djl.ndarray.types.Shape;
import ai.djl.translate.TranslateException;
import ai.djl.translate.Translator;
import ai.djl.translate.NoopTranslator;
import ai.djl.inference.Predictor;
import ai.djl.util.Pair;

public class LogisticRegression {

    public static void main(String[] args) throws IOException, TranslateException {
        Table spambase = Table.read().csv("spambase.csv");
        Table inputs = spambase.copy().removeColumns("is_spam");
        Table outputs = spambase.copy().retainColumns("is_spam");
        NDManager manager = NDManager.newBaseManager();
        NDArray x = manager.create(inputs.as().floatMatrix());
        NDArray scaled_x = Utils.normalize(x);

        NDArray y = manager.create(outputs.as().intMatrix());
        int batchSize = inputs.rowCount();
        ArrayDataset dataset = Utils.loadArray(scaled_x, y, batchSize, true);
        RandomAccessDataset[] datasets_split = dataset.randomSplit(80, 20);
        ArrayDataset trainingSet = (ArrayDataset) datasets_split[0];
        ArrayDataset testingSet = (ArrayDataset) datasets_split[1];

        Model model = Model.newInstance("logistic");
        SequentialBlock net = new SequentialBlock();
        Linear linearBlock = Linear.builder().optBias(true).setUnits(1).build();
        net.add(linearBlock);
        //net.setInitializer(new ConstantInitializer(0), Parameter.Type.WEIGHT);
        //net.initialize(manager, DataType.FLOAT32, x.getShape());

        model.setBlock(net);
        Loss loss = new SigmoidBinaryCrossEntropyLoss();

        float lr = 0.01f;

        Tracker lrt = Tracker.fixed(lr);
        Optimizer sgd = Optimizer.sgd().setLearningRateTracker(lrt).build();

        TrainingConfig config = new DefaultTrainingConfig(loss)
            .optOptimizer(sgd) // Optimizer
            .optDevices(manager.getEngine().getDevices(0)) // CPU
            .addEvaluator(new BinaryAccuracy()) // Model Accuracy
            .addTrainingListeners(TrainingListener.Defaults.logging()); // Logging

        Trainer trainer = model.newTrainer(config);

        trainer.initialize(new Shape(1, inputs.columnCount())); 
        Metrics metrics = new Metrics();
        trainer.setMetrics(metrics);

        int numEpochs = 1000; // only 10 with initialization of weights to 0
        EasyTrain.fit(trainer, numEpochs, trainingSet, testingSet);
    }
}

And the results are now close to 90% accuracy.

mai 14, 2024 5:17:01 PM ai.djl.training.listener.LoggingTrainingListener onEpoch
INFOS: Train: BinaryAccuracy: 0,91, SigmoidBinaryCrossEntropyLoss: 0,29
mai 14, 2024 5:17:01 PM ai.djl.training.listener.LoggingTrainingListener onEpoch
INFOS: Validate: BinaryAccuracy: 0,89, SigmoidBinaryCrossEntropyLoss: 0,31

If the weights are initialized with 0, the convergence is way much faster.I have solve my issue but I think the Accuracy() metric should take into account the binary case instead of having a BinaryAccuracy() metric.

[MohamedLEGH]

Hey @MohamedLEGH, I have question is NoopTranslator() is used for only for classification or can used for the Regression?

as well this is because weights I guess try to StandardScale or min max scaling on the input data.

I suppose you can also use it for Regression. I don't know as I'm only doing classification right now but I suppose it should work.

[adinath1233]

Hey @MohamedLEGH, I have question is NoopTranslator() is used for only for classification or can used for the Regression?

as well this is because weights I guess try to StandardScale or min max scaling on the input data.

I suppose you can also use it for Regression. I don't know as I'm only doing classification right now but I suppose it should work.

Yeah it worked.

[adinath1233]

the code updated :

Hey @MohamedLEGH , So you are using utils.normalise is utils your own custom class? and how you are performing the normalisation can you share the code snippet. Cause I am trying classification to normalise it but I am not getting proper output so.

[MohamedLEGH]

the code updated :

Hey @MohamedLEGH , So you are using utils.normalise is utils your own custom class? and how you are performing the normalisation can you share the code snippet. Cause I am trying classification to normalise it but I am not getting proper output so.

My Utils.java code below, hope it helps:

package machine_learning;

import ai.djl.training.dataset.Record;
import ai.djl.ndarray.NDArray;
import ai.djl.ndarray.NDManager;
import ai.djl.training.dataset.ArrayDataset;

public class Utils {
    public static ArrayDataset loadArray(NDArray features, NDArray labels, int batchSize, boolean shuffle) {
        return new ArrayDataset.Builder()
                    .setData(features) // set the features
                    .optLabels(labels) // set the labels
                    .setSampling(batchSize, shuffle) // set the batch size and random sampling
                    .build();
    }

    public static NDArray mean(NDArray X) {
        return X.mean(new int[]{0});
    }

    public static NDArray std(NDArray X) {
        NDArray mean = mean(X);
        NDArray squaredDiff = X.sub(mean).pow(2);
        NDArray variance = squaredDiff.mean(new int[]{0}); 
        return variance.sqrt(); 
    }

    public static NDArray normalize(NDArray X) {
        return X.sub(mean(X)).div(std(X));
    }
}