google / yggdrasil-decision-forests

A library to train, evaluate, interpret, and productionize decision forest models such as Random Forest and Gradient Boosted Decision Trees.
https://ydf.readthedocs.io/
Apache License 2.0
473 stars 49 forks source link

Training is not reproducible between Intel and ARM (M2) #92

Open lusis-ai opened 5 months ago

lusis-ai commented 5 months ago

When doing the same simple RF classifier training on an Intel Xeon and on Mac M2, metrics are different (and of course trees are also different).

import ydf  # Yggdrasil Decision Forests
import pandas as pd  # We use Pandas to load small datasets

# Download a classification dataset and load it as a Pandas DataFrame.
ds_path = "https://raw.githubusercontent.com/google/yggdrasil-decision-forests/main/yggdrasil_decision_forests/test_data/dataset"
train_ds = pd.read_csv(f"{ds_path}/adult_train.csv")
test_ds = pd.read_csv(f"{ds_path}/adult_test.csv")

model = ydf.RandomForestLearner(label="income",
                                task=ydf.Task.CLASSIFICATION).train(train_ds)

evaluation = model.evaluate(test_ds)

print(evaluation)

Accuracy is :

The other metrics and the threes are also different.

The same problem happened on Scikit-Learn and XGBoost when setting some hyper parameters using random number generation. It acts like if the random sequence is not the same between Intel and ARM.

But it does not happened with LightGBM that is the only one enabling perfect reproducibility no matter the CPU.