Open hahampis opened 3 years ago
mnist = fetch_openml('mnist_784', version=1, cache=True, as_frame=True)
X = mnist.data
y = mnist.target.astype(np.uint8)
X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=10000)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=10000)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
X_val = scaler.transform(X_val)
rnd_clf = RandomForestClassifier(n_estimators=100, n_jobs=-1)
ext_clf = ExtraTreesClassifier(n_jobs=-1, n_estimators=100)
linsvc_clf = LinearSVC(max_iter=100, tol=20)
for clf in [rnd_clf, ext_clf, linsvc_clf]:
clf.fit(X_train, y_train)
y_val_pred = clf.predict(X_val)
print(clf.__class__.__name__, accuracy_score(y_val, y_val_pred), f1_score(y_val, y_val_pred, average=None))
named_estimators = [
("Random_Forest", rnd_clf),
("Extra_Trees", ext_clf),
("Linear_SVC", linsvc_clf),
]
voting_clf = VotingClassifier(named_estimators, n_jobs=-1)
voting_clf.fit(X_train, y_train)
y_val_pred = voting_clf.predict(X_val)
print("Voting Classifier", accuracy_score(y_val, y_val_pred), f1_score(y_val, y_val_pred, average=None))
voting_clf.estimators
voting_clf.set_params(Linear_SVC=None)
voting_clf.estimators_
del voting_clf.estimators_[2]
y_val_pred = voting_clf.predict(X_val)
print("Voting Classifier", accuracy_score(y_val, y_val_pred), f1_score(y_val, y_val_pred, average=None))
voting_clf.voting='soft'
y_val_pred = voting_clf.predict(X_val)
print("Voting Classifier", accuracy_score(y_val, y_val_pred), f1_score(y_val, y_val_pred, average=None))
[estimator.score(X_test, y_test) for estimator in voting_clf.estimators_], voting_clf.score(X_test, y_test)
fig, (ax1, ax2) = plt.subplots(1,2, figsize=(20,10))
ax1.imshow(rnd_clf.feature_importances_.reshape(28, 28), cmap='binary')
ax2.imshow(ext_clf.feature_importances_.reshape(28, 28), cmap='binary')
Try running this on your jupyter and comment back if you still run into any problems. Run the code in different cells accordingly.
Hi @hahampis ,
Thanks for your feedback. I just ran the Colab notebook, and I got these results:
It's different from the results in the book:
LogisticRegression 0.864
RandomForestClassifier 0.872
SVC 0.888
VotingClassifier 0.896
Indeed, it's proven extremely difficult (well, impossible really) to ensure that the notebooks keep producing the exact same output over time:
fetch_mldata()
to download the MNIST dataset, but this relied on mldata.org, which was closed, so I had to use fetch_openml()
instead: it's the same dataset, but in a different order, so it changes the results.In short, unless you're using the exact same environment as I did when writing the book, you will probably get a slightly different result. In general, it's not a big deal if you get (say) 89.6% accuracy instead of 88.8%, but in this case it does show that the voting classifier will not always be better than the individual predictors.
Instead of trying to have the exact same environment as I used, if you really want to see an example where the voting classifier wins, you can try tweaking the hyperparameters, or simply changing the random_state
arguments. For example, when I set random_state=43
in the Colab notebook, I get this:
LogisticRegression 0.864
RandomForestClassifier 0.896
SVC 0.896
VotingClassifier 0.896
The voting classifier is not better than the SVC or the RandomForestClassifier.
But when I use random_state=44
, I get this:
LogisticRegression 0.864
RandomForestClassifier 0.888
SVC 0.896
VotingClassifier 0.912
Of course, many of these algorithms are stochastic, so getting slightly different results at each execution is to be expected when changing the random seeds.
I hope this helps!
@ageron Your answer is much appreciated, thank you! I obviously didn't expect to get the exact same output due to all the reasons you mentioned. My concern was that I couldn't get the Voting Classifier to win, which is the main idea. I guess it can happen... I'll try to change some more things around (like the seed) to see if I can get it to outperform the rest of the classifiers!
Hello,
I am trying the first code sample of Chapter 7 which is supposed to demonstrate an accuracy improvement of a Voting Classifier compared to the individual estimators. However, I am not able to achieve the scores shown in the book. In fact, the standalone SVC outperforms the Voting classifier:
It's not better with soft voting either:
Is this to be expected? I played around a bit when creating the moons dataset but couldn't make any difference. What are the exact parameters of the dataset used in the book? (n_samples, noise)
Thank you in advance!