Implement progress bar in ART

[ClonedOne]

Describe the bug The fit() method of AdversarialTrainer using BasicIterativeMethod attack produces an absurd amount of progress bars on the output, both in terminal and in Jupyter notebooks.

Having so much stuff on output will probably reduce performance.

ps: passing **{'verbosity':0} to fit will remove all output, which is very impractical.

To Reproduce

from art.utils import load_dataset
from art.classifiers import KerasClassifier
from art.attacks.fast_gradient import FastGradientMethod
from art.attacks.iterative_method import BasicIterativeMethod
from art.defences.adversarial_trainer import AdversarialTrainer

from keras.models import Sequential
from keras.layers import Dense, Flatten, Conv2D, MaxPooling2D, Dropout

import keras.backend as ke
import tensorflow as tf
import numpy as np

def cnn_model(x_train_shape):
    """
    Simple CNN model for MNIST
    """

    ke.set_learning_phase(1)

    model = Sequential()
    model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=x_train.shape[1:]))
    model.add(Conv2D(64, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(10, activation='softmax'))

    model.compile(
        loss='categorical_crossentropy',
        optimizer='adam', 
        metrics=['accuracy']
    )

    return model

(x_train, y_train), (x_test, y_test), min_, max_ = load_dataset('mnist')
cnn = cnn_model(x_train.shape)

robust_classifier = KerasClassifier(
    clip_values=(min_, max_), 
    model=cnn,
    use_logits=False
)

attacks = BasicIterativeMethod(robust_classifier, eps=0.3, eps_step=0.01, max_iter=5)

trainer = AdversarialTrainer(robust_classifier, attacks, ratio=1.0)
trainer.fit(x_train, y_train, nb_epochs=5, batch_size=128)

Expected behavior Possibly a single progress bar.

Screenshots

System information (please complete the following information):

• OS: MacOs Mojave 10.14.4
• Python version: 3.7.3
• ART version or commit number: Adversarial-Robustness-Toolbox==0.9.0
• TensorFlow / Keras / PyTorch / MXNet version: Keras==2.2.4, tensorflow==1.13.1, jupyter==1.0.0

[beat-buesser]

Hi @ClonedOne, thank you for your useful suggestions, we'll try to consider them as soon as possible. Would you have a commit or code ready to resolve this issue?

[ClonedOne]

Hi @beat-buesser, thanks for answering, I couldn't identify where the verbose logging originated, but I'll try to take a deeper look in the weekend

[ririnicolae]

@ClonedOne ART does not have any progress bar of its own, so the ones you are seeing are from Keras. For the case of adversarial training, it is normal to see so many of them, as each time the classifier is trained for one epoch before generating fresh adversarial samples for the next epoch. To deactivate them, it is enough to set verbose to 0 when the AdversarialTrainer calls fit.

@beat-buesser A few users have suggested that we add progress bars to ART. For that, we either need to implement our own, or use an existing solution (tqdm?). The bars should probably be printed with the logger, so they can be deactivated easily.

[beat-buesser]

Hi @ririnicolae Thank you for the insights and ideas. Are you aware if we have already a feature request open for the single progress bar idea that we could reference here?

I think with nb_epochs we have a good idea of the overall progress of fit.

I agree with @ClonedOne that if setting **{'verbosity':0} removes all output it would be nice to provide logging something else, e.g. the discussed single progress bar as an intermediate verbosity level.

[ririnicolae]

@beat-buesser There is no open issue for the progress bar, we could change the scope of the current one to cover progress bars for the entire library (attacks, defenses, training, etc.) if @ClonedOne is fine with that?

[ClonedOne]

@ririnicolae Yes a single library-wide progress bar sounds like a nice solution! Please let me know if I can help with testing

[ririnicolae]

@ClonedOne Perfect, your help will be very useful for testing.

Re-purposing this issue then. Options for implementing this:

• In-house solution
• External, adding dependency:
  • tqdm
  • progress
  • python-progressbar

The choice depends on which of these is able to serve the output in the ART logger. FYI, tqdm seems massively popular and well maintained.