Counterfactual "kills" stderr / consumes exceptions

[fhvilshoj]

Hi,

I had this weird issue that whenever I constructed an explainer, my scripts would stop printing errors. Turns out that sys.stderr disappears when calling cf = CounterFactual(cnn, shape=(1, 28, 28, 1)).

Here is a minimal example:

  0 import sys
  1 import traceback
  2 import tensorflow as tf
  3 tf.compat.v1.disable_v2_behavior() # disable TF2 behaviour as alibi code still relies on TF1 constructs
  4 
  5 from alibi.explainers import CounterFactual
  6 
  7 print('TF version: ', tf.__version__)
  8 print('Eager execution enabled: ', tf.executing_eagerly()) # Fals
  9 
 10 def f1(): return f2()    # couple of nested functions to get stack track
 11 def f2(): raise ValueError('[Custom Error]')
 12 
 13 model = tf.keras.Sequential()
 14 model.add(tf.keras.Input(shape=(16,)))
 15 model.add(tf.keras.layers.Dense(10))
 16 model.compile(loss='mse', optimizer='adam')
 17 
 18 stderr = sys.stderr
 19 try: f1()
 20 except ValueError as e:
 21     print("Before, got value error: ", e)
 22     traceback.print_exc()
 23 
 24 cf = CounterFactual(model, shape=(1, 16))
 25 
 26 sys.stderr = stderr
 27 try: f1()
 28 except ValueError as e:
 29     print("After, got value error: ", e)
 30     traceback.print_exc()

Output:

 TF version:  2.4.1 
 Eager execution enabled:  False 
 Before, got value error:  [Custom Error] 
 Traceback (most recent call last): 
   File "/path/to/example.py", line 19, in <module> 
     try: f1() 
   File "/path/to/example.py", line 10, in f1 
     def f1(): return f2()    # couple of nested functions to get stack track 
   File "/path/to/example.py", line 11, in f2 
     def f2(): raise ValueError('[Custom Error]') 
 ValueError: [Custom Error] 
 After, got value error:  [Custom Error] 

Expected stack trace both before and after.

Output after uncommenting line 18 and 26:

 TF version:  2.4.1 
 Eager execution enabled:  False 
 Before, got value error:  [Custom Error] 
 Traceback (most recent call last): 
   File "/path/to/example.py", line 19, in <module> 
     try: f1() 
   File "/path/to/example.py", line 10, in f1 
     def f1(): return f2()    # couple of nested functions to get stack track 
   File "/path/to/example.py", line 11, in f2 
     def f2(): raise ValueError('[Custom Error]') 
 ValueError: [Custom Error] 
 After, got value error:  [Custom Error] 
 Traceback (most recent call last): 
   File "/path/to/example.py", line 27, in <module> 
     try: f1() 
   File "/path/to/example.py", line 10, in f1 
     def f1(): return f2()    # couple of nested functions to get stack track 
   File "/path/to/example.py", line 11, in f2 
     def f2(): raise ValueError('[Custom Error]') 
 ValueError: [Custom Error] 

As expected, two stack traces.

[jklaise]

@fhvilshoj I'm getting both stack traces running the script on Tensorflow 2.4.1 and Alibi master. What version of Alibi are you using?

[fhvilshoj]

>>> import alibi
>>> alibi.__version__
'0.5.8dev'

and

$ pip list

yields

Package                Version
---------------------- -------------------
alibi                  0.5.8.dev0
...
numpy                  1.19.2
...
pandas                 1.2.4
Pillow                 8.2.0
...
scikit-image           0.18.1
scikit-learn           0.24.1
scipy                  1.6.2
...
tensorboard            2.4.0
tensorboard-plugin-wit 1.6.0
tensorflow             2.4.1
tensorflow-estimator   2.4.0
...
torch                  1.8.1
torchvision            0.9.1

It is not an issue in Google Colab with either version 0.5.9dev og version 0.5.8dev (You can see executions by following links).

I guess that it is some other dependency in my environment that does it them.

For completeness, I list the Google Colab dependencies below:

Package                       Version            
----------------------------- -------------------
alibi                         0.5.9.dev0         
...
Keras                         2.4.3              
keras-nightly                 2.5.0.dev2021032900
...
numpy                         1.19.5             
...
pandas                        1.1.5              
...
Pillow                        7.1.2              
...
scikit-image                  0.16.2             
scikit-learn                  0.22.2.post1       
scipy                         1.4.1              
...
sklearn                       0.0                
sklearn-pandas                1.8.0              
...
tensorboard                   2.5.0                 
tensorflow                    2.5.0              
tensorflow-datasets           4.0.1              
tensorflow-estimator          2.5.0              
tensorflow-gcs-config         2.5.0              
tensorflow-hub                0.12.0             
tensorflow-metadata           1.0.0              
tensorflow-probability        0.12.1          
torch                         1.8.1+cu101        
torchvision                   0.9.1+cu101     

[jklaise]

Need reproducing before proceeding.