layer_independent_normal and Keras functional API

[sgvignali]

I get the following error when I use the layer_independent_normal() with the keras functional API:

Error in py_call_impl(callable, dots$args, dots$keywords) : 
  NotImplementedError: 

Detailed traceback: 
  File "/usr/lib/python3.6/pprint.py", line 144, in pformat
    self._format(object, sio, 0, 0, {}, 0)
  File "/usr/lib/python3.6/pprint.py", line 161, in _format
    rep = self._repr(object, context, level)
  File "/usr/lib/python3.6/pprint.py", line 393, in _repr
    self._depth, level)
  File "/usr/lib/python3.6/pprint.py", line 405, in format
    return _safe_repr(object, context, maxlevels, level)
  File "/usr/lib/python3.6/pprint.py", line 555, in _safe_repr
    rep = repr(object)
  File "/home/vignali/Desktop/flight-heights-analysis/renv/python/virtualenvs/renv-python-3.6.9/lib/python3.6/site-packages/tensorflow/python/keras/engine/keras_tensor.py", line 329, in __repr__
    type_spec_string = 'type_spec=%s' % self.type_spec
  File "/home/vignali/Desktop/flight-heights-analysis/renv/python/virtualenvs/renv-python-3.6.9/lib/python3.6/site-packages/tensorflow/python/framework/type_spec.py", line 308, in __repr__
    return "%s%

Here is the working example taken from the documentation:

library(keras)
library(tfprobability)
input_shape <- c(28, 28, 1)
encoded_shape <- 2
n <- 2

model <- keras_model_sequential() %>% 
  layer_flatten(input_shape = input_shape) %>% 
  layer_dense(units = n) %>% 
  layer_dense(units = params_size_independent_normal(encoded_shape)) %>% 
  layer_independent_normal(event_shape = encoded_shape)

And this is using the functional API:

input <- layer_input(shape = input_shape)
output <- input %>% 
  layer_flatten(input_shape = input_shape) %>% 
  layer_dense(units = n) %>% 
  layer_dense(units = params_size_independent_normal(encoded_shape)) %>% 
  layer_independent_normal(event_shape = encoded_shape)

With the second example I get the error, am I doing something wrong?

This is my sessionInfo()

R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.5 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=de_CH.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=de_CH.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=de_CH.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=de_CH.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices datasets  utils     methods   base     

other attached packages:
[1] tfprobability_0.11.0.0 keras_2.3.0.0.9000    

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.6         rstudioapi_0.13    whisker_0.4       
 [4] magrittr_2.0.1     tidyselect_1.1.0   lattice_0.20-41   
 [7] R6_2.5.0           rlang_0.4.10       tools_4.0.2       
[10] grid_4.0.2         ellipsis_0.3.1     tfruns_1.4        
[13] tibble_3.0.5       lifecycle_0.2.0    forge_0.2.0       
[16] crayon_1.3.4       tensorflow_2.2.0   Matrix_1.2-18     
[19] purrr_0.3.4        vctrs_0.3.6        base64enc_0.1-3   
[22] tfestimators_1.9.1 zeallot_0.1.0      glue_1.4.2        
[25] compiler_4.0.2     pillar_1.4.7       tfdatasets_2.2.0  
[28] generics_0.1.0     reticulate_1.18    jsonlite_1.7.2    
[31] renv_0.12.0        pkgconfig_2.0.3

The same happens with layer_independent_bernoulli() or using layer_distribution_lambda(). Any help would be very much apreciated!

[sgvignali]

I had a look at the error again. It seems that the layer is created correctly even if the error is raised, but I didn't check yet if is possible to train the model. The summary includes the independent_normal layer though:

library(keras)
library(tfprobability)
input_shape <- c(28, 28, 1)
encoded_shape <- 2
n <- 2

input <- layer_input(shape = input_shape)

# The following generates the error
output <- input %>% 
  layer_flatten(input_shape = input_shape) %>% 
  layer_dense(units = n) %>% 
  layer_dense(units = params_size_independent_normal(encoded_shape)) %>% 
  layer_independent_normal(event_shape = encoded_shape)

# But the model can be created
model <- keras_model(input, output)
summary(model)

Interestingly, the error is printed only when the layer is assigned to a variable. For example the following doesn't generate the error:

input %>% 
  layer_flatten(input_shape = input_shape) %>% 
  layer_dense(units = n) %>% 
  layer_dense(units = params_size_independent_normal(encoded_shape)) %>% 
  layer_independent_normal(event_shape = encoded_shape)

For what I understood the error is generated by the python __str__ method, maybe because some information of the layer is missing, for example output$type_spec generates a similar error. I hope this helps to solve the problem.

[anirban-mukherjee]

I have tested this and I think this is an upstream bug. The following Python code receives a not implemented error but the model is formed correctly. I suggest we file a bug report with TensorFlow Probability.

import tensorflow as tf
import tensorflow_probability as tfp
tfkl = tf.keras.layers
tfpl = tfp.layers

input_shape = [28, 28, 1]
encoded_shape = 2

inputs = tf.keras.Input(shape=input_shape)
layer1 = tfkl.Flatten()(inputs)
layer2 = tfkl.Dense(units=10)(layer1)
layer3 = tfkl.Dense(units=tfpl.IndependentNormal.params_size(encoded_shape)) (layer2)
outputs = tfpl.IndependentNormal(event_shape = encoded_shape) (layer3)
model=tf.keras.Model(inputs, outputs)
model.summary()

[anirban-mukherjee]

To follow up, str and repr cannot be called on the object because it cannot be serialised. I suggest we close this issue as the limitation is upstream, is known (though poorly documented IMO and the error message is uninformative), and is unlikely to be fixed anytime soon. See: https://github.com/tensorflow/probability/issues/1500.