dfalbel
commented
5 years ago Woul;d you mind sharing the code for CrfRnnLayer? I have fixed something related to this recently, can you try reinstalling Keras from master?
Closed matthiasmace closed 4 years ago
dfalbel
commented
5 years ago Woul;d you mind sharing the code for CrfRnnLayer? I have fixed something related to this recently, can you try reinstalling Keras from master?
matthiasmace
commented
5 years ago Woul;d you mind sharing the code for
CrfRnnLayer? I have fixed something related to this recently, can you try reinstalling Keras from master?
Hi, and thanks for the interest. You probably want the code for the "CustomLayer" object ? Here it is :
CustomLayer <- R6::R6Class("Layer_CrfRnn",
inherit = KerasLayer,
public = list(
#output_dim = NULL,
kernel = NULL,
image_dims = NULL,
num_classes = NULL,
theta_alpha = NULL,
theta_beta = NULL,
theta_gamma = NULL,
num_iterations = NULL,
spatial_ker_weights = NULL,
bilateral_ker_weights = NULL,
compatibility_matrix = NULL,
initialize = function(image_dims,
num_classes,
theta_alpha,
theta_beta,
theta_gamma,
num_iterations,
...) {
self$image_dims = image_dims
self$num_classes = as.integer(num_classes)
self$theta_alpha = theta_alpha
self$theta_beta = theta_beta
self$theta_gamma = theta_gamma
self$num_iterations = num_iterations
self$spatial_ker_weights = NULL
self$bilateral_ker_weights = NULL
self$compatibility_matrix = NULL
},
build = function(input_shape, ...) {
# Weights of the spatial kernel
self$spatial_ker_weights <- self$add_weight(
name = 'spatial_ker_weights',
shape = c(self$num_classes, self$num_classes),
initializer = diagonal_initializer,
trainable = TRUE
)
# Weights of the bilateral kernel
self$bilateral_ker_weights <- self$add_weight(
name = 'bilateral_ker_weights',
shape = list(self$num_classes, self$num_classes),
initializer = diagonal_initializer,
trainable = TRUE
)
# Compatibility matrix
self$compatibility_matrix <- self$add_weight(
name = 'compatibility_matrix',
shape = list(self$num_classes, self$num_classes),
initializer = potts_model_initializer,
trainable = TRUE
)
},
call = function(inputs, ...) {
unaries = tf$transpose(inputs[[1]][1, , , ], perm = as.integer(c(2, 0, 1)))
rgb = tf$transpose(inputs[[2]][1, , , ], perm = as.integer(c(2, 0, 1)))
num_classes <- self$num_classes
height <- self$image_dims[1]
width <- self$image_dims[2]
all_ones <- tf$cast(
array(data = 1, dim = c(num_classes, height, width), )
, tf$float32)
# Prepare filter normalization coefficients
spatial_norm_vals = custom_module$high_dim_filter(all_ones, rgb, bilateral = FALSE,
theta_gamma = self$theta_gamma)
bilateral_norm_vals = custom_module$high_dim_filter(all_ones, rgb, bilateral = TRUE,
theta_alpha = self$theta_alpha,
theta_beta = self$theta_beta)
q_values <- unaries
for (i in 1:self$num_iterations) {
softmax_out <- tf$nn$softmax(q_values, as.integer(0))
# Spatial filtering
spatial_out <- custom_module$high_dim_filter(softmax_out, rgb, bilateral = FALSE,
theta_gamma = self$theta_gamma)
spatial_out <- spatial_out/spatial_norm_vals
# Bilateral filtering
bilateral_out <- custom_module$high_dim_filter(softmax_out, rgb, bilateral = TRUE,
theta_alpha=self$theta_alpha,
theta_beta=self$theta_beta)
bilateral_out <- bilateral_out/bilateral_norm_vals
# Weighting filter outputs
message_passing <- (tf$matmul(self$spatial_ker_weights,
tf$reshape(spatial_out, as.integer(c(num_classes, -1))))
+ tf$matmul(self$bilateral_ker_weights,
tf$reshape(bilateral_out, as.integer(c(num_classes, -1)))))
# Compatibility transform
pairwise <- tf$matmul(self$compatibility_matrix, message_passing)
# Adding unary potentials
pairwise <- tf$reshape(pairwise, as.integer(c(num_classes, height, width)))
q_values <- unaries - pairwise
}
return(tf$transpose(tf$reshape(q_values, as.integer(c(1, num_classes, height, width))), perm = as.integer(c(0, 2, 3, 1))))
#probs <- (tf$transpose(tf$reshape(q_values, as.integer(c(1, num_classes, height, width))), perm = as.integer(c(0, 2, 3, 1))))
#return(get_label_image(probs, 500, 500))
},
compute_output_shape = function(input_shape) {
return(input_shape)
}
)
)Woul;d you mind sharing the code for
CrfRnnLayer? I have fixed something related to this recently, can you try reinstalling Keras from master?
Hi @dfalbel Do you have an idea please ?
matthiasmace
commented
5 years ago Hi @dfalbel No it did not work...
dfalbel
commented
5 years ago @matthiasmace you can fix it by removing the ellipsis from the methods signature.
This is really a bug and I'll push a fix.
matthiasmace
commented
5 years ago @dfalbel
I removed all ... in the CustomLayer but now I get the following :
RuntimeError: Evaluation error: RuntimeError: Evaluation error: argument inutilis'e (dtype = <environment>).
Detailed traceback:
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 522, in add_weight
aggregation=aggregation)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 139, in make_variable
shape=variable_shape if variable_shape else None)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 258, in __call__
return cls._variable_v1_call(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/usr/local/lib/python3.7/site-packages/tensThis is most probably an error in you CustomLayer code. For example I don't have this diagonal_initializer. Please provide a minimal reproducible example so I can try to find what's wrong.
matthiasmace
commented
5 years ago @dfalbel
This is most probably an error in you CustomLayer code. For example I don't have this
diagonal_initializer. Please provide a minimal reproducible example so I can try to find what's wrong.
Here is the complete code :
################ from high_dim_filter_loader.py ##############
### présent dans la version python compilée pour le virtuel env r-tensorflow, présent sur le drive
custom_module <- tf$load_op_library("/Users/matthias/Documents_residents/2018_cavalls/Projet_2/2019_KERAS_CFRasRNN/crfasrnn-keras-R/crfasrnn_keras/src/cpp/high_dim_filter.so")
#custom_module <- tf$load_op_library("/users/p19039/orlando/matthias/CRFasRNN/crfasrnn_keras/src/cpp/high_dim_filter.so")
# @ops.RegisterGradient('HighDimFilter')
# tf$python$framework$ops$RegisterGradient('HighDimFilter')
high_dim_filter_grad <- function(op, grad){
rgb <- op$inputs[2]
grad_vals <- custom_module$high_dim_filter(grad, rgb,
bilateral=op$get_attr('bilateral'),
theta_alpha=op$get_attr('theta_alpha'),
theta_beta=op$get_attr('theta_beta'),
theta_gamma=op$get_attr('theta_gamma'),
backwards=True)
return(c(grad_vals, tf$zeros_like(rgb)))
}
g <- tf$RegisterGradient("HighDimFilter")
g(high_dim_filter_grad)
#############################################################
diagonal_initializer <- function(x){return(diag(1, x[[1]], x[[2]]))}
potts_model_initializer <- function(x){return((-1)*diagonal_initializer(x))}
#################################################################################################################
############################################# CRFasRNN Layer #############################################
CustomLayer <- R6::R6Class("Layer_CrfRnn",
inherit = KerasLayer,
public = list(
#output_dim = NULL,
kernel = NULL,
image_dims = NULL,
num_classes = NULL,
theta_alpha = NULL,
theta_beta = NULL,
theta_gamma = NULL,
num_iterations = NULL,
spatial_ker_weights = NULL,
bilateral_ker_weights = NULL,
compatibility_matrix = NULL,
initialize = function(image_dims,
num_classes,
theta_alpha,
theta_beta,
theta_gamma,
num_iterations
) {
# , ...) {
self$image_dims = image_dims
self$num_classes = as.integer(num_classes)
self$theta_alpha = theta_alpha
self$theta_beta = theta_beta
self$theta_gamma = theta_gamma
self$num_iterations = num_iterations
self$spatial_ker_weights = NULL
self$bilateral_ker_weights = NULL
self$compatibility_matrix = NULL
},
# build = function(input_shape, ...) {
build = function(input_shape) {
# Weights of the spatial kernel
self$spatial_ker_weights <- self$add_weight(
name = 'spatial_ker_weights',
shape = c(self$num_classes, self$num_classes),
initializer = diagonal_initializer,
trainable = TRUE
)
# Weights of the bilateral kernel
self$bilateral_ker_weights <- self$add_weight(
name = 'bilateral_ker_weights',
shape = list(self$num_classes, self$num_classes),
initializer = diagonal_initializer,
trainable = TRUE
)
# Compatibility matrix
self$compatibility_matrix <- self$add_weight(
name = 'compatibility_matrix',
shape = list(self$num_classes, self$num_classes),
initializer = potts_model_initializer,
trainable = TRUE
)
},
# call = function(inputs, ...) {
call = function(inputs) {
unaries = tf$transpose(inputs[[1]][1, , , ], perm = as.integer(c(2, 0, 1)))
rgb = tf$transpose(inputs[[2]][1, , , ], perm = as.integer(c(2, 0, 1)))
num_classes <- self$num_classes
height <- self$image_dims[1]
width <- self$image_dims[2]
all_ones <- tf$cast(
array(data = 1, dim = c(num_classes, height, width), )
, tf$float32)
# Prepare filter normalization coefficients
spatial_norm_vals = custom_module$high_dim_filter(all_ones, rgb, bilateral = FALSE,
theta_gamma = self$theta_gamma)
bilateral_norm_vals = custom_module$high_dim_filter(all_ones, rgb, bilateral = TRUE,
theta_alpha = self$theta_alpha,
theta_beta = self$theta_beta)
q_values <- unaries
for (i in 1:self$num_iterations) {
softmax_out <- tf$nn$softmax(q_values, as.integer(0))
# Spatial filtering
spatial_out <- custom_module$high_dim_filter(softmax_out, rgb, bilateral = FALSE,
theta_gamma = self$theta_gamma)
spatial_out <- spatial_out/spatial_norm_vals
# Bilateral filtering
bilateral_out <- custom_module$high_dim_filter(softmax_out, rgb, bilateral = TRUE,
theta_alpha=self$theta_alpha,
theta_beta=self$theta_beta)
bilateral_out <- bilateral_out/bilateral_norm_vals
# Weighting filter outputs
message_passing <- (tf$matmul(self$spatial_ker_weights,
tf$reshape(spatial_out, as.integer(c(num_classes, -1))))
+ tf$matmul(self$bilateral_ker_weights,
tf$reshape(bilateral_out, as.integer(c(num_classes, -1)))))
# Compatibility transform
pairwise <- tf$matmul(self$compatibility_matrix, message_passing)
# Adding unary potentials
pairwise <- tf$reshape(pairwise, as.integer(c(num_classes, height, width)))
q_values <- unaries - pairwise
}
return(tf$transpose(tf$reshape(q_values, as.integer(c(1, num_classes, height, width))), perm = as.integer(c(0, 2, 3, 1))))
#probs <- (tf$transpose(tf$reshape(q_values, as.integer(c(1, num_classes, height, width))), perm = as.integer(c(0, 2, 3, 1))))
#return(get_label_image(probs, 500, 500))
},
compute_output_shape = function(input_shape) {
return(input_shape)
}
)
)
#################################################################################################################
############################################# RNN #############################################
height = 500
width = 500
channels = 3
input_shape = c(height, width, 3)
#rescale=1. / 255,
img_input <- layer_input(shape = input_shape
#, batch_shape = c(10, input_shape)
, name = "img_input")
x <- img_input %>%
layer_zero_padding_2d(padding = c(100, 100), name = "padding")
# VGG-16 convolution block 1
x <- x %>%
layer_conv_2d(filters = 64, kernel_size = c(3, 3), activation='relu', padding='valid', name='conv1_1') %>%
layer_conv_2d(filters = 64, kernel_size = c(3, 3), activation='relu', padding='same', name='conv1_2') %>%
layer_max_pooling_2d(pool_size = c(2, 2), strides = c(2, 2), name='pool1')
# VGG-16 convolution block 2
x <- x %>%
layer_conv_2d(filters = 128, kernel_size = c(3, 3), activation='relu', padding='same', name='conv2_1') %>%
layer_conv_2d(filters = 128, kernel_size = c(3, 3), activation='relu', padding='same', name='conv2_2') %>%
layer_max_pooling_2d(pool_size = c(2, 2), strides = c(2, 2), name='pool2', padding='same')
# VGG-16 convolution block 3
x <- x %>%
layer_conv_2d(filters = 256, kernel_size = c(3, 3), activation='relu', padding='same', name='conv3_1') %>%
layer_conv_2d(filters = 256, kernel_size = c(3, 3), activation='relu', padding='same', name='conv3_2') %>%
layer_conv_2d(filters = 256, kernel_size = c(3, 3), activation='relu', padding='same', name='conv3_3') %>%
layer_max_pooling_2d(pool_size = c(2, 2), strides = c(2, 2), name='pool3', padding='same')
pool3 = x
# VGG-16 convolution block 4
x <- x %>%
layer_conv_2d(filters = 512, kernel_size = c(3, 3), activation='relu', padding='same', name='conv4_1') %>%
layer_conv_2d(filters = 512, kernel_size = c(3, 3), activation='relu', padding='same', name='conv4_2') %>%
layer_conv_2d(filters = 512, kernel_size = c(3, 3), activation='relu', padding='same', name='conv4_3') %>%
layer_max_pooling_2d(pool_size = c(2, 2), strides = c(2, 2), name='pool4', padding='same')
pool4 = x
# VGG-16 convolution block 5
x <- x %>%
layer_conv_2d(filters = 512, kernel_size = c(3, 3), activation='relu', padding='same', name='conv5_1') %>%
layer_conv_2d(filters = 512, kernel_size = c(3, 3), activation='relu', padding='same', name='conv5_2') %>%
layer_conv_2d(filters = 512, kernel_size = c(3, 3), activation='relu', padding='same', name='conv5_3') %>%
layer_max_pooling_2d(pool_size = c(2, 2), strides = c(2, 2), name='pool5', padding='same')
# Fully-connected layers converted to convolution layers
x <- x %>%
layer_conv_2d(filters = 4096, kernel_size = c(7, 7), activation='relu', padding='valid', name='fc6') %>%
layer_dropout(0.5) %>%
layer_conv_2d(filters = 4096, kernel_size = c(1, 1), activation='relu', padding='valid', name='fc7') %>% ##
layer_dropout(0.5) %>% #print.tf() %>%##
layer_conv_2d(filters = 21, kernel_size = c(1, 1), padding='valid', name='score-fr') ##
# Deconvolution
score2 <- x %>%
layer_conv_2d_transpose(filters = 21, kernel_size = c(4, 4), strides = 2, name='score2')
# Skip connections from pool4
score_pool4 <- pool4 %>%
layer_conv_2d(filters = 21, kernel_size = c(1, 1), name='score-pool4')
score_pool4c <- score_pool4 %>%
layer_cropping_2d(c(5, 5))
score_fused <- layer_add(c(score2, score_pool4c))
score4 <- score_fused %>%
layer_conv_2d_transpose(filters = 21, kernel_size = c(4, 4), strides=2, name='score4', use_bias = FALSE)
# Skip connections from pool3
score_pool3 <- pool3 %>%
layer_conv_2d(filters = 21, kernel_size = c(1, 1), name='score-pool3')
score_pool3c <- score_pool3 %>%
layer_cropping_2d(c(9, 9))
# Fuse things together
score_final <- layer_add(c(score4, score_pool3c))
# Final up-sampling and cropping
upsample <- score_final %>%
layer_conv_2d_transpose(filters = 21, kernel_size = c(16, 16), strides = 8, name='upsample', use_bias = FALSE)
upscore <- upsample %>%
layer_cropping_2d(cropping = list(c(31, 37), c(31, 37)))
# final_inputs <- layer_concatenate(inputs = list(img_input, upscore), name = 'final_inputs')
CrfRnnLayer <- function(object, image_dims, name = NULL,#input_shape,
num_classes, theta_alpha, theta_beta, theta_gamma, num_iterations) {
create_layer(CustomLayer, object, list(
# name,
image_dims,
num_classes = as.integer(num_classes),
theta_alpha,
theta_beta,
theta_gamma ,
num_iterations
# batch_shape = 10
)
)
}
output <- CrfRnnLayer(list(upscore, img_input),
image_dims = c(500, 500),
num_classes = 21,
theta_alpha = 160,
theta_beta = 3,
theta_gamma = 3,
num_iterations = 10
# name = "crfrnn"
)@skeydan : I really do not understand, it worked with your previous help
And more info :
TensorFlow v2.0.0 (/usr/local/lib/python3.7/site-packages/tensorflow)
Python v3.7 (~/.virtualenvs/r-reticulate/bin/python)This is most probably an error in you CustomLayer code. For example I don't have this
diagonal_initializer. Please provide a minimal reproducible example so I can try to find what's wrong.
I di d try many things but I failed to get rid of the error...
matthiasmace
commented
5 years ago @skeydan ? @dfalbel ?
matthiasmace
commented
5 years ago I tried also using a conda installation. Same output...
Hi everyone, I am using R (not Rstudio) on MacosX. I am trying to create a custom layer (translating the Python code from an existing layer in fact) using the following code :
And it returns the following error :
What am I doing wrong please ? Need to know that it worked fine last year but I updated Keras and Tensorflow meanwhile. Thanks a lot for the help (@skeydan )!
My python config : py_config()