Passing states to a bidirectional LSTM decoder

Hello,

I am doing some tests with your example:

https://keras.rstudio.com/articles/examples/lstm_seq2seq.html

Instead of LSTM layers in the enocder/decoder I would like to have bidirectional LSTM. The following code seems to fail in the bidirectional LSTM decoder:

## Create the model
## Define an input sequence and process it.
encoder_inputs  <- layer_input(shape=list(NULL,num_encoder_tokens))
encoder         <- layer_lstm(units=latent_dim, return_state=TRUE)
encoder_results <- encoder_inputs %>% bidirectional(encoder)
## We discard `encoder_outputs` and only keep the states.
encoder_states  <- encoder_results[2:3]

## Set up the decoder, using `encoder_states` as initial state.
decoder_inputs  <- layer_input(shape=list(NULL, num_decoder_tokens))
## We set up our decoder to return full output sequences,
## and to return internal states as well. We don't use the
## return states in the training model, but we will use them in inference.
decoder_lstm    <- layer_lstm(units=latent_dim, return_sequences=TRUE,
                              return_state=TRUE, stateful=FALSE)
decoder_results <- bidirectional(decoder_lstm(decoder_inputs, initial_state=encoder_states))
decoder_dense   <- layer_dense(units=num_decoder_tokens, activation='softmax')
decoder_outputs <- decoder_dense(decoder_results[[1]])

## Define the model that will turn
## `encoder_input_data` & `decoder_input_data` into `decoder_target_data`
model <- keras_model( inputs = list(encoder_inputs, decoder_inputs),
                      outputs = decoder_outputs )

What should be the correct statement? The encoder seems to be ok. I am not sure if encoder_states <- encoder_results[2:3] is still correct. However, I don't know how to implement the decoder. Could you please help me? I have also tried this without success:

decoder_results <- decoder_lstm(decoder_inputs, initial_state=encoder_states)
decoder_dense   <- layer_dense(units=num_decoder_tokens, activation='softmax')
decoder_outputs <- decoder_results[[1]] %>% bidirectional(decoder_results) %>% decoder_dense()

The way I understand it, you cannot use a bidirectional LSTM in the decoder - bidirectional would mean it already knew what it still has to generate, right? Or am I missing something?

Thanks @skeydan, I am fine with that. But in case I would like to change the encoder only? I mean bidirectional encoder and the same decoder we have here. I could I pass states?

I think in that case you'd have to think about what to do with the states:

> encoder_states
[[1]]
Tensor("bidirectional_1/while/Exit_2:0", shape=(?, 10), dtype=float32)

[[2]]
Tensor("bidirectional_1/while/Exit_3:0", shape=(?, 10), dtype=float32)

[[3]]
Tensor("bidirectional_1/while_1/Exit_2:0", shape=(?, 10), dtype=float32)

[[4]]
Tensor("bidirectional_1/while_1/Exit_3:0", shape=(?, 10), dtype=float32)

You'll have 4 instead of 2, but you can only pass 2 to the decoder:

decoder_results <- decoder_lstm(decoder_inputs, initial_state=encoder_states)

You could experiment with e.g. averaging...

Thanks @skeydan , this makes sense, However I am not very convinced we can't use bidirectional layer in the decoder. If we use in time series prediction such as in:

https://blogs.rstudio.com/tensorflow/posts/2017-12-20-time-series-forecasting-with-recurrent-neural-networks/

for this reason:

RNNs are notably order dependent, or time dependent: they process the timesteps of their input sequences in order, and shuffling or reversing the timesteps can completely change the representations the RNN extracts from the sequence. This is precisely the reason they perform well on problems where order is meaningful, such as the temperature-forecasting problem. A bidirectional RNN exploits the order sensitivity of RNNs: it consists of using two regular RNNs, such as the layer_gru and layer_lstm you’re already familiar with, each of which processes the input sequence in one direction (chronologically and antichronologically), and then merging their representations. By processing a sequence both ways, a bidirectional RNN can catch patterns that may be overlooked by a unidirectional RNN.

why can't this be used in a decoder? There are some articles on this such as:

https://arxiv.org/abs/1809.06662

What do you think about it?

I will try to implement the solution of the following in R:

https://stackoverflow.com/questions/50815354/seq2seq-bidirectional-encoder-decoder-in-keras

and see if it works. The key seems to be to use n_units*2 in the decoder layer. Please give me some time. Thanks.

Yeah, concatenating the states makes more sense than averaging. They don’t use a bidirectional decoder though, either. And what I meant was, when decoding the future tokens aren’t known yet, so logically it makes no sense (in contrast to the encoding task)

Yeah, concatenating the states makes more sense than averaging. They don’t use a bidirectional decoder though, either. And what I meant was, when decoding the future tokens aren’t known yet, so logically it makes no sense (in contrast to the encoding task)

This is because you think to prediction task. But if you think to anomaly detection or denoising task, don't you think it might be possible? This article is regarding a bidirectional LSTM autoencoder with 2 layers each:

https://arxiv.org/abs/1711.09919

I will post you a short example later. In the mean time, regarding this example:

https://keras.rstudio.com/articles/examples/lstm_seq2seq.html

is the part:

## Save model
save_model_hdf5(model,'s2s.h5')
save_model_weights_hdf5(model,'s2s-wt.h5')

suppose to work? In my case I am getting the following warnings:

C:\Users\XMACHI~1\AppData\Local\conda\conda\envs\TENSOR~1\lib\site-packages\keras\engine\network.py:877: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1/while/Exit_2:0' shape=(?, 64) dtype=float32>, <tf.Tensor 'lstm_1/while/Exit_3:0' shape=(?, 64) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time). '. They will not be included ' C:\Users\XMACHI~1\AppData\Local\conda\conda\envs\TENSOR~1\lib\site-packages\keras\engine\network.py:877: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'input_3:0' shape=(?, 64) dtype=float32>, <tf.Tensor 'input_4:0' shape=(?, 64) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time). '. They will not be included '

while saving and the following while loading:

C:\Users\XMACHI~1\AppData\Local\conda\conda\envs\TENSOR~1\lib\site-packages\keras\engine\saving.py:292: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.

I have noticed some threads in the Python forum, but I can't understand if this problem is solved.

This is because you think to prediction task. But if you think to anomaly detection or denoising task, don't you think it might be possible?

Right, I was thinking of using seq2seq to generate output (like in the example). Autoencoding would be a different matter...

Regarding the serialization warning, I don't get it with Keras 2.2.4 and not with 2.2.2 either...

@skeydan, If I use serialization:


#' Sequence to sequence example in Keras (character-level).
#'
#' This script demonstrates how to implement a basic character-level
#' sequence-to-sequence model. We apply it to translating
#' short English sentences into short French sentences,
#' character-by-character. Note that it is fairly unusual to
#' do character-level machine translation, as word-level
#' models are more common in this domain.
#'
#' **Algorithm**
#'
#' - We start with input sequences from a domain (e.g. English sentences)
#'     and correspding target sequences from another domain
#'     (e.g. French sentences).
#' - An encoder LSTM turns input sequences to 2 state vectors
#'     (we keep the last LSTM state and discard the outputs).
#' - A decoder LSTM is trained to turn the target sequences into
#'     the same sequence but offset by one timestep in the future,
#'     a training process called "teacher forcing" in this context.
#'     Is uses as initial state the state vectors from the encoder.
#'     Effectively, the decoder learns to generate `targets[t+1...]`
#'     given `targets[...t]`, conditioned on the input sequence.
#' - In inference mode, when we want to decode unknown input sequences, we:
#'     - Encode the input sequence into state vectors
#'     - Start with a target sequence of size 1
#'         (just the start-of-sequence character)
#'     - Feed the state vectors and 1-char target sequence
#'         to the decoder to produce predictions for the next character
#'     - Sample the next character using these predictions
#'         (we simply use argmax).
#'     - Append the sampled character to the target sequence
#'     - Repeat until we generate the end-of-sequence character or we
#'         hit the character limit.
#'
#' **Data download**
#'
#' English to French sentence pairs.
#' http://www.manythings.org/anki/fra-eng.zip
#'
#' Lots of neat sentence pairs datasets can be found at:
#' http://www.manythings.org/anki/
#'
#' **References**
#'
#' - Sequence to Sequence Learning with Neural Networks
#'     https://arxiv.org/abs/1409.3215
#' - Learning Phrase Representations using
#'     RNN Encoder-Decoder for Statistical Machine Translation
#'     https://arxiv.org/abs/1406.1078

library(keras)
library(data.table)

batch_size = 64  # Batch size for training.
epochs = 1  # Number of epochs to train for.
latent_dim = 256  # Latent dimensionality of the encoding space.
num_samples = 10000  # Number of samples to train on.

## Path to the data txt file on disk.
data_path = 'fra.txt'
text <- fread(data_path, sep="\t", header=FALSE, nrows=num_samples)

## Vectorize the data.
input_texts  <- text[[1]]
target_texts <- paste0('\t',text[[2]],'\n')
input_texts  <- lapply( input_texts, function(s) strsplit(s, split="")[[1]])
target_texts <- lapply( target_texts, function(s) strsplit(s, split="")[[1]])

input_characters  <- sort(unique(unlist(input_texts)))
target_characters <- sort(unique(unlist(target_texts)))
num_encoder_tokens <- length(input_characters)
num_decoder_tokens <- length(target_characters)
max_encoder_seq_length <- max(sapply(input_texts,length))
max_decoder_seq_length <- max(sapply(target_texts,length))

cat('Number of samples:', length(input_texts),'\n')
cat('Number of unique input tokens:', num_encoder_tokens,'\n')
cat('Number of unique output tokens:', num_decoder_tokens,'\n')
cat('Max sequence length for inputs:', max_encoder_seq_length,'\n')
cat('Max sequence length for outputs:', max_decoder_seq_length,'\n')

input_token_index  <- 1:length(input_characters)
names(input_token_index) <- input_characters
target_token_index <- 1:length(target_characters)
names(target_token_index) <- target_characters
encoder_input_data <- array(
  0, dim = c(length(input_texts), max_encoder_seq_length, num_encoder_tokens))
decoder_input_data <- array(
  0, dim = c(length(input_texts), max_decoder_seq_length, num_decoder_tokens))
decoder_target_data <- array(
  0, dim = c(length(input_texts), max_decoder_seq_length, num_decoder_tokens))

for(i in 1:length(input_texts)) {
  d1 <- sapply( input_characters, function(x) { as.integer(x == input_texts[[i]]) })
  encoder_input_data[i,1:nrow(d1),] <- d1
  d2 <- sapply( target_characters, function(x) { as.integer(x == target_texts[[i]]) })
  decoder_input_data[i,1:nrow(d2),] <- d2
  d3 <- sapply( target_characters, function(x) { as.integer(x == target_texts[[i]][-1]) })
  decoder_target_data[i,1:nrow(d3),] <- d3
}

##----------------------------------------------------------------------
## Create the model
##----------------------------------------------------------------------

## Define an input sequence and process it.
encoder_inputs  <- layer_input(shape=list(NULL,num_encoder_tokens))
encoder         <- layer_lstm(units=latent_dim, return_state=TRUE)
encoder_results <- encoder_inputs %>% encoder
## We discard `encoder_outputs` and only keep the states.
encoder_states  <- encoder_results[2:3]

## Set up the decoder, using `encoder_states` as initial state.
decoder_inputs  <- layer_input(shape=list(NULL, num_decoder_tokens))
## We set up our decoder to return full output sequences,
## and to return internal states as well. We don't use the
## return states in the training model, but we will use them in inference.
decoder_lstm    <- layer_lstm(units=latent_dim, return_sequences=TRUE,
                              return_state=TRUE, stateful=FALSE)
decoder_results <- decoder_lstm(decoder_inputs, initial_state=encoder_states)
decoder_dense   <- layer_dense(units=num_decoder_tokens, activation='softmax')
decoder_outputs <- decoder_dense(decoder_results[[1]])

## Define the model that will turn
## `encoder_input_data` & `decoder_input_data` into `decoder_target_data`
model <- keras_model( inputs = list(encoder_inputs, decoder_inputs),
                      outputs = decoder_outputs )

## Compile model
model %>% compile(optimizer='rmsprop', loss='categorical_crossentropy')

## Run model
model %>% fit( list(encoder_input_data, decoder_input_data), decoder_target_data,
               batch_size=batch_size,
               epochs=epochs,
               validation_split=0.2)

##----------------------------------------------------------------------
## Next: inference mode (sampling).
##----------------------------------------------------------------------
## Here's the drill:
## 1) encode input and retrieve initial decoder state
## 2) run one step of decoder with this initial state
## and a "start of sequence" token as target.
## Output will be the next target token
## 3) Repeat with the current target token and current states

## Define sampling models
encoder_model <-  keras_model(encoder_inputs, encoder_states)
decoder_state_input_h <- layer_input(shape=latent_dim)
decoder_state_input_c <- layer_input(shape=latent_dim)
decoder_states_inputs <- c(decoder_state_input_h, decoder_state_input_c)
decoder_results <- decoder_lstm(decoder_inputs, initial_state=decoder_states_inputs)
decoder_states  <- decoder_results[2:3]
decoder_outputs <- decoder_dense(decoder_results[[1]])
decoder_model   <- keras_model(
  inputs  = c(decoder_inputs, decoder_states_inputs),
  outputs = c(decoder_outputs, decoder_states))

## Save model

model <- serialize_model(model)
encoder_model <- serialize_model(encoder_model)
decoder_model <- serialize_model(decoder_model)
save(model,encoder_model,decoder_model, file="test")

rm(model,encoder_model,decoder_model)

load(file="test")
model <- unserialize_model(model)
encoder_model <- unserialize_model(encoder_model)
decoder_model <- unserialize_model(decoder_model)

## Reverse-lookup token index to decode sequences back to
## something readable.
reverse_input_char_index  <- as.character(input_characters)
reverse_target_char_index <- as.character(target_characters)

decode_sequence <- function(input_seq) {
  ## Encode the input as state vectors.
  states_value <- predict(encoder_model, input_seq)

  ## Generate empty target sequence of length 1.
  target_seq <- array(0, dim=c(1, 1, num_decoder_tokens))
  ## Populate the first character of target sequence with the start character.
  target_seq[1, 1, target_token_index['\t']] <- 1.

  ## Sampling loop for a batch of sequences
  ## (to simplify, here we assume a batch of size 1).
  stop_condition = FALSE
  decoded_sentence = ''
  maxiter = max_decoder_seq_length
  niter = 1
  while (!stop_condition && niter < maxiter) {

    ## output_tokens, h, c = decoder_model.predict([target_seq] + states_value)
    decoder_predict <- predict(decoder_model, c(list(target_seq), states_value))
    output_tokens <- decoder_predict[[1]]

    ## Sample a token
    sampled_token_index <- which.max(output_tokens[1, 1, ])
    sampled_char <- reverse_target_char_index[sampled_token_index]
    decoded_sentence <-  paste0(decoded_sentence, sampled_char)
    decoded_sentence

    ## Exit condition: either hit max length
    ## or find stop character.
    if (sampled_char == '\n' ||
        length(decoded_sentence) > max_decoder_seq_length) {
      stop_condition = TRUE
    }

    ## Update the target sequence (of length 1).
    ## target_seq = np.zeros((1, 1, num_decoder_tokens))
    target_seq[1, 1, ] <- 0
    target_seq[1, 1, sampled_token_index] <- 1.

    ## Update states
    h <- decoder_predict[[2]]
    c <- decoder_predict[[3]]
    states_value = list(h, c)
    niter <- niter + 1
  }    
  return(decoded_sentence)
}

for (seq_index in 1:100) {
  ## Take one sequence (part of the training test)
  ## for trying out decoding.
  input_seq = encoder_input_data[seq_index,,,drop=FALSE]
  decoded_sentence = decode_sequence(input_seq)
  target_sentence <- gsub("\t|\n","",paste(target_texts[[seq_index]],collapse=''))
  input_sentence  <- paste(input_texts[[seq_index]],collapse='')
  cat('-\n')
  cat('Input sentence  : ', input_sentence,'\n')
  cat('Target sentence : ', target_sentence,'\n')
  cat('Decoded sentence: ', decoded_sentence,'\n')
}

It still works but I am getting the following:

model <- serialize_model(model) C:\Users\XMACHI~1\AppData\Local\conda\conda\envs\TENSOR~1\lib\site-packages\keras\engine\network.py:877: UserWarning: Layer lstm_12 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_11/while/Exit_2:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'lstm_11/while/Exit_3:0' shape=(?, 256) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time). '. They will not be included ' encoder_model <- serialize_model(encoder_model) decoder_model <- serialize_model(decoder_model) C:\Users\XMACHI~1\AppData\Local\conda\conda\envs\TENSOR~1\lib\site-packages\keras\engine\network.py:877: UserWarning: Layer lstm_12 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'input_23:0' shape=(?, 256) dtype=float32>, <tf.Tensor 'input_24:0' shape=(?, 256) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time). '. They will not be included ' save(model,encoder_model,decoder_model, file="test")

rm(model,encoder_model,decoder_model)

load(file="test") model <- unserialize_model(model) encoder_model <- unserialize_model(encoder_model) C:\Users\XMACHI~1\AppData\Local\conda\conda\envs\TENSOR~1\lib\site-packages\keras\engine\saving.py:292: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' decoder_model <- unserialize_model(decoder_model)

What is the reason of these error messages? My version is 2.2.4

I don't get these warnings with keras 2.2.4.

To simplify, what happens if you just do

encoder_inputs  <- layer_input(shape=list(NULL,10))
encoder         <- layer_lstm(units=2, return_state=TRUE)
encoder_results <- encoder_inputs %>% encoder
encoder_states  <- encoder_results[2:3]

decoder_inputs  <- layer_input(shape=list(NULL, 10))
decoder_lstm    <- layer_lstm(units=2, return_sequences=TRUE,
                              return_state=TRUE, stateful=FALSE)
decoder_results <- decoder_lstm(decoder_inputs, initial_state=encoder_states)
decoder_dense   <- layer_dense(units=10, activation='softmax')
decoder_outputs <- decoder_dense(decoder_results[[1]])

model <- keras_model( inputs = list(encoder_inputs, decoder_inputs),
                      outputs = decoder_outputs )

model %>% compile(optimizer='rmsprop', loss='categorical_crossentropy')
model <- serialize_model(model)

Also, could you indicate the output of

keras:::keras_version()
tensorflow::tf_config()

@skeydan I am getting this:

model <- serialize_model(model) C:\Users\XMACHI~1\AppData\Local\conda\conda\envs\TENSOR~1\lib\site-packages\keras\engine\network.py:877: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1/while/Exit_2:0' shape=(?, 2) dtype=float32>, <tf.Tensor 'lstm_1/while/Exit_3:0' shape=(?, 2) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time). '. They will not be included '

this is my data:

keras:::keras_version() [1] '2.2.4' tensorflow::tf_config() TensorFlow v1.12.0 (C:\Users\XMACHI~1\AppData\Local\conda\conda\envs\TENSOR~1\lib\site-packages\keras__init__.p) Python v3.6 (C:\Users\Xmachines\AppData\Local\conda\conda\envs\tensorflow-gpu\python.exe)

What do you think?

Hi @skeydan , can you reproduce this issue?

Like I said, I can't (with same versions of Keras and TF).

Can you check if you get the same results/accuracy after training a model, vs. after saving and loading it?

Different question, do you also get the warning when you just save the weights?

This model is not making out-of-sample testing. So apparently I am getting the same results, but I don't think it is relevant. I need to test with an example making out-of-sample testing. For the second question, yes, I am getting the same message saving the weights. Have you tested in Windows 10 Pro?

No, I have linux, and from searching SO etc. this seems to occur for (some) Windows users and (some) MAC users using some versions ... No idea if this is relevant, but can you check the version of h5py in the environment you're using for keras?

A bit strange. The following command:

conda info hdf5

gives me the following result:


hdf5 1.10.1 h08c2580_1
----------------------
file name   : hdf5-1.10.1-h08c2580_1.tar.bz2
name        : hdf5
version     : 1.10.1
build string: h08c2580_1
build number: 1
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 17.6 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : a86636ebfddf6c8909a61fc944a6b0f8
platform    : None
subdir      : win-64
timestamp   : 1510863827266
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.1-h08c2580_1.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.1 h79de857_2
----------------------
file name   : hdf5-1.10.1-h79de857_2.tar.bz2
name        : hdf5
version     : 1.10.1
build string: h79de857_2
build number: 2
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 17.7 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 939b95fd5db63c808bed321e362d9c8b
platform    : None
subdir      : win-64
timestamp   : 1517073383707
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.1-h79de857_2.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.1 h98b8871_1
----------------------
file name   : hdf5-1.10.1-h98b8871_1.tar.bz2
name        : hdf5
version     : 1.10.1
build string: h98b8871_1
build number: 1
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 33.4 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 405ee1ad1df3c2a24856a27fe520d6bb
platform    : None
subdir      : win-64
timestamp   : 1510865045089
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.1-h98b8871_1.tar.bz2
dependencies:
    icc_rt >=16.0.4
    vc 14.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.1 vc14hb361328_0
--------------------------
file name   : hdf5-1.10.1-vc14hb361328_0.tar.bz2
name        : hdf5
version     : 1.10.1
build string: vc14hb361328_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 33.5 MB
arch        : None
constrains  : ()
features    : 
license     : HDF5
license_family: BSD
md5         : 78143be95c90e53a24d7256b394a150e
platform    : None
subdir      : win-64
timestamp   : 1506031687729
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.1-vc14hb361328_0.tar.bz2
dependencies:
    icc_rt >=16.0.4
    vc 14.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.1 vc14hbc849ce_0
--------------------------
file name   : hdf5-1.10.1-vc14hbc849ce_0.tar.bz2
name        : hdf5
version     : 1.10.1
build string: vc14hbc849ce_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 33.5 MB
arch        : None
constrains  : ()
features    : 
license     : HDF5
license_family: BSD
md5         : 94c4033eeb60a8a508e8049e02f68a79
platform    : None
subdir      : win-64
timestamp   : 1506013715525
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.1-vc14hbc849ce_0.tar.bz2
dependencies:
    icc_rt >=16.0.4
    vc 14.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.1 vc9h4ddfba0_0
-------------------------
file name   : hdf5-1.10.1-vc9h4ddfba0_0.tar.bz2
name        : hdf5
version     : 1.10.1
build string: vc9h4ddfba0_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 17.6 MB
arch        : None
constrains  : ()
features    : 
license     : HDF5
license_family: BSD
md5         : be8c591e7bd8443f7e0fa7064ecffd85
platform    : None
subdir      : win-64
timestamp   : 1506010119406
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.1-vc9h4ddfba0_0.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.1 vc9h9d13b45_0
-------------------------
file name   : hdf5-1.10.1-vc9h9d13b45_0.tar.bz2
name        : hdf5
version     : 1.10.1
build string: vc9h9d13b45_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 17.6 MB
arch        : None
constrains  : ()
features    : 
license     : HDF5
license_family: BSD
md5         : 3476ab409860c47fe84ed90758958f2c
platform    : None
subdir      : win-64
timestamp   : 1506030586378
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.1-vc9h9d13b45_0.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.2 h4b43f98_1
----------------------
file name   : hdf5-1.10.2-h4b43f98_1.tar.bz2
name        : hdf5
version     : 1.10.2
build string: h4b43f98_1
build number: 1
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 18.6 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 9bc995c34b809e6518d89ac47c31557b
platform    : None
subdir      : win-64
timestamp   : 1525884327264
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.2-h4b43f98_1.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.2 h530792d_2
----------------------
file name   : hdf5-1.10.2-h530792d_2.tar.bz2
name        : hdf5
version     : 1.10.2
build string: h530792d_2
build number: 2
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 18.6 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 186832417a4633d9bd83828fba895472
platform    : None
subdir      : win-64
timestamp   : 1538893176581
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.2-h530792d_2.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.2 hac2f561_1
----------------------
file name   : hdf5-1.10.2-hac2f561_1.tar.bz2
name        : hdf5
version     : 1.10.2
build string: hac2f561_1
build number: 1
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 34.8 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 99b01107c025293ae1cb82c56fac243e
platform    : None
subdir      : win-64
timestamp   : 1525886151855
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.2-hac2f561_1.tar.bz2
dependencies:
    icc_rt >=16.0.4
    vc 14.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.4 h530792d_0
----------------------
file name   : hdf5-1.10.4-h530792d_0.tar.bz2
name        : hdf5
version     : 1.10.4
build string: h530792d_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 18.4 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 035536f1c2e1b9e5d9642f2726b99a7e
platform    : None
subdir      : win-64
timestamp   : 1545323296982
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.4-h530792d_0.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.10.4 h7ebc959_0
----------------------
file name   : hdf5-1.10.4-h7ebc959_0.tar.bz2
name        : hdf5
version     : 1.10.4
build string: h7ebc959_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 19.2 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : de2d3c434711af9c473efc9741189816
platform    : None
subdir      : win-64
timestamp   : 1545244707767
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.10.4-h7ebc959_0.tar.bz2
dependencies:
    icc_rt >=2019.0.0
    vc >=14.1,<15.0a0
    zlib >=1.2.11,<1.3.0a0

hdf5 1.8.18 h33e7809_2
----------------------
file name   : hdf5-1.8.18-h33e7809_2.tar.bz2
name        : hdf5
version     : 1.8.18
build string: h33e7809_2
build number: 2
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 11.9 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 88eb98e51e3de2f8dbc59b6b460d2b09
platform    : None
subdir      : win-64
timestamp   : 1517514935853
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.8.18-h33e7809_2.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.8.18 h7b6e65e_1
----------------------
file name   : hdf5-1.8.18-h7b6e65e_1.tar.bz2
name        : hdf5
version     : 1.8.18
build string: h7b6e65e_1
build number: 1
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 11.9 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : b7fca90c5f9c29c0b447dec4bbf7a517
platform    : None
subdir      : win-64
timestamp   : 1510863489678
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.8.18-h7b6e65e_1.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.8.18 hcf527f2_1
----------------------
file name   : hdf5-1.8.18-hcf527f2_1.tar.bz2
name        : hdf5
version     : 1.8.18
build string: hcf527f2_1
build number: 1
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 26.2 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : e896170bc53160a94baec84dd59548b6
platform    : None
subdir      : win-64
timestamp   : 1510864777471
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.8.18-hcf527f2_1.tar.bz2
dependencies:
    icc_rt >=16.0.4
    vc 14.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.8.18 vc14h7a021fe_0
--------------------------
file name   : hdf5-1.8.18-vc14h7a021fe_0.tar.bz2
name        : hdf5
version     : 1.8.18
build string: vc14h7a021fe_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 26.2 MB
arch        : None
constrains  : ()
features    : 
license     : HDF5
license_family: BSD
md5         : 9a9f7aa477f4ad9dc16ce3b8140b71e0
platform    : None
subdir      : win-64
timestamp   : 1506022918822
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.8.18-vc14h7a021fe_0.tar.bz2
dependencies:
    icc_rt >=16.0.4
    vc 14.*
    zlib 1.2.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.8.18 vc9hfb11c06_0
-------------------------
file name   : hdf5-1.8.18-vc9hfb11c06_0.tar.bz2
name        : hdf5
version     : 1.8.18
build string: vc9hfb11c06_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 11.9 MB
arch        : None
constrains  : ()
features    : 
license     : HDF5
license_family: BSD
md5         : d78e262db11eef0aa72b3c7ebb0dcb24
platform    : None
subdir      : win-64
timestamp   : 1506022378144
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.8.18-vc9hfb11c06_0.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib 1.2.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.8.20 h4b43f98_1
----------------------
file name   : hdf5-1.8.20-h4b43f98_1.tar.bz2
name        : hdf5
version     : 1.8.20
build string: h4b43f98_1
build number: 1
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 12.2 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 8a1f895e5160f829e84db60a8e101f51
platform    : None
subdir      : win-64
timestamp   : 1533201919004
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.8.20-h4b43f98_1.tar.bz2
dependencies:
    icc_rt >=13.1.6
    vc 9.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.8.20 hac2f561_1
----------------------
file name   : hdf5-1.8.20-hac2f561_1.tar.bz2
name        : hdf5
version     : 1.8.20
build string: hac2f561_1
build number: 1
channel     : https://repo.anaconda.com/pkgs/main/win-64
size        : 26.7 MB
arch        : None
constrains  : ()
license     : HDF5
license_family: BSD
md5         : 933b03dbb3394515ac3d8fa4649c1a98
platform    : None
subdir      : win-64
timestamp   : 1533203309221
url         : https://repo.anaconda.com/pkgs/main/win-64/hdf5-1.8.20-hac2f561_1.tar.bz2
dependencies:
    icc_rt >=16.0.4
    vc 14.*
    zlib >=1.2.11,<1.3.0a0

hdf5 1.8.13 0
-------------
file name   : hdf5-1.8.13-0.tar.bz2
name        : hdf5
version     : 1.8.13
build string: 0
build number: 0
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 2.3 MB
arch        : x86_64
constrains  : ()
date        : 2014-09-11
license     : BSD-like
license_family: BSD
md5         : 14fececf963b59dd7f1ba4045957a7c7
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.13-0.tar.bz2
dependencies:

hdf5 1.8.15.1 2
---------------
file name   : hdf5-1.8.15.1-2.tar.bz2
name        : hdf5
version     : 1.8.15.1
build string: 2
build number: 2
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 1.4 MB
arch        : x86_64
constrains  : ()
date        : 2015-06-22
license     : BSD-like
license_family: BSD
md5         : da9a90f534cf9e47b03156af5ddadd7d
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.15.1-2.tar.bz2
dependencies:
    zlib 1.2*

hdf5 1.8.15.1 vc10_3
--------------------
file name   : hdf5-1.8.15.1-vc10_3.tar.bz2
name        : hdf5
version     : 1.8.15.1
build string: vc10_3
build number: 3
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 1.4 MB
arch        : x86_64
constrains  : ()
date        : 2015-10-14
license     : BSD-like
license_family: BSD
md5         : e33594a7a8871a564b3cdf49db1d4675
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.15.1-vc10_3.tar.bz2
dependencies:
    vc 10
    zlib

hdf5 1.8.15.1 vc10_4
--------------------
file name   : hdf5-1.8.15.1-vc10_4.tar.bz2
name        : hdf5
version     : 1.8.15.1
build string: vc10_4
build number: 4
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 1.4 MB
arch        : x86_64
constrains  : ()
date        : 2015-11-30
license     : BSD-like
license_family: BSD
md5         : 56a26b25af42a5a6a0b0396428417208
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.15.1-vc10_4.tar.bz2
dependencies:
    vc 10
    zlib 1.2*

hdf5 1.8.15.1 vc14_3
--------------------
file name   : hdf5-1.8.15.1-vc14_3.tar.bz2
name        : hdf5
version     : 1.8.15.1
build string: vc14_3
build number: 3
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 1.6 MB
arch        : x86_64
constrains  : ()
date        : 2015-10-15
license     : BSD-like
license_family: BSD
md5         : 85eeb204341c34010d347cdbc67d2974
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.15.1-vc14_3.tar.bz2
dependencies:
    vc 14
    zlib

hdf5 1.8.15.1 vc14_4
--------------------
file name   : hdf5-1.8.15.1-vc14_4.tar.bz2
name        : hdf5
version     : 1.8.15.1
build string: vc14_4
build number: 4
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 1.5 MB
arch        : x86_64
constrains  : ()
date        : 2015-11-30
license     : BSD-like
license_family: BSD
md5         : 7fb4055ab0af78754044f18b1cee8f04
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.15.1-vc14_4.tar.bz2
dependencies:
    vc 14
    zlib 1.2*

hdf5 1.8.15.1 vc9_3
-------------------
file name   : hdf5-1.8.15.1-vc9_3.tar.bz2
name        : hdf5
version     : 1.8.15.1
build string: vc9_3
build number: 3
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 1.5 MB
arch        : x86_64
constrains  : ()
date        : 2015-10-14
license     : BSD-like
license_family: BSD
md5         : 15277b51954dcb35e3e03aa87efb9f28
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.15.1-vc9_3.tar.bz2
dependencies:
    vc 9
    zlib

hdf5 1.8.15.1 vc9_4
-------------------
file name   : hdf5-1.8.15.1-vc9_4.tar.bz2
name        : hdf5
version     : 1.8.15.1
build string: vc9_4
build number: 4
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 1.4 MB
arch        : x86_64
constrains  : ()
date        : 2015-11-30
license     : BSD-like
license_family: BSD
md5         : 1b13fdf67882759bcf73fb7cf44bba9f
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.15.1-vc9_4.tar.bz2
dependencies:
    vc 9
    zlib 1.2*

hdf5 1.8.16 vc10_0
------------------
file name   : hdf5-1.8.16-vc10_0.tar.bz2
name        : hdf5
version     : 1.8.16
build string: vc10_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 11.7 MB
arch        : x86_64
constrains  : ()
date        : 2016-04-14
license     : BSD-like
license_family: BSD
md5         : c4228cd63840e66f84138b6c46b23d81
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.16-vc10_0.tar.bz2
dependencies:
    vc 10
    zlib 1.2.*

hdf5 1.8.16 vc14_0
------------------
file name   : hdf5-1.8.16-vc14_0.tar.bz2
name        : hdf5
version     : 1.8.16
build string: vc14_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 12.3 MB
arch        : x86_64
constrains  : ()
date        : 2016-04-14
license     : BSD-like
license_family: BSD
md5         : c935a1d232cbe8fe09c1ffe0a64a322b
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.16-vc14_0.tar.bz2
dependencies:
    vc 14
    zlib 1.2.*

hdf5 1.8.16 vc9_0
-----------------
file name   : hdf5-1.8.16-vc9_0.tar.bz2
name        : hdf5
version     : 1.8.16
build string: vc9_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 11.8 MB
arch        : x86_64
constrains  : ()
date        : 2016-04-14
license     : BSD-like
license_family: BSD
md5         : 63a87df2327c8fa5431cb7da58b884a7
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.16-vc9_0.tar.bz2
dependencies:
    vc 9
    zlib 1.2.*

hdf5 1.8.17 vc10_0
------------------
file name   : hdf5-1.8.17-vc10_0.tar.bz2
name        : hdf5
version     : 1.8.17
build string: vc10_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 11.8 MB
arch        : x86_64
constrains  : ()
date        : 2016-07-20
license     : BSD-like
license_family: BSD
md5         : 4ddcf5f3db712b13d14807926a4483a6
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.17-vc10_0.tar.bz2
dependencies:
    vc 10
    zlib 1.2.*

hdf5 1.8.17 vc14_0
------------------
file name   : hdf5-1.8.17-vc14_0.tar.bz2
name        : hdf5
version     : 1.8.17
build string: vc14_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 12.3 MB
arch        : x86_64
constrains  : ()
date        : 2016-07-20
license     : BSD-like
license_family: BSD
md5         : 0646e7557d751ffacc468398b4820fec
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.17-vc14_0.tar.bz2
dependencies:
    vc 14
    zlib 1.2.*

hdf5 1.8.17 vc9_0
-----------------
file name   : hdf5-1.8.17-vc9_0.tar.bz2
name        : hdf5
version     : 1.8.17
build string: vc9_0
build number: 0
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 11.8 MB
arch        : x86_64
constrains  : ()
date        : 2016-07-20
license     : BSD-like
license_family: BSD
md5         : 37f9ca638564560ba2983edc5c151617
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.17-vc9_0.tar.bz2
dependencies:
    vc 9
    zlib 1.2.*

hdf5 1.8.9 0
------------
file name   : hdf5-1.8.9-0.tar.bz2
name        : hdf5
version     : 1.8.9
build string: 0
build number: 0
channel     : https://repo.anaconda.com/pkgs/free/win-64
size        : 2.4 MB
arch        : x86_64
constrains  : ()
date        : 2012-11-06
license     : BSD-like
license_family: BSD
md5         : f52c2283c6a251072f8bc7faedbd3bf6
platform    : win32
subdir      : win-64
url         : https://repo.anaconda.com/pkgs/free/win-64/hdf5-1.8.9-0.tar.bz2
dependencies:

What I meant was the version of h5py indicated by conda list in that env, but I don't know if that will lead us anywhere...

Hi @skeydan, the following is my 'conda list'

(base) C:\Users\Xmachines>conda list
# packages in environment at C:\ProgramData\Miniconda3:
#
# Name                    Version                   Build  Channel
anaconda-client           1.7.2                    py37_0
anaconda-navigator        1.9.6                    py37_0
asn1crypto                0.24.0                   py37_0
ca-certificates           2018.03.07                    0
certifi                   2018.11.29               py37_0
cffi                      1.11.5           py37h74b6da3_1
chardet                   3.0.4                    py37_1
clyent                    1.2.2                    py37_1
conda                     4.5.12                   py37_0
conda-env                 2.6.0                         1
console_shortcut          0.1.1                         3
cryptography              2.4.2            py37h7a1dbc1_0
decorator                 4.3.0                    py37_0
freetype                  2.9.1                ha9979f8_1
html5lib                  1.0.1                    py37_0
icu                       58.2                 ha66f8fd_1
idna                      2.8                      py37_0
ipython_genutils          0.2.0                    py37_0
jpeg                      9b                   hb83a4c4_2
jsonschema                2.6.0                    py37_0
jupyter_core              4.4.0                    py37_0
libpng                    1.6.35               h2a8f88b_0
libtiff                   4.0.9                h36446d0_2
menuinst                  1.4.14           py37hfa6e2cd_0
nbformat                  4.4.0                    py37_0
olefile                   0.46                     py37_0
openssl                   1.1.1a               he774522_0
pillow                    5.3.0            py37hdc69c19_0
pip                       18.1                     py37_0
psutil                    5.4.8            py37he774522_0
pycosat                   0.6.3            py37hfa6e2cd_0
pycparser                 2.19                     py37_0
pyopenssl                 18.0.0                   py37_0
pyqt                      5.9.2            py37h6538335_2
pysocks                   1.6.8                    py37_0
python                    3.7.1                h8c8aaf0_6
python-dateutil           2.7.5                    py37_0
pytz                      2018.7                   py37_0
pywin32                   223              py37hfa6e2cd_1
pyyaml                    3.13             py37hfa6e2cd_0
qt                        5.9.7            vc14h73c81de_0
qtpy                      1.5.2                    py37_0
requests                  2.21.0                   py37_0
ruamel_yaml               0.15.46          py37hfa6e2cd_0
setuptools                40.6.3                   py37_0
sip                       4.19.8           py37h6538335_0
six                       1.12.0                   py37_0
sqlite                    3.26.0               he774522_0
tk                        8.6.8                hfa6e2cd_0
traitlets                 4.3.2                    py37_0
urllib3                   1.24.1                   py37_0
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.15.26706          h3a45250_0
webencodings              0.5.1                    py37_1
wheel                     0.32.3                   py37_0
win_inet_pton             1.0.1                    py37_1
wincertstore              0.2                      py37_0
yaml                      0.1.7                hc54c509_2
zlib                      1.2.11               h62dcd97_3

can h5py be included in another package?

I think you would need to check in the r-tensorflow environment... It looks like this for me:

h5py                      2.9.0           mpi_openmpi_hff00661_1001    conda-forge
h5py                      2.9.0                     <pip>

(no idea why there are 2 of them)

But this is might not lead anywhere anyway... From

https://github.com/keras-team/keras/issues/9914

I see that in Python-Keras, they've added an additional example

https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq_restore.py

which does a bit more work when restoring a saved model.

I assume that if you adapt your code accordingly, you will still get the warning while saving but hopefully, the results will be ok.

At the current time, it is pretty fair to say that this way of doing seq2seq will be bit a bit obsoleted quite soon, as seq2seq per se lends itself much better to eager execution.... This is why I'd rather not spend too much time now on optimizing that example. Instead, we might replace it with an eager version not-too-far in time. The eager version would look be similar to

https://blogs.rstudio.com/tensorflow/posts/2018-07-30-attention-layer/

but simpler, as it would not have an attention module, just an encoder and a decoder.

My h5py version is 2.9.0 Do you have a plan to update the existing version and replace with attention?

So the version is the same...

Do you have a plan to update the existing version and replace with attention?

No, I meant a simpler version without attention (doing basically the same thing as that example). However I don't have the time for it now.

For now, could you try what I suggested in the middle of my above comment

I see that in Python-Keras, they've added an additional example

https://github.com/keras-team/keras/blob/master/examples/lstm_seq2seq_restore.py

which does a bit more work when restoring a saved model.

I assume that if you adapt your code accordingly, you will still get the warning while saving but hopefully, the results will be ok.

and see if that yields correct results?

(Sorry for not being more helpful at this current point.)

Any updates for solutions on this?

I didn't have time to check it but it should work as proposed by @skeydan

So what exactly was the issue? Was it a problem with passing 'initial_state'? I am asking because I am working on a similar problem encountering a similar issue, though not in the keras seq2seq example. I apologize for maybe asking an elementary question, I am just starting out in deep learning.

no the issue is in saving the model

Essentially that initial_state is not saved when saving the model?

it is giving me an error message

Is there any sample code that defines the initial states for a Bidirectional layer ?

@MeJokar Here is the bidirectional model

enc_in = Input(shape=(None, self.total_characters), name='enc_in')
enc_lstm_1 = Bidirectional(CuDNNLSTM(units=128, return_sequences=True, return_state=True, name='enc_lstm_1'))
enc_lstm_2 = Bidirectional(CuDNNLSTM(units=64, return_sequences=False, return_state=True, name='enc_lstm_2'))
enc_out_1, forward_eh_1, forward_ec_1, back_eh_1, back_ec_1 = enc_lstm_1(enc_in)
enc_out_2, forward_eh_2, forward_ec_2, back_eh_2, back_ec_2 = enc_lstm_2(enc_out_1)

enc_states = [Concatenate()([forward_eh_1, back_eh_1]), Concatenate()([forward_ec_1, back_ec_1]),
              Concatenate()([forward_eh_2, back_eh_2]), Concatenate()([forward_ec_2, back_ec_2])]

dec_in = Input(shape=(None, self.total_characters), name='dec_in')

dec_lstm_1 = CuDNNLSTM(units=256, return_sequences=True, return_state=True, name='dec_lstm_1')
dec_lstm_2 = CuDNNLSTM(units=128, return_sequences=True, return_state=True, name='dec_lstm_2')

dec_out_1, _, _ = dec_lstm_1(dec_in, initial_state=enc_states[:2])
dec_out_2, _, _ = dec_lstm_2(dec_out_1, initial_state=enc_states[2:4])
dec_dense = Dense(units=self.total_characters, activation='softmax', name='dec_dense')
dec_out_f = dec_dense(dec_out_2)

model = Model([enc_in, dec_in], dec_out_f)
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')
model.summary()

I, too can't save the states correctly...

rstudio / keras3

Passing states to a bidirectional LSTM decoder #624