Anybody succeeded to run the image captioning task?

[Jiakui]

I am just wondering is it possible to use the current code to run the image captioning task. If so, can anybody give me some hint?

Thanks!

[kyleyeung]

@Jiakui The author has already released an experimental model here: https://github.com/google/seq2seq/blob/master/seq2seq/models/image2seq.py. It works, but needs a few modification.

I'm afraid there could be some problems with it. The training stopped after several thousands of steps, gonna work on it later.

[syed-ahmed]

Hi @Jiakui , I got it working with the following config. You would just have to create this yaml config file and use it in the train script:

model: Image2Seq
model_params:
  attention.class: AttentionLayerBahdanau
  attention.params:
    num_units: 512
  bridge.class: InitialStateBridge
  embedding.dim: 128
  encoder.class: InceptionV3Encoder
  decoder.class: AttentionDecoder
  decoder.params:
    rnn_cell:
      cell_class: LSTMCell
      cell_params:
        num_units: 512
      dropout_input_keep_prob: 0.8
      dropout_output_keep_prob: 1.0
      num_layers: 1
  optimizer.name: Adam
  optimizer.params:
    epsilon: 0.0000008
  optimizer.learning_rate: 0.0001
  target.max_seq_len: 50

You would also have to consider the data input pipeline and see how the TFRecords are being read and set the features keys accordingly (https://github.com/google/seq2seq/blob/master/seq2seq/data/input_pipeline.py#L281)