Unable to use pre-trained weight for another training

[takuma-yoneda]

Currently, I am trying to use pre-trained DAM/ESIM model to train on another task. (i.e., train model on SNLI and use that weight as an initial weight for another training)

In model_training.ipynb, I expected that replacing cell 23

snli_reader = readers.readers["dam_snli_reader"](shared_resources)
snli_reader.setup_from_data(snli_train_data, is_training=True)

with

snli_reader = readers.reader_from_file("dam")

makes it work. However, I got the following error.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-13-dcdc0b9508d6> in <module>()
      3                   hooks=hooks,
      4                   max_epochs=20,
----> 5                   training_set=snli_train_data)

~/workspace/UCL/fever/jack/jack/core/tensorflow.py in train(self, optimizer, training_set, batch_size, max_epochs, hooks, l2, clip, clip_op, summary_writer, **kwargs)
    218         """
    219         batches, loss, min_op, summaries = self._setup_training(
--> 220             batch_size, clip, optimizer, training_set, summary_writer, l2, clip_op, **kwargs)
    221 
    222         self._train_loop(min_op, loss, batches, hooks, max_epochs, summaries, summary_writer, **kwargs)

~/workspace/UCL/fever/jack/jack/core/tensorflow.py in _setup_training(self, batch_size, clip, optimizer, training_set, summary_writer, l2, clip_op, **kwargs)
    233         batches = self.input_module.batch_generator(training_set, batch_size, is_eval=False)
    234         logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
--> 235         loss = self.model_module.tensors[Ports.loss]
    236         summaries = None
    237         if summary_writer is not None:

KeyError: <TensorPort (loss)>

I could not really spend time for investigating it, but if there is a proper way to use pre-trained model for another training, please teach me.

[dirkweissenborn]

hi, did you try load_dir=./dam in addition to your command line call? this should load pretrained models. Should work also for partial models.

[dirkweissenborn]

readers.reader_from_file("dam") sets up the reader in eval mode, so it doesn't create the training part of the model. So in a notebook you can simply do:

snli_reader = readers.readers["dam_snli_reader"](shared_resources)
snli_reader.setup_from_data(snli_train_data, is_training=True)
snli_reader.load('dam')

[dirkweissenborn]

@takuma-ynd just to add, for training you might want to think about using the command-line. The notebook is just an example for understanding the inner workings of jack. What is your task? The simplest way to set this up would be to convert your data into jack-json format, see example file .

[takuma-yoneda]

@dirkweissenborn Just adding load_dir=./dam in command line call worked!

As you said, I've been currently converting the dataset to jack-json format and using it! Thanks for the kind explanation!