Pre-trained model

[vshan]

Hello, I would like to run this project, however I lack the GPU resources to train the model. Could a pre-trained model be shared so that it can be downloaded and the project can quickly be replicated locally?

[chrisc36]

Yes, I would like to add pre-trained models. For now, here is a link to the model being used in the demo (which is cpu-compatible). :

https://drive.google.com/open?id=1RA3wf5NX0GS2LLWPcPcJXl91OEKETHmy

Adding the full set of models will take a bit more time to setup, but its on the TODO list.

[vinaymundada27]

@chrisc36 Thanks for sharing the trained model! Could you please also give the exact details of directly running the test/evaluation part using your given trained model? We tried running as follows : python docqa/eval/squad_eval.py -o output.json --ema -c dev open-shared-norm-cpu/ But it gave the following errors :

~/document-qa$ python docqa/eval/squad_eval.py -o output.json --ema -c dev open-shared-norm-cpu
Using latest checkpoint
Setting up model
Loading word vec glove.840B.300d for SquadCorpus from cache
Building embeddings for 3 special_tokens
Had pre-trained word embeddings for 26244 of 27612 words
2017-11-09 11:13:20.225290: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-09 11:13:20.225410: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-11-09 11:13:20.225431: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
Restoring variables
Restoring EMA variables
2017-11-09 11:13:32.155645: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key apply_attention/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.163883: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/input_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.174874: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key char-embed/embedding/conv1d/filters/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.190741: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key apply_attention/dot_w/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.201062: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/gates/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.202067: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/input_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.203051: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/hidden_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.207187: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/hidden_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.207457: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key char-embed/char_emb_mat/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.208215: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/hidden_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.209055: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/gates/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.209462: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/gates/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.209632: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/hidden_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.209781: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/input_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.210971: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/hidden_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.211291: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/hidden_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.211609: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key apply_attention/input_w/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.211576: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/gates/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.211940: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/gates/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.212241: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/input_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.212459: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/gates/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.213812: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/hidden_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.213970: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/gates/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.214315: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key apply_attention/key_w/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.214498: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key char-embed/embedding/conv1d/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.214748: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/input_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.214858: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/gates/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.214950: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/input_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.215135: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/input_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.215245: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/hidden_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.215529: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/input_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.218428: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/input_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.218691: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/hidden_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.219755: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/input_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.220078: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/input_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.219335: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/input_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.222893: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/hidden_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.226270: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/hidden_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.226482: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key map_embed/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/hidden_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.226581: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/gates/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.226698: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/chained-out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/gates/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.227400: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/start_pred/fully_connected/biases/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.227725: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/bw/cudnn_compatible_gru_cell/gates/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.228050: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key token_embed/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.229473: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/gates/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.233411: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/gates/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.239219: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/start_pred/fully_connected/weights/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.239464: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/input_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.239483: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/input_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.240808: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_4/dense/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.241237: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/hidden_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.241958: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_0/dense/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.243079: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/gates/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.243300: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_4/dense/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.243467: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/gates/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.244308: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/hidden_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.244661: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_3/no-alignment-bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.245639: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/bw/cudnn_compatible_gru_cell/candidate/input_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.245757: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/input_projection/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.245862: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/end_pred/fully_connected/biases/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.246114: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_3/dot_w/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.246404: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/hidden_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.246721: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_3/input_w/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.246874: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_3/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.251395: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/bounds_encoding/out/bidirectional_rnn/fw/cudnn_compatible_gru_cell/candidate/hidden_projection/bias/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.251856: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_1/bidirectional_rnn/fw/cudnn_compatible_gru_cell/gates/kernel/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.252160: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_1/layer_3/key_w/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.252691: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key predict/end_pred/fully_connected/weights/ExponentialMovingAverage not found in checkpoint
2017-11-09 11:13:32.252736: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key process_attention/layer_0/dense/kernel/ExponentialMovingAverage not found in checkpoint
Traceback (most recent call last):
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call
    return fn(*args)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1306, in _run_fn
    status, run_metadata)
  File "/home/administrator/anaconda3/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Key apply_attention/bias/ExponentialMovingAverage not found in checkpoint
         [[Node: save_1/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save_1/Const_0_0, save_1/RestoreV2/tensor_names, save_1/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "docqa/eval/squad_eval.py", line 115, in <module>
    main()
  File "docqa/eval/squad_eval.py", line 88, in main
    corpus.get_resource_loader(), checkpoint, args.ema)[args.corpus]
  File "/home/administrator/document-qa/docqa/trainer.py", line 670, in test
    saver.restore(sess, checkpoint)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key apply_attention/bias/ExponentialMovingAverage not found in checkpoint
         [[Node: save_1/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save_1/Const_0_0, save_1/RestoreV2/tensor_names, save_1/RestoreV2/shape_and_slices)]]

Caused by op 'save_1/RestoreV2', defined at:
  File "docqa/eval/squad_eval.py", line 115, in <module>
    main()
  File "docqa/eval/squad_eval.py", line 88, in main
    corpus.get_resource_loader(), checkpoint, args.ema)[args.corpus]
  File "/home/administrator/document-qa/docqa/trainer.py", line 669, in test
    saver = tf.train.Saver({ema.average_name(x): x for x in tf.trainable_variables()})
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
    self.build()
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1172, in build
    filename=self._filename)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 688, in build
    restore_sequentially, reshape)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
    dtypes=dtypes, name=name)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/administrator/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key apply_attention/bias/ExponentialMovingAverage not found in checkpoint
         [[Node: save_1/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save_1/Const_0_0, save_1/RestoreV2/tensor_names, save_1/RestoreV2/shape_and_slices)]]

[chrisc36]

This occurs because the provided model does not have EMA (exponential moving average) weights, so when the script tried to load them it got an error.

I updated the eval scripts so they detect whether EMA weights exist automatically, so you no longer need to use the "--ema" flag. Note the linked model was trained on TriviaQA unfiltered, so I am unsure how strong it will be on SQuAD. To replicate the evaluation we did you can use:

python docqa/eval/triviaqa_full_document_eval.py --n_processes 6 -c open-dev open-shared-norm-cpu -p out.csv

It should reach 67 F1.

I am unsure how strong it will be on SQuAD, it was not trained on that dataset.

[chrisc36]

The full set of models is now linked to on the README.