How do I use only Reformulator with checkpoint of the reformulator?

[bhoomit]

How do I use only Reformulator with checkpoint of the reformulator https://storage.cloud.google.com/pretrained_models/translate.ckpt-6156696.zip

[rodrigonogueira4]

1. Extract the .zip file to $PRETRAINED

2. Write the checkpoint path in the files $REFORMULATOR_DIR/checkpoint and $REFORMULATOR_DIR/initial_checkpoint.txt, like this: echo "model_checkpoint_path: \"$PRETRAINED_DIR/translate.ckpt-6156696\"" > checkpoint cp -f checkpoint $REFORMULATOR_DIR cp -f checkpoint $REFORMULATOR_DIR/initial_checkpoint.txt

3. Start Training: python -m px.nmt.reformulator_and_selector_training \ --environment_server_address=localhost:10000 \ --hparams_path=px/nmt/example_configs/reformulator.json \ --enable_reformulator_training=true \ --enable_selector_training=false \ --train_questions=$SQUAD_DIR/train-questions.txt \ --train_annotations=$SQUAD_DIR/train-annotation.txt \ --train_data=data/squad/data_train.json \ --dev_questions=$SQUAD_DIR/dev-questions.txt \ --dev_annotations=$SQUAD_DIR/dev-annotation.txt \ --dev_data=data/squad/data_dev.json \ --glove_path=$GLOVE_DIR/glove.6B.100d.txt \ --out_dir=$REFORMULATOR_DIR \ --tensorboard_dir=$OUT_DIR/tensorboard

[bhoomit]

Thanks @rodrigonogueira4 for the response.

But I don't want to train, just want to do inference.

[rodrigonogueira4]

In this case, type on the terminal:

echo "model_checkpoint_path: "$PRETRAINED_DIR/translate.ckpt-6156696"" > checkpoint cp -f checkpoint $REFORMULATOR_DIR cp -f checkpoint $REFORMULATOR_DIR/initial_checkpoint.txt

And use the following code (you only need to replace 'path/to/reformulator_dir' with your $REFORMULATOR_DIR)

from px.nmt import reformulator
from px.proto import reformulator_pb2

questions = ['question 1', 'question 2']

reformulator_instance = reformulator.Reformulator(
    hparams_path='px/nmt/example_configs/reformulator.json',
    source_prefix='<en> <2en> ',
    out_dir='path/to/reformulator_dir',
    environment_server_address='localhost:10000')

# Change from GREEDY to BEAM if you want 20 rewrites instead of one.
responses = reformulator_instance.reformulate(
    questions=questions,
    inference_mode=reformulator_pb2.ReformulatorRequest.GREEDY)

# Since we are using greedy decoder, keep only the first rewrite.
reformulations = [r[0].reformulation for r in responses]

print reformulations

[bhoomit]

This is great. One more question: environment_server_address is not required for inference, right?

[rodrigonogueira4]

That is right, you can use a dummy address or environment_server_address=None

[graviraja]

The reformulator with the checkpoint 1460356 is giving better results than with the checkpoint 6156696. Why is this happening?

For questions : ['how can i apply for nsa?', 'what is the minimum working hours required for a day?']

The 1460356 ckpt model is giving the following results.

Where as the 6156696 ckpt model is giving

[rodrigonogueira4]

The model trained on the machine translation task (checkpoint 1460356) produces sentences that are more grammatically correct, but the model trained with RL on the Q&A task (checkpoint 6156696) produces sentences that have a better F1 score despite not being grammatically correct.

[bvnagaraju]

where can I download checkpoint 1460356 ?

[rodrigonogueira4]

The link is in the README: https://storage.googleapis.com/pretrained_models/translate.ckpt-1460356.zip

[bvnagaraju]

Is anyone had the below error while just doing inferences? from px.proto import aqa_pb2 ImportError: cannot import name 'aqa_pb2

[bvnagaraju]

The model trained on the machine translation task (checkpoint 1460356) produces sentences that are more grammatically correct, but the model trained with RL on the Q&A task (checkpoint 6156696) produces sentences that have a better F1 score despite not being grammatically correct.

hey how did you get this working? I am seeing below error ImportError: cannot import name 'aqa_pb2

[graviraja]

The model trained on the machine translation task (checkpoint 1460356) produces sentences that are more grammatically correct, but the model trained with RL on the Q&A task (checkpoint 6156696) produces sentences that have a better F1 score despite not being grammatically correct.

hey how did you get this working? I am seeing below error ImportError: cannot import name 'aqa_pb2

I am using python2 and in compile_protos.sh i modified the python to python2 and the versions are the following : grpc==0.3.post19 grpcio==1.16.1 grpcio-tools==1.16.1 protobuf==3.6.1

[graviraja]

The model trained on the machine translation task (checkpoint 1460356) produces sentences that are more grammatically correct, but the model trained with RL on the Q&A task (checkpoint 6156696) produces sentences that have a better F1 score despite not being grammatically correct.

I have ran the selector model on dev data and the reformulations, scores produced by the model with checkpoint 1460356 are better than 6156696.

Reformulations, scores for 1460356 ckpt model.

Reforumlations, scores for the 6156696 ckpt model.

[JohannesTK]

Does the gRPC server have to run? Getting a grpc.FutureTimeoutError:

python2 reformulate.py 
Num encoder layer 2 is different from num decoder layer 4, so set pass_hidden_state to False
# hparams:
  src=source
  tgt=target
  train_prefix=None
  dev_prefix=None
  test_prefix=None
  train_annotations=None
  dev_annotations=None
  test_annotations=None
  out_dir=/tmp/active-qa/reformulator
# Vocab file data/spm2/spm.unigram.16k.vocab.nocount.notab.source exists
  using source vocab for target
# Use the same embedding for source and target
Traceback (most recent call last):
  File "reformulate.py", line 10, in <module>
    environment_server_address='localhost:10000')
  File "/root/active-qa/px/nmt/reformulator.py", line 130, in __init__
    use_placeholders=True)
  File "/root/active-qa/px/nmt/model_helper.py", line 171, in create_train_model
    trie=trie)
  File "/root/active-qa/px/nmt/gnmt_model.py", line 56, in __init__
    trie=trie)
  File "/root/active-qa/px/nmt/attention_model.py", line 65, in __init__
    trie=trie)
  File "/root/active-qa/px/nmt/model.py", line 137, in __init__
    hparams.environment_server, mode=hparams.environment_mode))
  File "/root/active-qa/px/nmt/environment_client.py", line 152, in make_environment_reward_fn
    grpc.channel_ready_future(channel).result(timeout=30)
  File "/root/active-qa/venv/local/lib/python2.7/site-packages/grpc/_utilities.py", line 134, in result
    self._block(timeout)
  File "/root/active-qa/venv/local/lib/python2.7/site-packages/grpc/_utilities.py", line 84, in _block
    raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError

[kaustumbh7]

The reformulator with the checkpoint 1460356 is giving better results than with the checkpoint 6156696. Why is this happening?

For questions : ['how can i apply for nsa?', 'what is the minimum working hours required for a day?']

The 1460356 ckpt model is giving the following results.

Where as the 6156696 ckpt model is giving

@graviraja Hey! I am trying to do the same thing but I get invalid reformulations like this- My questions were- ['how can i apply for nsa?', 'what is the minimum working hours required for a day?']

[graviraja]

@kaustumbh7 which glove you are using?

[kaustumbh7]

In this case, type on the terminal:

echo "model_checkpoint_path: "$PRETRAINED_DIR/translate.ckpt-6156696"" > checkpoint cp -f checkpoint $REFORMULATOR_DIR cp -f checkpoint $REFORMULATOR_DIR/initial_checkpoint.txt

And use the following code (you only need to replace 'path/to/reformulator_dir' with your $REFORMULATOR_DIR)

from px.nmt import reformulator
from px.proto import reformulator_pb2

questions = ['question 1', 'question 2']

reformulator_instance = reformulator.Reformulator(
    hparams_path='px/nmt/example_configs/reformulator.json',
    source_prefix='<en> <2en> ',
    out_dir='path/to/reformulator_dir',
    environment_server_address='localhost:10000')

# Change from GREEDY to BEAM if you want 20 rewrites instead of one.
responses = reformulator_instance.reformulate(
    questions=questions,
    inference_mode=reformulator_pb2.ReformulatorRequest.GREEDY)

# Since we are using greedy decoder, keep only the first rewrite.
reformulations = [r[0].reformulation for r in responses]

print reformulations

@graviraja I just followed these instructions. Glove, do you mean for the server ? I simply followed all the steps as given in active-qa readme.

[graviraja]

@kaustumbh7 yes glove i mean for the server. Then in that case probably loading of the pretrained model is not happening correctly. Can you once cross check whether the checkpoint is loading correctly or not properly.

[kaustumbh7]

@graviraja Glove for the server is the same as mentioned in the readme section.

It does show- loaded train model parameters from data/pretrained/translate.ckpt-1460356, time 0.88s.

Did you run the same code as I did ? https://github.com/google/active-qa/issues/9#issuecomment-432807557

Did you run the code in python2?

Is any pre-processing of questions = ['question 1', 'question 2'] required?

[graviraja]

@kaustumbh7 I have run the reformulator_and_selector_training.py code by keeping the --enable_reformulator_training=false, and modified a bit like adding the questions = ['question 1', 'question 2'] in the code. Yes, I ran the code with python 2.

[kaustumbh7]

@graviraja Okay. I will try that out. Thanks for your help! 😊

[konrajak]

@kaustumbh7 I am running into the same problem, only getting completely nonsensical reformulations. Have you managed to solve the issue?

[kaustumbh7]

@konrajak Not yet. But I am working on it.

[kaustumbh7]

@kaustumbh7 I have run the reformulator_and_selector_training.py code by keeping the --enable_reformulator_training=false, and modified a bit like adding the questions = ['question 1', 'question 2'] in the code. Yes, I ran the code with python 2.

@graviraja I tried this as well but still got the same result.

Did you download the selector weights as well?

Did you give --enable_selector_training=false in reformulator_and_selector_training.py?

Can you please share your modified reformulator_and_selector_training.py file?

[graviraja]

Hi @kaustumbh7

Below are the steps I have done:

• After the pre-processing step, run the environment
• In the training reformulator code in readme, i have done the copying the checkpoint steps.
• I have run the same command in the training reformulator section, only change is keeping the argument --enable_reformulator_training=false , --enable_selector_training=false

Code changes:

• The code is same till the read data part in main method.

  # Read data.
  questions, annotations, docid_2_answer = read_data(
    questions_file=FLAGS.train_questions,
    annotations_file=FLAGS.train_annotations,
    answers_file=FLAGS.train_data,
    preprocessing_mode=FLAGS.mode)
  dev_questions, dev_annotations, dev_docid_2_answer = read_data(
    questions_file=FLAGS.dev_questions,
    annotations_file=FLAGS.dev_annotations,
    answers_file=FLAGS.dev_data,
    preprocessing_mode=FLAGS.mode,
    max_lines=FLAGS.max_dev_examples)

• After that, I have done the following:

  custom_questions = ['How can i apply for nsa?']
  responses = reformulator_instance.reformulate(
    questions=custom_questions,
    inference_mode=reformulator_pb2.ReformulatorRequest.GREEDY)
  
  # Discard answers.
  custom_reformulations = [[rf.reformulation for rf in rsp] for rsp in responses]

• And commented all the later code.

Hope this helps!

[kaustumbh7]

Hi @kaustumbh7

Below are the steps I have done:

• After the pre-processing step, run the environment
• In the training reformulator code in readme, i have done the copying the checkpoint steps.
• I have run the same command in the training reformulator section, only change is keeping the argument --enable_reformulator_training=false , --enable_selector_training=false

Code changes:

• The code is same till the read data part in main method.

  # Read data.
  questions, annotations, docid_2_answer = read_data(
      questions_file=FLAGS.train_questions,
      annotations_file=FLAGS.train_annotations,
      answers_file=FLAGS.train_data,
      preprocessing_mode=FLAGS.mode)
  dev_questions, dev_annotations, dev_docid_2_answer = read_data(
      questions_file=FLAGS.dev_questions,
      annotations_file=FLAGS.dev_annotations,
      answers_file=FLAGS.dev_data,
      preprocessing_mode=FLAGS.mode,
      max_lines=FLAGS.max_dev_examples)

• After that, I have done the following:

  custom_questions = ['How can i apply for nsa?']
  responses = reformulator_instance.reformulate(
      questions=custom_questions,
      inference_mode=reformulator_pb2.ReformulatorRequest.GREEDY)

  # Discard answers.
  custom_reformulations = [[rf.reformulation for rf in rsp] for rsp in responses]

• And commented all the later code.

Hope this helps!

@graviraja It didn't work. I still get nonsensical reformulations. I don't understand what the problem is. Thank you very much for your help.

[graviraja]

@kaustumbh7 which pretrained model of reformulator you are using? If it is 6156696-ckpt, the reformulations are not good.

[kaustumbh7]

@graviraja I am using 1460356 provided here-https://storage.googleapis.com/pretrained_models/translate.ckpt-1460356.zip

[graviraja]

@kaustumbh7 which tensorflow version you are using?

[kaustumbh7]

@graviraja My tensorflow version is 1.13.1.

[graviraja]

@kaustumbh7 Use tensorflow version 1.12.2 and if there is tensorflow-estimator installed, uninstall it.

[kaustumbh7]

@graviraja Thank you very much! It worked! 👍🏼😁

[Pem14604]

from px.nmt import reformulator from px.proto import reformulator_pb2

Hi when i try to import this it says from px.proto import aqa_pb2 ImportError: cannot import name 'aqa_pb2' from 'px.proto' (C:\Anaconda\lib\site-packages\px\proto__init__.py).

As there is no .py file in proto folfer, please help.