KeyError & groudtruth and input sentence mismatching on custom data

[sallyqus]

Hi thanks for great work. I am trying to use it for my custom data, for a video grounding project.

1. Everything went well with sample data. But when I ran it on my own data, it raised this error:

Traceback (most recent call last):
  File "generate_paraphrases_sow_reap.py", line 284, in <module>
    main(args)
  File "generate_paraphrases_sow_reap.py", line 177, in main
    if n2_size == 1 and sent.tokens[n2.start_idx].pos in TAGS_TO_IGNORE + list(string.punctuation):
KeyError: 12

I am sure that I have done the tokenization and parsing. Any hint about this bug?

2. When I checked the output file, great mismatching between the input sentence and groud-truth sentence. It happened everywhere. Not a single case. Any clue about it?

Input Sentence: person walks back to the doorway . 
Ground Truth Sentence: the person is eating by the refrigerator . 
Reordered Sentence: to the doorway , he walks back . , person . 
len: 2
Rule Sequence: 
('person walks back to the doorway .=>X Y .======>Y , X .', -2.1061782836914062)
('walks back to the doorway=>X back Y======>Y , he X back .', -3.08864164352417)
5 3 4 0 1 2 6
Rule Scores: -5.194820 
Generated Sentence: at the door , he walks back . 
Generation Score: -5.482741 
Generated Sentence: to the door , he comes back . 
Generation Score: -5.852587 
Generated Sentence: to the entrance , he comes back . 
Generation Score: -5.996949 
Generated Sentence: at the door , he comes back . 
Generation Score: -6.175200 
Generated Sentence: to the door , he 'll walk back . 
Generation Score: -6.269359 
Generated Sentence: at the door , he comes back to the person . 
Generation Score: -6.557562 
Generated Sentence: at the door , he comes back to the man . 
Generation Score: -6.593718 
Generated Sentence: to the entrance , he comes back to the person . 
Generation Score: -6.670309 
Generated Sentence: at the entrance , he comes back to the person . 
Generation Score: -6.675691 
Generated Sentence: to the door , he comes back to the person . 
Generation Score: -6.723729 
Ground Truth nll: 6.338918 

Could you help with these two problems? Any help is greatly appreciated.

Best, Sally

[tagoyal]

Hmm, that's unexpected. Can you send me a sample for your dataset (just a couple of examples) so that I can test it on my end?

[sallyqus]

Hi thanks for quick response. Yeah it is a bit weird.

For sure, here are some example sentences without parsing and tokenization:

a person is in a kitchen holding some dishes.
a person walks to a doorway.
a person is fixing a light.
person they open another door.
person turns on the light.
person turns a light on above the door.
a person replaces a lightbulb above a doorway.
a person is throwing clothes into a laundry basket.
person throws clothes in bucket.
person walks through the doorway.

FYI, I used exactly the same command to run and same version of tokenizer and parser.

Can this model be used without groundtruth? Can I set every second sentence (groundtruth) in the pair to be empty? Or it can work if I set the groundtruth as the same input sentence? Any suggestions?

[tagoyal]

The generate_paraphrases_sow_reap.py expects an input file with an even number of sentences (input sent and ground truth). My guess is that your input file has an odd number of sentences, and the error occurs as it tries to parse the last empty line.

The model can be used without the ground truth. But instead of setting every second sentence in the pair to be empty, set the ground truth to be the same as the input sentence.

[sallyqus]

Hi Tanya. Thank you for getting back here.

What I was doing, before I raised this issue and talked with you two days ago, was the use of same input as groundtruth. But the quality is not very good.

Will this operation (the use of same input sentence as groundtruth) induce any impact on the output?

[tagoyal]

The use of the same input as groundtruth does not impact the output, the ground truth isn't used during the generation process at all.

In the training data used to train the SOW-REAP model, we filtered the data to only retain sentences above 8 words. Most of the examples I see here are shorter, so there is a natural mismatch between the training data and your test data.

For the length of sentences we were considering, we decided to incorporate upto 3 syntax reorderings of the subtree within each sentence level reordering. For shorter sentences, I would suggest that you reduce this number to 1 or 2, as those work better for shorter sentences.

[sallyqus]

Thank you for your advice. Will take it. 👍