WanzhengZhu / GRUEN

GRUEN for Evaluating Linguistic Quality of Generated Text (EMNLP 2020 Findings)
https://arxiv.org/pdf/2010.02498.pdf
MIT License
27 stars 12 forks source link

Focus score #5

Open rhear0yishere opened 1 year ago

rhear0yishere commented 1 year ago

Focus score does not seem to be changing from 0, no matter what the candidate texts are.

It seems like it never runs this: ` score.append(1.0 / (1.0 + math.exp(-sim + 7)))

score.append(1.0 / (1.0 + math.exp(-doc1.similarity(doc2) + 7)))

         `

When trying to run on its own, I get the error: 'SpacySimilarityHook' object has no attribute 'similarity'

WanzhengZhu commented 1 year ago

Maybe it is the package version problem? Can you try using the package version from the requirement.txt?

meng-wenlong commented 1 year ago

The error occurs because the SimilarityHook function returns a SpacySimilarityHook object. It should return a doc object. SpacySimilarityHook needs to be initialized first and then called. I guess the code in compute_sentence_similarity() should be:

nlp = spacy.load('en_core_web_md')

similarity_hook = WMD.SpacySimilarityHook(nlp)

@Language.component("simhook")
def SimilarityHook(doc):
    return similarity_hook(doc)

nlp.add_pipe('simhook', last=True)