ygorg
commented
4 years ago [Edit 27/10/2020] Updated merge_compound code to use Doc.retokenize
Hi ! Thanks for highlighting this issue.
This is linked to the spacy tokenisation.
According to How to merge words split by hyphen? and Spacy - Tokenize quoted string and lemma of merged spans only has lemma of first token.
Adding a pipe to the spacy model seem to answer some question. I didn't have a proper document but by using the string you shared, the code I wrote works (but it needs to be tested with real data).
The idea here is to create a spacy model add a pipe and then use the spacy_model parameter of load_document to process the documents. See code below.
The problem this can generate are:
- part of compound can not be a keyphrase (in 'driver-assistance', 'assistance' will never be a candidate (except if it appear by itself))
- the stemming will be applied on the whole token
import pke
import spacy
nlp = spacy.load('en') # or any model
nlp.add_pipe(merge_compounds)
doc = "self-driving, coast-to-coast, peer-to-peer, driver-assistance,web-service"
e = pke.unsupervised.MultipartiteRank()
e.load_document(doc, spacy_model=nlp)
print(e.sentences[0].stems) # compounds should appear
e.candidate_selection()
print(e.candidates) # compounds should now appear
# etc ...
```python
def merge_compounds(d):
""" Merge compounds to be one token
A compound is two tokens separated by a hyphen when the tokens are right next to the hyphen
d (spacy.Doc): Document
Returns: spacy.Doc
> [t.text for t in nlp('peer-to-peer-to-peer')]
['peer', '-', 'to', '-', 'peer', 'to', '-', 'peer']
> [t.text for t in merge_compounds(nlp('peer-to-peer-to-peer'))]
['peer-to-peer-to-peer']
"""
# Returns beginning and end offset of spacy.Token
offsets = lambda t: (t.idx, t.idx+len(t))
# Identify the hyphens
# for each token is it a hyphen and the next and preceding token are right next to the hyphen
spans = [(i-1, i+1) for i in range(len(d))
if i != 0 and i != len(d) and d[i].text == '-' and \
offsets(d[i-1])[1] == offsets(d[i])[0] and \
offsets(d[i+1])[0] == offsets(d[i])[1]
]
# merging spans to account for multi-compound terms
merged = []
for i, (b, e) in enumerate(spans):
# if the last spans ends when the current span begins,
# merge those
if merged and b == merged[-1][1]:
merged[-1] = (merged[-1][0], e)
else:
merged.append((b, e))
# Merge the compounds in the document
with d.retokenize() as retok:
for b, e in merged:
retok.merge(d[b:e+1], attrs={
'POS': d[b].pos_,
'LEMMA': ''.join(t.lemma_ for t in d[b:e+1])
})
return d
```
Benja1972
I have a text with many common abbreviation such as "self-driving, coast-to-coast, peer-to-peer, driver-assistance,web-service" When I run TopicRank and MultipartiteRank, I see that they are not captured. I use a stoplist and pos like this
How to fix this behavior? I would like this terms to be present in output Thank you