Closed leobeeson closed 1 year ago
For user-defined POS patterns, I suggest that we allow users to define a custom MWE pattern in the nltk.RegexpParser
format so that we can directly use this NLTK method to extract those patterns.
# e.g. define only noun-noun or adj-noun compounds
mwe_pattern_nnc_bigram = 'NNC: {<JJ><NN.*>}'
mwe_pattern_jnc_bigram = 'JNC: {<JJ><NN.*>}'
# or define a generic <adj* noun+> compound pattern
mwe_pattern_nnc_ngram = 'NC: {<JJ>*<NN.*>+}'
We can predefine and offer a number of these patterns to the users, for standard MWE patterns. E.g. NCs, LVCs, VPCs, etc. so that the user can already select from a bunch of existing patterns.
Then we continue with NLTK to extract those patterns from sentences:
chunk_parser = nltk.RegexpParser(mwe_pattern_nnc_bigram)
sentence = "An example sentence"
tagged_sentence = nltk.pos_tag(nltk.word_tokenize(sentence))
parsed_sentence = chunk_parser.parse(tagged_sentence)
NNCs = []
for subtree in parsed_sentence.subtrees(filter=lambda t: t.label() == 'NNC'):
NNCs.append(' '.join(word for word, tag in subtree.leaves()))
Sounds good. I'll check out the nltk.RegexpParser
and focus on it, and chose an initial set of standard MWE patterns. Thanks.
Situation
mwe_utils.extract_mwes_from_sent()
only generates bigram candidates for collocations of the following sequences:[["NN", "NNS"], ["NN", "NNS"]]
[["JJ"], ["NN", "NNS"]]
New York State
, we would get the following results:mwe_type == "NC"
you captureYork State
(which is a false positive).mwe_type == "JNC"
you captureNew York
(which is also a false positive).Solution
mwe_utils.extract_mwes_from_sent()
to extract:mwe_utils.generate_candidate_mwe()
mwe_utils.generate_candidate_collocations()
User-defined POS Pattern for Syntactical Structure
mwe_type
parameter or generate an additional optional parameter in themwe_utils.extract_mwes_from_sent()
method to be of typelist[list[str]]
.mwe_type
or the POS-sequence passed to the new parameter of typelist[list[str]]
.NC
is an alias for[["NN", "NNS"], ["NN", "NNS"]]
andJNC
for[["JJ"], ["NN", "NNS"]]
.Variable-Length MWE
.
"[.]?"
would allow any POS tag, zero or one time."[[DT][.]?[NN]]"
would catchthe bus
as well asthe blue bus
.?
,+
,*
,{n}
,{n,}
,{n,m}
[<POS>]{1}
[^]
"[[^DT][.]]"
Objective
"[[.][.][.]]"
Justification
mwe_type
argument options (i.e.NC
andJNC
) significantly restrictswordview
's utility to users (e.g. NLP Engineers).mwe_type
argument options (i.e.NC
andJNC
) significantly restricts real-world, enterprise use cases/problemswordview
can solve.Downstream Work
^
and$
in POS sequences relative to a sentence or user-defined text segment.(?=)
/(?!)
and lookbehinds(?<=)
/(?<!)
in POS sequences.NB
tracer
for tagging syntactical sequences of higher-order concepts (e.g. controlled vocabularies/mappings), but applied strictly to POS tags (i.e. lower-order syntactical concepts/mappings).