v2.0 system for definition extractor

[dykang]

Develop v2.0 system for doc-level user interfaces and doc-level evaluation

(See dykang/definition_extractor_v2.0)

• [x] Model ensemble
  • [x] Development/Testing in the internal NLP repo (+2.0 F1)
  • [x] Deployment to frontend system (not sure to do this or not, because of scalability)
• [ x] Fine-tuning roberta-large with S2ORC (with @kyleclo ) (deprecated due to low performance)
  • [x] Initial development in the internal NLP repo
  • [ ] Scale-up for the entire S2ORC dataset: data split and caching
  • [ ] Deployment to frontend system
• [x] Categorization of term types: (with @andrewhead )
  • [x] Types:
    • [x] Symbols,
    • [x] Protologisms (i.e., new words defined in the paper)
    • [x] Abbreviations
  • [x] Entities for each type:
    • [x] Symbols: nicknames, definitions
    • [x] Protologisms: definitions
    • [x] Abbreviations: expansions
  • [x ] Extend the acronym/nickname detector to non-definitional sentences
• [x] Add document-level features (e.g., term position, document frequency)
  • [x] Added position_ratio for relative position of a term in the entire document
  • [x] Added section_name for the name of section that the term appears
• [x] Add cross-document-level features (e.g., term position, document frequency, cross-distance)
  • [x] Make a dictionary of previously detected terms and symbols
• [ ] within-document term grouping based on their definitions (deprecated)
  • [ ] Add coreference links
  • [ ] Analysis between term occurrences and detected term definitions
• [x] add confidence score
  • [x] Development/Testing in the internal NLP repo
  • [x] Deployment to frontend system
• [x] fix BERT tokenization error

[dykang]

• Split the abbreviation term into an acronym as a term and its expansion as a definition.
• Detect nickname type of symbol's definition using their POS or noun phrase patterns.
  • [comments from @andrewhead] This sounds like a good idea. There's an important tricky part to nickname detection, in that we need to do lightweight modification of the extracted text. A key example is in paper https://arxiv.org/pdf/2002.04138.pdf They refer to "d" as "d features", where the right nickname for is it "number of features" and not "features".
  • Another example is when they refer to 'the "ith" feature', where "i"s nickname should be "index of feature"

[dykang]

From @andrewhead: The potential gold protologisms in 2002.04138

• Integrated Hessians (the best candidate for protologism)
• interaction completeness axiom (though this might just a "nonce word")
• interaction completeness error
• interaction symmetry (not detected by our system yet)

[dykang]

A couple of feedback about nickname extractor from @andrewhead

• There are a few symbols that I expected to see in there---for instance, f, g, and d.
• Run nickname extractor for 1909.13433.

[dykang]

Manually collected set of definitions in 1909.13433 by @andrewhead .

Term	Sentence ID	Body
deep amortized clustering	44304	a neural architecture which learns to cluster datasets efficiently using a few forward passes
DAC	44304	deep amortized clustering
DAC	43915	Deep Amortized Clustering
DAC	44305	implicitly learns what makes a cluster, how to group data points into clusters, and how to count the number of clusters in datasets
MoG	43909	Mixture of Gaussians
Neural Processes	43914	meta-learns from multiple datasets to learn a prior over functions
ST	43782	Set Transformer
Set Transformer	43782	a permutation-invariant set-input neural network that uses self-attention operations as building blocks
MAB	43784	Multihead Attention Block
PMA	43793	Pooling by Multihead Attention
ACT	44365	Adaptive Computation Time
filtering	44368	a forward pass through $f$ that takes a set $X$ and outputs a parameter $\theta$ to describe a cluster along with a membership probability vector $\mask \in [0, 1]^{n_{\ssc{X}}}$ where $n_X$ is the number of elements in $X$
MAF	44418	Masked Autogregressive Flow
VAE	44419	Variational Autoencoder
Learning to cluster	43821	the task of learning a clustering algorithm from data
NCP	43831	Neural Clustering Process
Neural Clustering Process	43831	an amortized clustering method
VBDPM	44161	Variational Bayesian Dirichlet Process Mixture Model
LL	44337	log-likelihood
ARI	44337	adjusted Rand index
NMI	44337	normalized mutual information
NS	44211	Neural Statistician
"DEC	44228	Deep Embedded Clustering
Omniglot dataset	44243	handwritten characters from 50 different alphabets
NMI	44255	normalized mutual information
DAC	44026	an approach to amortized clustering using set-input neural networks

[dykang]

Target papers for general user study

• [x] 1804.08199, 2002.04138, 1909.13433
• [x] 1704.05572, 1609.05143, 1611.01603

Target papers for comparison study

• [ ] ?

[dykang]

Closing this issue as we approach merging #146 to master. The remaining features mentioned here will be merged to v3 system or deprecated due to the lack of usefulness.