fabriziosalmi / UglyFeed

Retrieve, aggregate, filter, evaluate, rewrite and serve RSS feeds using Large Language Models for fun, research and learning purposes
GNU Affero General Public License v3.0
127 stars 5 forks source link

additional evaluation metrics #1

Closed fabriziosalmi closed 6 months ago

fabriziosalmi commented 6 months ago

25 additional metrics to evaluate the quality of the rewritten articles, focusing on factual accuracy, completeness, and coherence:

  1. Precision
  2. Recall
  3. Information Retrieval Precision
  4. Information Retrieval Recall
  5. Pointwise Mutual Information (PMI)
  6. Normalized Pointwise Mutual Information (NPMI)
  7. Token Overlap
  8. Word Precision
  9. Word Recall
  10. Term Frequency (TF)
  11. Inverse Document Frequency (IDF)
  12. Content Density
  13. Syntactic Dependency Accuracy
  14. N-gram Coverage
  15. Phrasal Precision
  16. Phrasal Recall
  17. Named Entity Precision
  18. Named Entity Recall
  19. Keyphrase Precision
  20. Keyphrase Recall
  21. Conceptual Similarity
  22. Paraphrase Detection Score
  23. Discourse Coherence
  24. Lexical Cohesion
  25. Topical Consistency
fabriziosalmi commented 6 months ago

note: this list dora's not contain diversity and frequency metrics included in repo atm

Readability and Complexity Metrics

  1. Syllable Count
  2. Flesch-Kincaid Grade Level
  3. SMOG Index
  4. Readability Ease Score

Lexical and Syntactic Metrics

  1. Named Entity Recognition (NER) Coverage
  2. Dependency Tree Depth
  3. Syntactic Variability
  4. Lexical Density
  5. Passive Voice Percentage
  6. Longest Increasing Subsequence

Statistical Metrics

  1. Jaro-Winkler Distance
  2. Honore’s Statistic
  3. Sichel’s Measure
  4. Brunet’s Measure
  5. Yule’s Characteristic K
  6. MTLD (Measure of Textual Lexical Diversity)
  7. HD-D (Hypergeometric Distribution D)
  8. Variability Index

Information and Density Metrics

  1. Information Density
  2. Referential Density
  3. Cohesive Harmony Index

Noun and Verb Metrics

  1. Concrete Noun Ratio
  2. Abstract Noun Ratio

Punctuation and Function Word Metrics

  1. Punctuation Frequency
  2. Ellipsis Frequency
  3. Conjunction Usage Frequency
  4. Preposition Usage Frequency

Structural Metrics

  1. Subordination Index
  2. Coordination Index
  3. Discourse Marker Frequency

Sentiment and Subjectivity Metrics

  1. Sentiment Consistency
  2. Polarity Score
  3. Subjectivity Intensity
  4. Sentiment Variability

Cohesion and Concreteness Metrics

  1. Cohesion Score
  2. Concreteness Score
  3. Imageability Score
  4. Age of Acquisition Score
  5. Text Familiarity Index

Narrative and Persuasiveness Metrics

  1. Narrative Index
  2. Argument Strength
  3. Persuasiveness Score
  4. Engagement Score

Emotional and Analytical Metrics

  1. Emotional Tone Score
  2. Analytical Thinking Score

These lists group the metrics into categories based on their types and focus areas, making it easier to manage and understand the different aspects of text evaluation.