Qualitative Analysis Issues
• 0 Relevance on Comments: If your qualitative analysis returned 0 relevance on comments that previously showed higher relevance with GPT-3.5, it suggests a potential flaw in how relevance is being calculated or interpreted.
• Possible Causes:
• Changes in Scoring Logic: The method used to assess relevance might have changed or become too restrictive.
• Sample Size or Input Quality: The difference in performance might also stem from the samples being used or the quality of the data.
• Algorithmic Differences: The underlying algorithm in v1 might differ from the one used with GPT-3.5, leading to a disparity in results.
• Suggested Tweaks:
• Revisit Scoring Criteria: Ensure that the criteria for relevance are clearly defined and aligned with your goals. Consider revising the thresholds or weights applied to different factors in the relevance score.
• Cross-Check with Prior Data: Run the same samples through GPT-3.5 and compare the outputs to identify specific changes or regressions in relevance detection.
• Introduce a Feedback Loop: Allow for manual intervention or feedback to adjust relevance scoring when the automated system fails.
Quantitative Analysis Issues
• Problems with Quantitative Analysis: If there are discrepancies or errors in the quantitative analysis, it could affect the reliability of your results.
• Possible Causes:
• Data Integrity: Issues might arise from incorrect or incomplete data being fed into the system.
• Calculation Errors: Ensure that the formulas or algorithms used for quantitative analysis are correctly implemented.
• Inconsistent Metrics: Ensure that the metrics you’re measuring are consistent and align with your objectives.
• Suggested Tweaks:
• Validate Data Sources: Double-check the data sources for completeness and accuracy.
• Audit Calculations: Conduct a thorough audit of the calculations involved in the quantitative analysis to spot any errors.
• Benchmark Against Known Values: Compare your results with known values or established benchmarks to identify inconsistencies.
Image Credit
• Image Credit Issues: If you’re not receiving proper image credit, this could be due to metadata not being properly attached or recognized by the system.
• Suggested Fixes:
• Ensure Metadata is Attached: Verify that the images you’re using have the correct metadata, including credit information.
• Check System Compatibility: Ensure that the platforms or tools you’re using recognize and display the metadata correctly.
• Manually Validate Credits: Manually check a few cases where image credit should be applied to confirm if the issue is widespread or isolated.
Next Steps
1. Review Implementation: Start by reviewing the implementation of both the qualitative and quantitative analyses to identify specific issues.
2. Test with Known Samples: Run known samples through the system to see if you can replicate the issue, making it easier to pinpoint the problem.
3. Iterate on Feedback: Based on the findings, refine the qualitative analysis logic, audit the quantitative methods, and ensure image credits are properly attributed.
Quantitative Analysis Issues
Image Credit
Next Steps
Originally posted by @Abuchtela in https://github.com/ubiquibot/conversation-rewards/issues/97#issuecomment-2322093454