search

BU-Spark / ml-herbarium

Herbaria ML
15 stars 12 forks source link

Multiple Labels #105

Open funkyvoong opened 1 year ago

funkyvoong commented 1 year ago
kabilanmohanraj commented 1 year ago

Updates (24th July 2023):

  1. Label segmentation task: 1.1 Reading about docAI in general, and LayoutLM model versions and their implementations. 1.2 Working to label images to fine-tune the LayoutLM model.
kabilanmohanraj commented 1 year ago

Updates (15th August):

  1. Integrated the DETR model into the original CRAFT-TrOCR-TaxonNERD pipeline. Now, DETR-CRAFT-TrOCR-TaxonNERD. Masked the non-label regions instead of cropping the labels.
  2. Resolved Python module dependency issues mentioned during last week's meeting; all models work coherently in one environment (will commit the conda export file).
  3. Upgraded DETR inference pipeline to use DETRImageProcessor instead of DETRFeatureExtractor (this upgrade is necessary as DETRFeatureExtractor is to be removed from transformers==5.0).
  4. Evaluating the performance of the pipeline with and without DETR (for label extraction). 4.1 Having issues with the TaxonNERD step with label extraction => some outputs from this step are shifted one or two indices causing the accuracy to plummet to 4% (72% previously). This is caused by images where no labels are extracted, returning fully masked images (a very subtle issue with the loop construct).

Pending: ~1. Retrain the DETR model with more labeled data. I have labeled only a few images this week.

  1. ReadMe for DETR and Classification model.
  2. Yet to push this week's work to the repo. The codebase has not been cleaned, as I am still debugging the pipeline. Please feel free to take a look at the up to date codebase on SCC.~
  3. Rank labels based on the year in each of them. (Not as straight forward as I thought, will leave notes on attempts)