search

microsoft / rag-experiment-accelerator

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.
https://github.com/microsoft/rag-experiment-accelerator
Other
193 stars 68 forks source link

Bump unstructured from 0.14.8 to 0.15.0 #640

Closed dependabot[bot] closed 3 months ago

dependabot[bot] commented 4 months ago

Bumps unstructured from 0.14.8 to 0.15.0.

Release notes

Sourced from unstructured's releases.

0.15.0

Enhancements

  • Improve text clearing process in email partitioning. Updated the email partitioner to remove both =\n and =\r\n characters during the clearing process. Previously, only =\n characters were removed.
  • Bump unstructured.paddleocr to 2.8.0.1.
  • Refine HTML parser to accommodate block element nested in phrasing. HTML parser no longer raises on a block element (e.g. <p>, <div>) nested inside a phrasing element (e.g. <strong> or <cite>). Instead it breaks the phrasing run (and therefore element) at the block-item start and begins a new phrasing run after the block-item. This is consistent with how the browser determines element boundaries in this situation.
  • Install rewritten HTML parser to fix 12 existing bugs and provide headroom for refinement and growth. A rewritten HTML parser resolves a collection of outstanding bugs with HTML partitioning and provides a firm foundation for further elaborating that important partitioner.
  • CI check for dependency licenses Adds a CI check to ensure dependencies are appropriately licensed.

Features

  • Add support for specifying OCR language to partition_pdf(). Extend language specification capability to PaddleOCR in addition to TesseractOCR. Users can now specify OCR languages for both OCR engines when using partition_pdf().
  • Add AstraDB source connector Adds support for ingesting documents from AstraDB.

Fixes

  • Remedy error on Windows when nltk binaries are downloaded. Work around a quirk in the Windows implementation of tempfile.NamedTemporaryFile where accessing the temporary file by name raises PermissionError.
  • Move Astra embedded_dimension to write config

0.14.10

Enhancements

  • Update unstructured-client dependency Change unstructured-client dependency pin back to greater than min version and updated tests that were failing given the update.
  • .doc files are now supported in the arm64 image.. libreoffice24 is added to the arm64 image, meaning .doc files are now supported. We have follow on work planned to investigate adding .ppt support for arm64 as well.
  • Add table detection metrics: recall, precision and f1
  • Remove unused _with_spans metrics

Features

Fixes

  • Fix counting false negatives and false positives in table structure evaluation
  • Fix Slack CI test Change channel that Slack test is pointing to because previous test bot expired
  • Remove NLTK download Removes nltk.download in favor of downloading from an S3 bucket we host to mitigate CVE-2024-39705

0.14.9

Enhancements

  • Added visualization and OD model result dump for PDF In PDF hi_res strategy the analysis parameter can be used to visualize the result of the OD model and dump the result to a file. Additionally, the visualization of bounding boxes of each layout source is rendered and saved for each page.
  • partition_docx() distinguishes "file not found" from "not a ZIP archive" error. partition_docx() now provides different error messages for "file not found" and "file is not a ZIP archive (and therefore not a DOCX file)". This aids diagnosis since these two conditions generally point in different directions as to the cause and fix.

Features

Fixes

  • Fix a bug where multiple soffice processes could be attempted Add a wait mechanism in convert_office_doc so that the function first checks if another soffice is running already: if yes wait till the other process finishes or till the wait timeout before spawning a subprocess to run soffice

... (truncated)

Changelog

Sourced from unstructured's changelog.

0.15.0

Enhancements

  • Improve text clearing process in email partitioning. Updated the email partitioner to remove both =\n and =\r\n characters during the clearing process. Previously, only =\n characters were removed.
  • Bump unstructured.paddleocr to 2.8.0.1.
  • Refine HTML parser to accommodate block element nested in phrasing. HTML parser no longer raises on a block element (e.g. <p>, <div>) nested inside a phrasing element (e.g. <strong> or <cite>). Instead it breaks the phrasing run (and therefore element) at the block-item start and begins a new phrasing run after the block-item. This is consistent with how the browser determines element boundaries in this situation.
  • Install rewritten HTML parser to fix 12 existing bugs and provide headroom for refinement and growth. A rewritten HTML parser resolves a collection of outstanding bugs with HTML partitioning and provides a firm foundation for further elaborating that important partitioner.
  • CI check for dependency licenses Adds a CI check to ensure dependencies are appropriately licensed.

Features

  • Add support for specifying OCR language to partition_pdf(). Extend language specification capability to PaddleOCR in addition to TesseractOCR. Users can now specify OCR languages for both OCR engines when using partition_pdf().
  • Add AstraDB source connector Adds support for ingesting documents from AstraDB.

Fixes

  • Remedy error on Windows when nltk binaries are downloaded. Work around a quirk in the Windows implementation of tempfile.NamedTemporaryFile where accessing the temporary file by name raises PermissionError.
  • Move Astra embedded_dimension to write config

0.14.10

Enhancements

  • Update unstructured-client dependency Change unstructured-client dependency pin back to greater than min version and updated tests that were failing given the update.
  • .doc files are now supported in the arm64 image.. libreoffice24 is added to the arm64 image, meaning .doc files are now supported. We have follow on work planned to investigate adding .ppt support for arm64 as well.
  • Add table detection metrics: recall, precision and f1.
  • Remove unused _with_spans metrics.

Features

Add Object Detection Metrics to CI Add object detection metrics (average precision, precision, recall and f1-score) implementations.

Fixes

  • Fix counting false negatives and false positives in table structure evaluation.
  • Fix Slack CI test Change channel that Slack test is pointing to because previous test bot expired
  • Remove NLTK download Removes nltk.download in favor of downloading from an S3 bucket we host to mitigate CVE-2024-39705

0.14.9

Enhancements

  • Added visualization and OD model result dump for PDF In PDF hi_res strategy the analysis parameter can be used to visualize the result of the OD model and dump the result to a file. Additionally, the visualization of bounding boxes of each layout source is rendered and saved for each page.
  • partition_docx() distinguishes "file not found" from "not a ZIP archive" error. partition_docx() now provides different error messages for "file not found" and "file is not a ZIP archive (and therefore not a DOCX file)". This aids diagnosis since these two conditions generally point in different directions as to the cause and fix.

Features

Fixes

... (truncated)

Commits
  • ec59abf enhancement: improve text clearing process in email partitioning (#3422)
  • 1df7908 feat: save file id for all fsspec connectors if present (#3405)
  • 0eb461a refactor: restructure PDF/Image example document organization (#3410)
  • 5d38703 bugfix: google drive connector metadata safegaurds (#3407)
  • e99e5a8 rfctr(file): make FileType enum a file-type descriptor (#3411)
  • 35ee6bf bugfix: conform all connectors to be added to registry (#3408)
  • a5c9a36 rfctr(file): improve file-type auto-detect (#3409)
  • 48bdf94 feat: partition_pdf() support language specification for PaddleOCR (#3400)
  • 6b1d5f2 rfctr: move astra arg (#3383)
  • 56ca39c rfctr(file): improve filetype tests (#3402)
  • Additional commits viewable in compare view


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependabot[bot] commented 3 months ago

Superseded by #657.