search

microsoft / rag-experiment-accelerator

The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.
https://github.com/microsoft/rag-experiment-accelerator
Other
193 stars 68 forks source link

Bump unstructured from 0.14.8 to 0.15.1 #657

Closed dependabot[bot] closed 3 months ago

dependabot[bot] commented 3 months ago

Bumps unstructured from 0.14.8 to 0.15.1.

Release notes

Sourced from unstructured's releases.

0.15.1

Enhancements

  • Improve pdfminer embedded image extraction to exclude text elements and produce more accurate bounding boxes. This results in cleaner, more precise element extraction in pdf partitioning.

Features

  • Update partition_eml and partition_msg to capture cc, bcc, and message_id fields Cc, bcc, and message_id information is captured in element metadata for both msg and email partitioning and Recipient elements are generated for cc and bcc when include_headers=True for email partitioning.
  • Mark ingest as deprecated Begin sunset of ingest code in this repo as it's been moved to a dedicated repo.
  • Add pdf_hi_res_max_pages argument for partitioning, which allows rejecting PDF files that exceed this page number limit, when the high_res strategy is chosen. By default, it will allow parsing PDF files with an unlimited number of pages.

Fixes

  • Update HuggingFaceEmbeddingEncoder to use HuggingFaceEmbeddings from langchain_huggingface package instead of the deprecated version from langchain-community. This resolves the deprecation warning and ensures compatibility with future versions of langchain.
  • Update OpenAIEmbeddingEncoder to use OpenAIEmbeddings from langchain-openai package instead of the deprecated version from langchain-community. This resolves the deprecation warning and ensures compatibility with future versions of langchain.
  • Update import of Pinecone exception Adds compatibility for pinecone-client>=5.0.0
  • File-type detection catches non-existent file-path. detect_filetype() no longer silently falls back to detecting a file-type based on the extension when no file exists at the path provided. Instead FileNotFoundError is raised. This provides consistent user notification of a mis-typed path rather than an unpredictable exception from a file-type specific partitioner when the file cannot be opened.
  • EML files specified as a file-path are detected correctly. Resolved a bug where an EML file submitted to partition() as a file-path was identified as TXT and partitioned using partition_text(). EML files specified by path are now identified and processed correctly, including processing any attachments.
  • A DOCX, PPTX, or XLSX file specified by path and ambiguously identified as MIME-type "application/octet-stream" is identified correctly. Resolves a shortcoming where a file specified by path immediately fell back to filename-extension based identification when misidentified as "application/octet-stream", either by asserted content type or a mis-guess by libmagic. An MS Office file misidentified in this way is now correctly identified regardless of its filename and whether it is specified by path or file-like object.
  • Textual content retrieved from a URL with gzip transport compression now partitions correctly. Resolves a bug where a textual file-type (such as Markdown) retrieved by passing a URL to partition() would raise when gzip compression was used for transport by the server.
  • A DOCX, PPTX, or XLSX content-type asserted on partition is confirmed or fixed. Resolves a bug where calling partition() with a swapped MS-Office content_type would cause the file-type to be misidentified. A DOCX, PPTX, or XLSX MIME-type received by partition() is now checked for accuracy and corrected if the file is for a different MS-Office 2007+ type.
  • DOC, PPT, XLS, and MSG files are now auto-detected correctly. Resolves a bug where DOC, PPT, and XLS files were auto-detected as MSG files under certain circumstances.

0.15.0

Enhancements

  • Improve text clearing process in email partitioning. Updated the email partitioner to remove both =\n and =\r\n characters during the clearing process. Previously, only =\n characters were removed.
  • Bump unstructured.paddleocr to 2.8.0.1.
  • Refine HTML parser to accommodate block element nested in phrasing. HTML parser no longer raises on a block element (e.g. <p>, <div>) nested inside a phrasing element (e.g. <strong> or <cite>). Instead it breaks the phrasing run (and therefore element) at the block-item start and begins a new phrasing run after the block-item. This is consistent with how the browser determines element boundaries in this situation.
  • Install rewritten HTML parser to fix 12 existing bugs and provide headroom for refinement and growth. A rewritten HTML parser resolves a collection of outstanding bugs with HTML partitioning and provides a firm foundation for further elaborating that important partitioner.
  • CI check for dependency licenses Adds a CI check to ensure dependencies are appropriately licensed.

Features

  • Add support for specifying OCR language to partition_pdf(). Extend language specification capability to PaddleOCR in addition to TesseractOCR. Users can now specify OCR languages for both OCR engines when using partition_pdf().
  • Add AstraDB source connector Adds support for ingesting documents from AstraDB.

Fixes

  • Remedy error on Windows when nltk binaries are downloaded. Work around a quirk in the Windows implementation of tempfile.NamedTemporaryFile where accessing the temporary file by name raises PermissionError.
  • Move Astra embedded_dimension to write config

0.14.10

Enhancements

  • Update unstructured-client dependency Change unstructured-client dependency pin back to greater than min version and updated tests that were failing given the update.

... (truncated)

Changelog

Sourced from unstructured's changelog.

0.15.1

Enhancements

  • Improve pdfminer embedded image extraction to exclude text elements and produce more accurate bounding boxes. This results in cleaner, more precise element extraction in pdf partitioning.

Features

  • Update partition_eml and partition_msg to capture cc, bcc, and message_id fields Cc, bcc, and message_id information is captured in element metadata for both msg and email partitioning and Recipient elements are generated for cc and bcc when include_headers=True for email partitioning.
  • Mark ingest as deprecated Begin sunset of ingest code in this repo as it's been moved to a dedicated repo.
  • Add pdf_hi_res_max_pages argument for partitioning, which allows rejecting PDF files that exceed this page number limit, when the high_res strategy is chosen. By default, it will allow parsing PDF files with an unlimited number of pages.

Fixes

  • Update HuggingFaceEmbeddingEncoder to use HuggingFaceEmbeddings from langchain_huggingface package instead of the deprecated version from langchain-community. This resolves the deprecation warning and ensures compatibility with future versions of langchain.
  • Update OpenAIEmbeddingEncoder to use OpenAIEmbeddings from langchain-openai package instead of the deprecated version from langchain-community. This resolves the deprecation warning and ensures compatibility with future versions of langchain.
  • Update import of Pinecone exception Adds compatibility for pinecone-client>=5.0.0
  • File-type detection catches non-existent file-path. detect_filetype() no longer silently falls back to detecting a file-type based on the extension when no file exists at the path provided. Instead FileNotFoundError is raised. This provides consistent user notification of a mis-typed path rather than an unpredictable exception from a file-type specific partitioner when the file cannot be opened.
  • EML files specified as a file-path are detected correctly. Resolved a bug where an EML file submitted to partition() as a file-path was identified as TXT and partitioned using partition_text(). EML files specified by path are now identified and processed correctly, including processing any attachments.
  • A DOCX, PPTX, or XLSX file specified by path and ambiguously identified as MIME-type "application/octet-stream" is identified correctly. Resolves a shortcoming where a file specified by path immediately fell back to filename-extension based identification when misidentified as "application/octet-stream", either by asserted content type or a mis-guess by libmagic. An MS Office file misidentified in this way is now correctly identified regardless of its filename and whether it is specified by path or file-like object.
  • Textual content retrieved from a URL with gzip transport compression now partitions correctly. Resolves a bug where a textual file-type (such as Markdown) retrieved by passing a URL to partition() would raise when gzip compression was used for transport by the server.
  • A DOCX, PPTX, or XLSX content-type asserted on partition is confirmed or fixed. Resolves a bug where calling partition() with a swapped MS-Office content_type would cause the file-type to be misidentified. A DOCX, PPTX, or XLSX MIME-type received by partition() is now checked for accuracy and corrected if the file is for a different MS-Office 2007+ type.
  • DOC, PPT, XLS, and MSG files are now auto-detected correctly. Resolves a bug where DOC, PPT, and XLS files were auto-detected as MSG files under certain circumstances.

0.15.0

Enhancements

  • Improve text clearing process in email partitioning. Updated the email partitioner to remove both =\n and =\r\n characters during the clearing process. Previously, only =\n characters were removed.
  • Bump unstructured.paddleocr to 2.8.0.1.
  • Refine HTML parser to accommodate block element nested in phrasing. HTML parser no longer raises on a block element (e.g. <p>, <div>) nested inside a phrasing element (e.g. <strong> or <cite>). Instead it breaks the phrasing run (and therefore element) at the block-item start and begins a new phrasing run after the block-item. This is consistent with how the browser determines element boundaries in this situation.
  • Install rewritten HTML parser to fix 12 existing bugs and provide headroom for refinement and growth. A rewritten HTML parser resolves a collection of outstanding bugs with HTML partitioning and provides a firm foundation for further elaborating that important partitioner.
  • CI check for dependency licenses Adds a CI check to ensure dependencies are appropriately licensed.

Features

  • Add support for specifying OCR language to partition_pdf(). Extend language specification capability to PaddleOCR in addition to TesseractOCR. Users can now specify OCR languages for both OCR engines when using partition_pdf().
  • Add AstraDB source connector Adds support for ingesting documents from AstraDB.

Fixes

  • Remedy error on Windows when nltk binaries are downloaded. Work around a quirk in the Windows implementation of tempfile.NamedTemporaryFile where accessing the temporary file by name raises PermissionError.
  • Move Astra embedded_dimension to write config

0.14.10

Enhancements

  • Update unstructured-client dependency Change unstructured-client dependency pin back to greater than min version and updated tests that were failing given the update.
  • .doc files are now supported in the arm64 image.. libreoffice24 is added to the arm64 image, meaning .doc files are now supported. We have follow on work planned to investigate adding .ppt support for arm64 as well.

... (truncated)

Commits
  • 7e88744 chore: Cut the 0.15.1 release (#3481)
  • b749b89 fix: disabled checking max pages for images (#3473)
  • 147514f feat: msg and email metadata (#3444)
  • 0f05718 Improve pdfminer embedded image extraction in pdf partitioning (#3456)
  • 8fd216c feat/pdf-page-limit-in-hi-res (#3431)
  • 482f093 feat: Add deprecation warning on import of any ingest code (#3443)
  • 4e61acc fix(file): fix OLE-based file-type auto-detection (#3437)
  • 432d209 fix(file): confirm or correct asserted DOCX, PPTX, and XLSX content types (#3...
  • 560cc0e fix: update HuggingFaceEmbeddingEncoder to use langchain_huggingface instea...
  • 798dcc0 fix: update OpenAIEmbeddingEncoder to use langchain-openai instead of `lang...
  • Additional commits viewable in compare view


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
dependabot[bot] commented 3 months ago

Looks like unstructured is up-to-date now, so this is no longer needed.