Pipeline hanging on partition

I am using unstructured-ingest version 0.3.0, using the following code:


from unstructured_ingest.v2.interfaces import ProcessorConfig
from unstructured_ingest.v2.pipeline.pipeline import Pipeline
from unstructured_ingest.v2.processes.chunker import ChunkerConfig
from unstructured_ingest.v2.processes.connectors.local import (
    LocalConnectionConfig,
    LocalDownloaderConfig,
    LocalIndexerConfig,
    LocalUploaderConfig,
)
from unstructured_ingest.v2.processes.partitioner import PartitionerConfig

def ingest(work_dir: Path, storage_dir: Path, partition_dir: Path):
    """ingest data from different file types using unstructured-ingest"""
    num_processes = 8
    process_config = ProcessorConfig(
        reprocess=True,
        verbose=True,
        num_processes=num_processes,
        work_dir=str(work_dir.resolve()),
        iter_delete=True,
    )
    indexer_config = LocalIndexerConfig(input_path=str(storage_dir.resolve()), recursive=True)
    downloader_config = LocalDownloaderConfig()
    source_connection_config = LocalConnectionConfig()
    partitioner_config = PartitionerConfig(
        strategy="auto",
        hi_res_model_name="yolox",
        fields_include=["element_id", "text", "type"],
        additional_partition_args=dict(
            include_page_breaks=True, analysis=True, process_attachments=False
        ),
    )
    uploader_config = LocalUploaderConfig(output_dir=str(partition_dir.resolve()))
    chunker_config = ChunkerConfig(
        chunking_strategy="by_title",
        chunk_combine_text_under_n_chars=500,
        chunk_max_characters=1500,
        chunk_new_after_n_chars=1000,
        chunk_overlap=100,
    )

    Pipeline.from_configs(
        context=process_config,
        indexer_config=indexer_config,
        downloader_config=downloader_config,
        source_connection_config=source_connection_config,
        partitioner_config=partitioner_config,
        chunker_config=chunker_config,
        uploader_config=uploader_config,
    ).run()

I am running this code in a docker container, I also tested it locally but I always get into the same issue. The pipeline never finishes and never enter the chunking phase. I am processing about 43K documents of different types (html, xml, pdf, docx, json ...) since JSON is no supported they just don't get processed. The pipeline hangs on the partition phase and never enter chunking.

Unstructured-IO / unstructured-ingest

Pipeline hanging on partition #246