Zipstack / unstract

No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents
https://unstract.com
GNU Affero General Public License v3.0
2.44k stars 150 forks source link
etl-pipeline llm-platform unstructured-data
# Unstract ## Intelligent Document Processing 2.0 (IDP 2.0) Platform Powered by Large Language Models #### No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents ## [![pdm-managed](https://img.shields.io/badge/pdm-managed-blueviolet)](https://pdm-project.org) [![CLA assistant](https://cla-assistant.io/readme/badge/Zipstack/unstract)](https://cla-assistant.io/Zipstack/unstract) [![pre-commit.ci status](https://results.pre-commit.ci/badge/github/Zipstack/unstract/main.svg)](https://results.pre-commit.ci/latest/github/Zipstack/unstract/main) [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=Zipstack_unstract&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) [![Bugs](https://sonarcloud.io/api/project_badges/measure?project=Zipstack_unstract&metric=bugs)](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) [![Code Smells](https://sonarcloud.io/api/project_badges/measure?project=Zipstack_unstract&metric=code_smells)](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) [![Coverage](https://sonarcloud.io/api/project_badges/measure?project=Zipstack_unstract&metric=coverage)](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract) [![Duplicated Lines (%)](https://sonarcloud.io/api/project_badges/measure?project=Zipstack_unstract&metric=duplicated_lines_density)](https://sonarcloud.io/summary/new_code?id=Zipstack_unstract)

πŸ€– Prompt Studio

Prompt Studio's primary reason for existence is so you can develop the necessary prompts for document data extraction super efficiently. It is a purpose-built environment that makes this not just easy for youβ€”but, lot of fun! The document sample, its variants, the prompts you're developing, outputs from different LLMs, the schema you're developing, costing details of the extraction and various tools that let you measure the effectiveness of your prompts are just a click away and easily accessible. Prompt Studio is designed for effective and high speed development and iteration of prompts for document data extraction. Welcome to IDP 2.0!

img Prompt Studio

πŸ§˜β€β™€οΈ Three step nirvana with Workflow Studio

Automate critical business processes that involve complex documents with a human in the loop. Go beyond RPA with the power of Large Language Models.

🌟 Step 1: Add documents to no-code Prompt Studio and do prompt engineering to extract required fields
🌟 Step 2: Configure Prompt Studio project as API deployment or configure input source and output destination for ETL Pipeline
🌟 Step 3: Deploy Workflows as unstructured data APIs or unstructured data ETL Pipelines!

img Using Unstract

πŸš€ Getting started

System Requirements

Prerequisites

Next, either download a release or clone this repo and do the following:

βœ… ./run-platform.sh
βœ… Now visit http://frontend.unstract.localhost in your browser
βœ… Use user name and password unstract to login

That's all there is to it!

See user guide for more details on managing the platform.
Another really quick way to experience Unstract is by signing up for our hosted version.

⏩ Quick Start Guide

Unstract comes well documented. You can get introduced to the basics of Unstract, and learn how to connect various systems like LLMs, Vector Databases, Embedding Models and Text Extractors to it. The easiest way to wet your feet is to go through our Quick Start Guide where you actually get to do some prompt engineering in Prompt Studio and launch an API to structure varied credit card statements!

🀝 Ecosystem support

LLM Providers

Provider Status
OpenAI βœ… Working
Google VertexAI, Gemini Pro βœ… Working
Azure OpenAI βœ… Working
Anthropic βœ… Working
Ollama βœ… Working
Bedrock βœ… Working
Google PaLM βœ… Working
Anyscale βœ… Working
Mistral AI βœ… Working
Replicate πŸ—“οΈ Coming soon!

Vector Databases

Provider Status
Qdrant βœ… Working
Weaviate βœ… Working
Pinecone βœ… Working
PostgreSQL βœ… Working
Milvus βœ… Working

Embeddings

Provider Status
OpenAI βœ… Working
Azure OpenAI βœ… Working
Google PaLM βœ… Working
Ollama βœ… Working

Text Extractors

Provider Status
Unstract LLMWhisperer βœ… Working
Unstructured.io Community βœ… Working
Unstructured.io Enterprise βœ… Working
LlamaIndex Parse βœ… Working

ETL Sources

Provider Status
AWS S3 βœ… Working
Minio βœ… Working
Google Cloud Storage βœ… Working
Azure Cloud Storage βœ… Working
Google Drive βœ… Working
Dropbox βœ… Working
SFTP βœ… Working
Box πŸ—“οΈ Coming soon!
HTTP/HTTPS πŸ—“οΈ Coming soon!

ETL Destinations

Provider Status
Snowflake βœ… Working
Amazon Redshift βœ… Working
Google Bigquery βœ… Working
PostgreSQL βœ… Working
MySQL βœ… Working
MariaDB βœ… Working
Microsoft SQL Server βœ… Working

πŸ™Œ Contributing

Contributions are welcome! Please read CONTRIBUTING.md for further details on setting up the development environment, etc. It also points you to other detailed documents as needed.

πŸ‘‹ Join the LLM-powered automation community

🚨 Backup encryption key

Do copy the value of ENCRYPTION_KEY config in either backend/.env or platform-service/.env file to a secure location.

Adapter credentials are encrypted by the platform using this key. Its loss or change will make all existing adapters inaccessible!

πŸ“Š A note on analytics

In full disclosure, Unstract integrates Posthog to track usage analytics. As you can inspect the relevant code here, we collect the minimum possible metrics. Posthog can be disabled if desired by setting REACT_APP_ENABLE_POSTHOG to false in the frontend's .env file.