Add Upstage AI Integration (Document Parse, Embeddings, and Generation)

[hunkim]

This PR introduces comprehensive integration with Upstage AI services, adding three new components.

They have high performance. (For more information, visit https://www.upstage.ai/ and https://www.upstage.ai/products/document-parse)

1. Document Processing:

   • Added UpstageDocumentParseReader for converting documents to structured HTML format
   • Integrated with Upstage Document AI API for enhanced document parsing

2. Embedding Support:

   • Added UpstageEmbedder supporting 'embedding-query' and 'embedding-passage' models
   • Implemented vector generation for both documents and queries

3. Text Generation:

   • Added UpstageGenerator supporting Solar models (solar-pro, solar-mini)
   • Implemented streaming response generation with context handling

Configuration Updates:

• Added new environment variables:
  • UPSTAGE_API_KEY: For authentication
  • UPSTAGE_BASE_URL: For custom endpoint configuration (optional)
• Updated README.md to reflect new capabilities and configuration options

The integration is available in both production and development environments, maintaining consistent API patterns with existing integrators.

Testing:

• Verified document parsing functionality with various file formats
• Tested embedding generation and similarity search
• Validated streaming text generation with context windows
• Confirmed proper error handling and rate limiting

Dependencies:

• No new package dependencies required
• Uses existing aiohttp/httpx for API communication

Documentation:

• Updated README.md with new features and configuration options
• Added inline documentation for all new components

This integration enhances Verba's capabilities with state-of-the-art document processing, embedding, and generation features from Upstage AI.

[weaviate-git-bot]

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Slack channel?

[hunkim]

If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

I agree. Thanks!!