简体中文 ∙ English ∙ 日本語 ∙ 한국어

[!WARNING] CSGHub Portal has migrated the web framework from Rails to Gin. If you're using our Rails version before, you may need to refer to our new documents migrate to Gin to see how to migrate to Gin version.

CSGHub README

CSGHub is an open source, trustworthy large model asset management platform that can assist users in governing the assets involved in the lifecycle of LLM and LLM applications (datasets, model files, codes, etc).

With CSGHub, users can perform operations on LLM assets, including uploading, downloading, storing, verifying, and distributing, through Web interface, Git command line, or natural language Chatbot. Meanwhile, the platform provides microservice submodules and standardized OpenAPIs, which could be easily integrated with users' own systems.

CSGHub is committed to bringing users an asset management platform that is natively designed for large models and can be deployed On-Premise for fully offline operation. CSGHub offers functionalities similar to a privatized Huggingface(on-premise Huggingface), managing LLM assets in a manner akin to how OpenStack Glance manages virtual machine images, Harbor manages container images, and Sonatype Nexus manages artifacts.

You can try the free SaaS version of CSGHub through the OpenCSG Community official website.https://opencsg.com/models
You can also jump to the Quick Start section to quickly launch your local instance and explore all the features of CSGHub.

UPDATES

• [2024.09.15] v0.9.0 CSGHub portal been migrated to the Go version, git server support Gitaly and use it as default, collections function been added, new resource console page, running space endpoint finetune do not need domain anymore other UX optimization and bug fix.
• [2024.08.15] v0.8.0 Big release to introduce a standalone user_server service and move user, org, token management from CSGHub to backend server CSGHub-server, introduce a standalone starhub_server_runner service for uniformed deployment of Application Space, Mode Inference and Finetune. Resource management enchanced, multiple k8s clusters in different regions are not supported.
• [2024.07.15] v0.7.0 Big release to support Multiple Resource Sync for models and datasets, One-Click Fintune, Resource Usage Metering.
• [2024.06.21] v0.6.1 bug fix and user experience enhancement.
• [2024.06.18] v0.6.0 Big release to support dedicated model inference endpoint, support streamlit Space, allow use to like repos allow to set repo's industry tag, enhance git history and commit diff details.
• [2024.05.14] v0.5.0 Enhance Space user experience, auto build relations between repos(model,dataset,code and spaces), support multiple files uploading.
• [2024.04.18] v0.4.0 Allow to run Application Space (gradio app), add a widget to try model inference, support new repo type Code, support organization members management, support wechat login.
• [2024.03.15] v0.3.0 Plan: Files online editing, organization edit, dataset preview.
• [2024.02.15] v0.2.0 Improve the function of model dataset hosting, and add the feature of inviting new organization members.
• [2024.01.15] v0.1.0 CSGHub Alpha version release, supports model and dataset management functions, detailed function is as below.

CORE FUNCTIONS

In the era of LLM, data and models are increasingly becoming the most important digital assets for businesses and individual users. However, there are currently issues such as fragmented management tools, limited management methods, and localization, which not only pose potential threats to secure operations but also might hinder the updating and iteration of enterprise-scale models. If you believe that large models will become a major driving force in the upcoming revolution, you may also be considering how to manage core assets — models, data, and large model application code — more efficiently and securely. CSGHub is an open-source project designed to address these issues.

CSGHub's core funtions（update reguarly）：

• Unified Management of LLM Assets: A one-stop Hub for unified management of model files, datasets, and large-scale model application codes.
• Development Ecosystem Compatibility: Supports both HTTPS and SSH protocols for Git commands and web interface operations, ensuring convenient usage for different users.
• Large Model Capability Expansion: Natively supports version management, model format conversion, automatic data processing, and dataset preview functions.
• Permissions and Security: Supports integration with corporate user systems, setting of asset visibility, and zero-trust authentication interface design for both external and internal users, maximizing security.
• Support for Private Deployment: Independent of internet and cloud vendors, enabling one-click initiation of private deployment.
• Native Design for Large Models: Supports natural language interaction, one-click model deployment, and asset management for Agent and Copilot App.

TECH DESIGN

The technical design of CSGHub are as follows:

• CSGHub integrates multiple technologies including Git Servers, Git LFS (Large File Storage) protocol, and Object Storage Service (OSS), providing a reliable data storage layer, a flexible infrastructure access layer, and extensive support for development tools.
• Utilizing a service-oriented architecture, CSGHub offers backend services through CSGHub Server and a management interface via CSGHub Web Service. Ordinary users can quickly initiate services using Docker compose or Kubernetes Helm Chart for enterprise-level asset management. Users with in-house development capabilities can utilize CSGHub Server for secondary development to integrate management functions into external systems or to customize advanced features.
• Leveraging outstanding open-source projects like Apache Arrow and DuckDB, CSGHub supports previewing of Parquet data file formats, facilitating localized dataset management for researchers and common users.
• CSGHub provides an intuitive web interface and permission design for enterprise organization structure. Users can realize version control management, online browsing and downloading through the web UI, as well as set the visibility scope of datasets and model files to realize data security isolation, and can also initiate topic discussions on models and datasets.

Our R&D team has been focusing on AI + DevOps for a long time, and we hope to solve the pain points in the development process of large models through the CSGHub project. We encourage everyone to contribute high-quality development and operation and maintenance documents, and work together to improve the platform, so that large models assets can be more traceable and efficient.

DEMO VIDEO

In order to help users to quickly understand the features and usage of CSGHub, we have recorded a demo video. You can watch this video to get a quick understanding of the main features and operation procedures of this program.

• CSGHub Demo video is as below，you can also check it at YouTube or Bilibili

ROADMAP

• Asset Management
  • [x] Built-in Code Repo: Built-in Code Repo management function to associate the code of model, dataset, Space space application.
  • [x] Multi-source data synchronization: Support configure and enable remote repository, automatic data synchronization, support OpenCSG community, Huggingface and other remote sources。
• AI Enhancement
  • [x] One-Click Fine-Tuning: Support integration with OpenCSG llm-finetune tool to start model fine-tuning training with one click.
  • [x] One-Click Inference: Support integration with OpenCSG llm-inference tool to start model reasoning service with one click.
• LLM App and Enterprise Features
  • [x] App Space: Support hosting Gradio/Streamlit applications and publishing them to App Space.
  • [x] Fine-grained Permission Control: Fine-grained permission and access control settings for enterprise architecture.
• Security Compliance
  • [ ] GitServer Adapter: Generic GitServer adapter to support multiple major Git repository types through Adaptor mode.
  • [x] Asset Metadata: Asset metadata management mechanism, supporting customized metadata types and corresponding AutoTag rules.

The detailed roadmap is designed as follows: full roadmap

ARCHITECTURE

CSGHub is made with two typical parts: Portal and Server. This repo corresponds to CSGHub Portal, while CSGHub Server is another high-performance backend project implemented with Golang.

If you want to dive deep into CSGHub Server detail or wish to integrate the Server with your own frontend system or more, you can check the CSGHub Server open-source project.

CSGHub Portal Architecture

CSGHub Server Architecture

QUICK START

You can refer to herequickly deploy a basic csghub instance.

Tech docs in detail

• setup development env

Contributing

We welcome developers of all levels to contribute to our open-source project, CSGHub. If you would like to get involved, please refer to our contributing guidelines. We look forward to your participation and suggestions.

ACKNOWLEDGEMENTS

This project is based on Rails, Vue3, Tailwind CSS, Administrate, Postgresql, Apache Arrow, DuckDB and GoGin, whose open source contributions are deeply appreciated!

CONTACT WITH US

If you meet any problem during usage, you can contact with us by any following way:

1. initiate an issue in github
2. join our WeChat group by scaning wechat helper qrcode
3. join our official discord channel: OpenCSG Discord Channel
4. join our slack workspace:OpenCSG Slack Channel