janhq / jan

Jan is an open source alternative to ChatGPT that runs 100% offline on your computer. Multiple engine support (llama.cpp, TensorRT-LLM)
https://jan.ai/
GNU Affero General Public License v3.0
22.57k stars 1.3k forks source link

feat: Custom path issues for GGUFs #2995

Open eckartal opened 4 months ago

eckartal commented 4 months ago

Problem Using GGUF files with custom paths has been problematic. Even when following the exact naming scheme and structure, there are issues. Every time Jan is launched, it creates default folders for models that users do not have or want. Setting permissions to 644 prevents this but requires manual ‘sudo ln’ commands to add .gguf models due to import errors caused by permissions.

Success Criteria

Additional context Related Reddit comment to be updated: https://www.reddit.com/r/LocalLLaMA/comments/1d8n9wr/comment/l788tx8/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

louis-jan commented 4 months ago

Migrate Jan to use Cortex API could help address this, we are on it. There will be no longer any pre-populated models, and no default model folders will be created.

Van-QA commented 4 months ago

related to https://github.com/janhq/cortex/issues/665