Discussion: Cortex.cpp installation & uninstallation

[namchuai]

Motivation Couple of options:

1. via Package managers (brew, winget, launchpad, etc.)
   • Pros:
   • Easy to update (maybe?) just provide package manager team
   • Easy for acquire user? (like just running brew install cortexcpp or something like that)
   • Cons:
   • Take long time for new version come out
   • Hard (or impossible) to choose a place to install binary/files (not entirely sure. Open for discussion)
2. self hosted
   • Pros:
   • Not being constrained by Package managers
   • Shorter time to market
   • Easy to have multiple variants (stable, beta, nightly)
   • Let user decide where to store files. More managable?
   • Cons:
   • Have to handle upgrade ourselves
   • User acquisition is more difficult?
3. Hmm, both?

Related Issues

• https://github.com/janhq/jan/issues/2643

• 775

• https://github.com/janhq/jan/issues/3360

[hiento09]

Installer:

• Windows
  • Format: .exe
  • Binary location:
    • Stable: C:\Users\<username>\AppData\Local\cortex\cortex.exe
    • Beta: C:\Users\<username>\AppData\Local\cortex-beta\cortex-beta.exe
    • Nighty: C:\Users\<username>\AppData\Local\cortex-nightly\cortex-nightly.exe
  • Configuration file:
    • Stable: C:\Users\<username>\.cortexrc
    • Beta: C:\Users\<username>\.cortexrc-beta
    • Nighty: C:\Users\<username>\.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: C:\Users\<username>\.cortexcpp
    • Beta: C:\Users\<username>\.cortexcpp-beta
    • Nighty: C:\Users\<username>\.cortexcpp-nightly
• Linux:
  • Format: .deb
  • Binary location:
    • Stable: /usr/bin/cortex
    • Beta: /usr/bin/cortex-beta
    • Nighty: /usr/bin/cortex-nightly
  • Configuration file:
    • Stable: /home/<username>/.cortexrc
    • Beta: /home/<username>/.cortexrc-beta
    • Nighty: /home/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /home/<username>/.cortex
    • Beta: /home/<username>/.cortexcpp-beta
    • Nighty: /home/<username>/.cortexcpp-nightly
• Macos:
  • Format: .pkg
  • Binary location:
    • Stable: /usr/local/bin/cortex
    • Beta: /usr/local/bin/cortex-beta
    • Nighty: /usr/local/bin/cortex-nightly
  • Configuration file:
    • Stable: /Users/<username>/.cortexrc
    • Beta: /Users/<username>/.cortexrc-beta
    • Nighty: /Users/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /Users/<username>/.cortexcpp
    • Beta: /Users/<username>/.cortexcpp-beta
    • Nighty: /Users/<username>/.cortexcpp-nightly

[0xSage]

@hiento09 I spot some inconsistencies in your folder/file naming vs #1040

[hiento09]

@hiento09 I spot some inconsistencies in your folder/file naming vs #1040

I just edited my comment base on this comment https://github.com/janhq/cortex/issues/1040#issuecomment-2327938276 from @dan-homebrew

[dan-homebrew]

@hiento09 I made some changes to your folder/binary layout, based on our earlier discussion #1092

Changes

• CLI command is cortex, and binary name is cortex (or cortex-beta, cortex-nightly)
• Otherwise, our product is cortexcpp (with versions: cortexcpp-beta, cortexcpp-nightly)
• Our convention is to hyphenate product-version
• Data folder is pending discussion on visible vs. hidden folder in #1040

Installer:

• Windows
  • Format: .exe
  • Binary location:
    • Stable: C:\Users\<username>\AppData\Local\cortexcpp\cortex.exe
    • Beta: C:\Users\<username>\AppData\Local\cortexcpp-beta\cortex-beta.exe
    • Nighty: C:\Users\<username>\AppData\Local\cortexcpp-nightly\cortex-nightly.exe
  • Configuration file:
    • Stable: C:\Users\<username>\.cortexrc
    • Beta: C:\Users\<username>\.cortexrc-beta
    • Nighty: C:\Users\<username>\.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: C:\Users\<username>\.cortexcpp
    • Beta: C:\Users\<username>\.cortexcpp-beta
    • Nighty: C:\Users\<username>\.cortexcpp-nightly
• Linux:
  • Format: .deb
  • Binary location:
    • Stable: /usr/bin/cortexcpp
    • Beta: /usr/bin/cortexcpp-beta
    • Nighty: /usr/bin/cortexcpp-nightly
  • Configuration file:
    • Stable: /home/<username>/.cortexrc
    • Beta: /home/<username>/.cortexrc-beta
    • Nighty: /home/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /home/<username>/.cortexcpp
    • Beta: /home/<username>/.cortexcpp-beta
    • Nighty: /home/<username>/.cortexcpp-nightly
• Macos:
  • Format: .pkg
  • Binary location:
    • Stable: /usr/local/bin/cortexcpp
    • Beta: /usr/local/bin/cortexcpp-beta
    • Nighty: /usr/local/bin/cortexcpp-nightly
  • Configuration file:
    • Stable: /Users/<username>/.cortexrc
    • Beta: /Users/<username>/.cortexrc-beta
    • Nighty: /Users/<username>/.cortexrc-nightly
  • Data folder (contains engines, models, logs)
    • Stable: /Users/<username>/.cortexcpp
    • Beta: /Users/<username>/.cortexcpp-beta
    • Nighty: /Users/<username>/.cortexcpp-nightly

[hiento09]

@hiento09 I made some changes to your folder/binary layout, based on our earlier discussion #1092

Changes

• CLI command is cortex, and binary name is cortex (or cortex-beta, cortex-nightly)

• Our convention is to hyphenate name-version

• Data folder is pending discussion on visible vs. hidden folder in #1040

Installer:

• Windows

  • Format: .exe

  • Binary location:

    • Stable: C:\Users\<username>\AppData\Local\cortexcpp\cortex.exe

    • Beta: C:\Users\<username>\AppData\Local\cortexcpp-beta\cortex-beta.exe

    • Nighty: C:\Users\<username>\AppData\Local\cortexcpp-nightly\cortex-nightly.exe

  • Configuration file:

    • Stable: C:\Users\<username>\.cortexrc

    • Beta: C:\Users\<username>\.cortexrc-beta

    • Nighty: C:\Users\<username>\.cortexrc-nightly

  • Data folder (contains engines, models, logs)

    • Stable: C:\Users\<username>\.cortexcpp

    • Beta: C:\Users\<username>\.cortexcpp-beta

    • Nighty: C:\Users\<username>\.cortexcpp-nightly

• Linux:

  • Format: .deb

  • Binary location:

    • Stable: /usr/bin/cortexcpp

    • Beta: /usr/bin/cortexcpp-beta

    • Nighty: /usr/bin/cortexcpp-nightly

  • Configuration file:

    • Stable: /home/<username>/.cortexrc

    • Beta: /home/<username>/.cortexrc-beta

    • Nighty: /home/<username>/.cortexrc-nightly

  • Data folder (contains engines, models, logs)

    • Stable: /home/<username>/.cortexcpp

    • Beta: /home/<username>/.cortexcpp-beta

    • Nighty: /home/<username>/.cortexcpp-nightly

• Macos:

  • Format: .pkg

  • Binary location:

    • Stable: /usr/local/bin/cortexcpp

    • Beta: /usr/local/bin/cortexcpp-beta

    • Nighty: /usr/local/bin/cortexcpp-nightly

  • Configuration file:

    • Stable: /Users/<username>/.cortexrc

    • Beta: /Users/<username>/.cortexrc-beta

    • Nighty: /Users/<username>/.cortexrc-nightly

  • Data folder (contains engines, models, logs)

    • Stable: /Users/<username>/.cortexcpp

    • Beta: /Users/<username>/.cortexcpp-beta

    • Nighty: /Users/<username>/.cortexcpp-nightly

Thank you @dan-homebrew

[hiento09]

Task to do here https://github.com/janhq/cortex.cpp/issues/1030

[hiento09]

For installer, llamacpp engine will be installed by default

[dan-homebrew]

@hiento09 Can I clarify my understanding of the Installer, as there are a couple of things that are not aligned from our discussion last week:

Last week's discussion: https://link.excalidraw.com/l/kFY0dI05mm/3rpQK5NI8Xb

Installer Contents

• Cortex will ship with llama.cpp as a default engine
• TensorRT-LLM and ONNX will (for now) be initialized by running cortex init <engine>

Installation Methods

For the 3 operating systems, my understanding was as follows:

• Windows: .exe
• Mac: .dmg
• Linux: Install script (we would handle .deb and .appimage later on), in the format of:

curl -fsSL https://get.cortex.so -o get-cortex.sh

From the post above, it seems like we're going directly to .deb and .appImage.

My question: Does the .deb and .appImage wrap the Installer Script? There are definite advantages to having a .deb and .appimage, I just want to know how much more effort it is.

Installation Logic

My understanding is that our Installation logic focuses on two tasks:

• Hardware Detection (e.g. GPU type, and CPU instruction set - e.g. AVX2, AVX512)
• Pulling the correct version of llama.cpp at install time

Alternatives

I am increasingly thinking that it is a better idea for us to pack a larger installer:

• Cortex will be used in offline environments (e.g. USB install), and packing a full installer is better
• We can delete the unused llama.cpp versions during the installation process, to optimize long-term disk usage
• This seems to be the approach taken by the other llama.cpp UIs and libraries (e.g. Ollama, LMStudio)

[hiento09]

My question: Does the .deb and .appImage wrap the Installer Script? There are definite advantages to having a .deb and .appimage, I just want to know how much more effort it is.

This is an example of template structure for .deb file in linux Yes, it wraps bash scripts and allows us to interact with hooks like pre-install, post-install, pre-uninstall, and post-uninstall. It also has installation and uninstallation functions. If we go with a bash script, then we have to write all of the above functions ourselves.

[hiento09]

Link the final decision for naming here https://github.com/janhq/cortex.cpp/issues/1092