prefix-dev / pixi

Package management made easy
https://pixi.sh
BSD 3-Clause "New" or "Revised" License
3.44k stars 197 forks source link

Pixi offline upgrading and everything else #1765

Open toiletsandpaper opened 3 months ago

toiletsandpaper commented 3 months ago

Problem description

Hi! Pixi is a great tool, I'm enjoing using it on my personal laptops and other.

But using pixi for serious commercial projects with cybersec limitations is a huge pain. For example, I have disconnected all my GPU clusters from internet access and setted up mirrors and cachers: Sonatype Nexus, Harbor, etc.; so I can work in a controlled environment. And other tools, like docker, podman, pdm, poetry, pip, conda/mamba is ready and can be upgraded with apt/dnf/pipx - pixi is not (self-upgrade requires internet access and https://github.com/prefix-dev/pixi/issues/1764)

I see only two ways to fix this:

  1. Write and open-source your own Nexus-like tool for mirror and cache packages (maybe like panamax for Rust crates)
  2. Make pixi offline-friendly and also distribute it with apt/dnf/pipx/etc.

Anyways, thank you for your work! Great tool!

pavelzw commented 3 months ago

getting pixi into apt and dnf would be great 👍🏻 not sure how the rules for these packaging ecosystems are, though. you can definitely maintain your own packages but it would be better if we managed to get them upstream into the main repos

ruben-arts commented 3 months ago

You can setup a selfhosted conda package mirror with Quetz

I'd be open to get pixi on pypi but currently don't have the capacity. apt or dnf would be even harder for me.

That said contributions are always welcome! So if anyone has the time and knowledge please help us get pixi distributed though other ecosystems.

toiletsandpaper commented 3 months ago

You can setup a selfhosted conda package mirror with Quetz

I have a problem with this. You can check my other opened issue, that I've linked in topic message, I've explained everything there.

tdejager commented 3 months ago

@toiletsandpaper would you also require like an --offline mode, where it does not touch the internet during pixi operations as well?

ruben-arts commented 3 months ago

I have a problem with this. You can check my other opened issue, that I've linked in topic message, I've explained everything there.

This issue in #1764 is unrelated to the self-hosted conda package mirror. Let me explain that issue in #1764.

toiletsandpaper commented 3 months ago

@toiletsandpaper would you also require like an --offline mode, where it does not touch the internet during pixi operations as well?

What kind of operation requires internet connection? I don't really think that tools like pixi really should require internet at all. Maybe for self-update or initial installation, but nothing else.

I saw pixi trying to dowload pypi-mappings from github. If you want to keep it up to date even if user don't want to upgrade pixi - yeah, --offline mode that forces pixi to disable mapping updating would be great. Maybe even as a flag in .config folder?

tdejager commented 3 months ago

Well, a lot of people like the option of having up to date packages and index data by default 😄 This is how most package managers operate. However, thats why uv and other tools offer a --offline flag.

toiletsandpaper commented 3 months ago

Well, a lot of people like the option of having up to date packages and index data by default 😄 This is how most package managers operate. However, thats why uv and other tools offer a --offline flag.

Yeah, I get it, but I have pypi and conda mirrors and --offline will ignore my mirrors too, right? If so, then --offline is not that relevant in my case, idk about others :)

But ability to export all packages, move it to offline and set it up with --offline flag looks like a great idea. Sounds like a https://github.com/pdm-project/pdm-download

ruben-arts commented 3 months ago

If you only have conda dependencies you can use pixi-pack which is the equivalent of conda-pack.

toiletsandpaper commented 3 months ago

I do really hope that someone with good rust expirience can make use of https://github.com/PyO3/maturin that was built to publish rust tools on PyPI. The closes example is https://github.com/astral-sh/uv as it can be downloaded via pip (pipx install uv)