feat: layered package install and cache

[oocococo]

Description

I met a problem when I tried to install nvidia-tensorrt==7.2.3.4, it must be installed after nvidia-pyindex and in a seperate pip install process. However envd will merge all pip packages together. I can install tensorrt by run(["pip install nvidia-tensorrt==7.2.3.4"]) in the end, but this process can't be cached. I think layered package install and cache like docker-compose is very useful in this situation.

my build.envd

def build():
    base(os="ubuntu22.04", language="python3.8")
    config.pip_index(url="https://mirror.sjtu.edu.cn/pypi/web/simple")
    config.apt_source(source="""
    deb https://mirror.sjtu.edu.cn/ubuntu jammy main restricted
    deb https://mirror.sjtu.edu.cn/ubuntu jammy-updates main restricted
    deb https://mirror.sjtu.edu.cn/ubuntu jammy universe
    deb https://mirror.sjtu.edu.cn/ubuntu jammy-updates universe
    deb https://mirror.sjtu.edu.cn/ubuntu jammy multiverse
    deb https://mirror.sjtu.edu.cn/ubuntu jammy-updates multiverse
    deb https://mirror.sjtu.edu.cn/ubuntu jammy-backports main restricted universe multiverse
    deb http://archive.canonical.com/ubuntu jammy partner
    deb https://mirror.sjtu.edu.cn/ubuntu jammy-security main restricted universe multiverse
    """)
    config.conda_channel(channel='''
channels:
    - defaults
show_channel_urls: true
default_channels:
    - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
    - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
    - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
custom_channels:
    conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
''')
    config.gpu(count=1)

    install.apt_packages(name = [
            "htop",
            "fzf"
    ])

    install.python_packages(name = [
            "setuptools",
            "pip",
            "nvidia-pyindex",
            "nvidia-tensorrt==7.2.3.4",
            "sktime[dl,all-extras]",
            "seaborn",
            "pyts",
            "ipykernel",
            "tensorflow-gpu",
            "keras",
            "scipy",
            "numpy",
            "pandas",
            "scikit-learn",
            "h5py",
            "matplotlib",
    ])
    install.vscode_extensions([
             "ms-python.python",
             "ms-toolsai.jupyter",
#            "ms-python.vscode-pylance",
#            "ms-python.isort",
#            "GitHub.copilot"
    ])
    shell("zsh")
    run(["pip install nvidia-tensorrt==7.2.3.4"])

---

Message from the maintainers:

Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.

[gaocegege]

Thanks for the issue!

/cc @kemingy @VoVAllen

[kemingy]

I think we can merge PyPI packages in one install.python_packages.

run() should be cached since by default it won't mount the working directory.

Which version are you using? @oocococo envd version

[oocococo]

@kemingy In my build.envd, install.python_packages and run will be rerun every envd up

envd: v0.2.5-alpha.8
  BuildDate: 2022-11-23T13:24:58Z
  GitCommit: c24e555c0b7f6e84d87266a498acdf88a325b433
  GitTreeState: clean
  GitTag: v0.2.5-alpha.8
  GoVersion: go1.18.7
  Compiler: gc
  Platform: linux/amd64

[kemingy]

Hi @oocococo, we have changed the behavior in https://github.com/tensorchord/envd/pull/1289.

So next time, Python packages in different install.python_packages commands will be installed in different steps. Hope this can solve your issue.

You can test with the latest code or wait for the next release.

[oocococo]

Thank you @kemingy, I will have a try after the next release.