Proposed conda-vendor improvements

[rigzba21]

Edit: Closed https://github.com/MetroStar/conda-vendor/pull/32 in favor of tracking progress here as this is a much bigger refactor.

Background

conda-lock has some awesome improvements in 1.x 🔭 that will allow us to reduce duplicated functionality in conda-vendor's implementation.

Example conda-lock usage for 1.x:

Given an environment.yaml:

name: minimal
channels:
  - conda-forge
dependencies:
  - python
  - pip
  - micromamba

Generating a lockfile (conda-lock supports multiple solvers such as mamba and micromamba): conda lock --file environment.yaml -p linux-64 --mamba

Produces the following conda-lock.yml:

# This lock file was generated by conda-lock (https://github.com/conda-incubator/conda-lock). DO NOT EDIT!
#
# A "lock file" contains a concrete list of package versions (with checksums) to be installed. Unlike
# e.g. `conda env create`, the resulting environment will not change as new package versions become
# available, unless you explicitly update the lock file.
#
# Install this environment as "YOURENV" with:
#     conda-lock install -n YOURENV --file conda-lock.yml
# To update a single package to the latest version compatible with the version constraints in the source:
#     conda-lock lock --lockfile conda-lock.yml --update PACKAGE
# To re-solve the entire environment, e.g. after changing a version constraint in the source file:
#     conda-lock -f environment.yaml --lockfile conda-lock.yml
metadata:
  channels:
  - url: conda-forge
    used_env_vars: []
  content_hash:
    linux-64: 0e3f55cef4b08ecec24c4dda8e5ce0617f9f55f4e457bca0d9c16c9d8fe00bc5
  platforms:
  - linux-64
  sources:
  - environment.yaml
package:
- category: main
  dependencies: {}
  hash:
    md5: d7c89558ba9fa0495403155b64376d81
    sha256: fe51de6107f9edc7aa4f786a70f4a883943bc9d39b3bb7307c04c41410990726
  manager: conda
  name: _libgcc_mutex
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2
  version: '0.1'
- category: main
  dependencies: {}
  hash:
    md5: 575611b8a84f45960e87722eeb51fa26
    sha256: d13c8774129e0d8d1427f5758fba53cfa915b6a12cd4dbd2bfe612d9eab0506d
  manager: conda
  name: ca-certificates
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2021.10.8-ha878542_0.tar.bz2
  version: 2021.10.8
- category: main
  dependencies: {}
  hash:
    md5: bd4f2e711b39af170e7ff15163fe87ee
    sha256: ad7985a9ff622880cf87c42db1ffe2dfb040d8175c1bb352fc8f3705c7e0962f
  manager: conda
  name: ld_impl_linux-64
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ld_impl_linux-64-2.36.1-hea4e1c9_2.tar.bz2
  version: 2.36.1
- category: main
  dependencies: {}
  hash:
    md5: 24072cb5ef3fa80347bd35f184dfdaed
    sha256: f8d6d9ab832401f8f32e161d5043b28fd7f043d8f0829ab5388f6e4a4256524a
  manager: conda
  name: micromamba
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/micromamba-0.22.0-0.tar.bz2
  version: 0.22.0
- category: main
  dependencies: {}
  hash:
    md5: 84be5301069417a2221187d2f435e0f7
    sha256: 74d8c1fbccae1a78c9bd2b2d1cda73df425cc28717a637198c23bd1c9b53b60e
  manager: conda
  name: tzdata
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/tzdata-2022a-h191b570_0.tar.bz2
  version: 2022a
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
  hash:
    md5: a77fb1a92411cb8d979de1c2d81dd210
    sha256: 1da28d8d10c93e43c78fb5020dd9022fe24687f759acc25de699185bdfa84e9b
  manager: conda
  name: libgomp
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgomp-11.2.0-h1d223b6_14.tar.bz2
  version: 11.2.0
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
    libgomp: '>=7.5.0'
  hash:
    md5: 561e277319a41d4f24f5c05a9ef63c04
    sha256: 81c74d38c80345e195106dc3a5b4063b61f2209402bf9f6c7e2abadef4f544a3
  manager: conda
  name: _openmp_mutex
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-1_gnu.tar.bz2
  version: '4.5'
- category: main
  dependencies:
    _libgcc_mutex: 0.1 conda_forge
    _openmp_mutex: '>=4.5'
  hash:
    md5: 47e6c01d149b26090748d9d1ac32491b
    sha256: d24e25272239827012441e3376abcd2859a29418da825e6a593fc517b0c20f61
  manager: conda
  name: libgcc-ng
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-11.2.0-h1d223b6_14.tar.bz2
  version: 11.2.0
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
  hash:
    md5: a1fd65c7ccbf10880423d82bca54eb54
    sha256: cb521319804640ff2ad6a9f118d972ed76d86bea44e5626c09a13d38f562e1fa
  manager: conda
  name: bzip2
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/bzip2-1.0.8-h7f98852_4.tar.bz2
  version: 1.0.8
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: d645c6d2ac96843a2bfaccd2d62b3ac3
    sha256: ab6e9856c21709b7b517e940ae7028ae0737546122f83c2aa5d692860c3b149e
  manager: conda
  name: libffi
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libffi-3.4.2-h7f98852_5.tar.bz2
  version: 3.4.2
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: 39b1328babf85c7c3a61636d9cd50206
    sha256: 32f4fb94d99946b0dabfbbfd442b25852baf909637f2eed1ffe3baea15d02aad
  manager: conda
  name: libnsl
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libnsl-2.0.0-h7f98852_0.tar.bz2
  version: 2.0.0
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
  hash:
    md5: 772d69f030955d9646d3d0eaf21d859d
    sha256: 54f118845498353c936826f8da79b5377d23032bcac8c4a02de2019e26c3f6b3
  manager: conda
  name: libuuid
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libuuid-2.32.1-h7f98852_1000.tar.bz2
  version: 2.32.1
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
  hash:
    md5: 757138ba3ddc6777b82e91d9ff62e7b9
    sha256: b46b66d1cb171be2227a275e226195ca9e56c6f5b16250b85645e82a69518378
  manager: conda
  name: libzlib
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/libzlib-1.2.11-h166bdaf_1014.tar.bz2
  version: 1.2.11
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
  hash:
    md5: fb31bcb7af058244479ca635d20f0f4a
    sha256: bcb38449634bfe58e821c28d6814795b5bbad73514f0c7a9af7a710bbffc8243
  manager: conda
  name: ncurses
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/ncurses-6.3-h9c3ff4c_0.tar.bz2
  version: '6.3'
- category: main
  dependencies:
    ca-certificates: ''
    libgcc-ng: '>=10.3.0'
  hash:
    md5: 49bf4e64d1e86676b90a8657c1142f01
    sha256: 123f0bd67843220fb27da6b71ba126934edbe714415a630ddec0f8c8a2b88cf0
  manager: conda
  name: openssl
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/openssl-3.0.2-h166bdaf_1.tar.bz2
  version: 3.0.2
- category: main
  dependencies:
    libgcc-ng: '>=7.5.0'
  hash:
    md5: 33f601066901f3e1a85af3522a8113f9
    sha256: 1e2823cb2a526bc3a7031ad5dbfb992891f9ff9740d1c17cb6dbb8ebdfd33b27
  manager: conda
  name: xz
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.5-h516909a_1.tar.bz2
  version: 5.2.5
- category: main
  dependencies:
    libgcc-ng: '>=9.3.0'
    ncurses: '>=6.2,<7.0.0a0'
  hash:
    md5: 5788de3c8d7a7d64ac56c784c4ef48e6
    sha256: 30464670b3c81ac739e8df6b2c3c57b56d1e1408572540dec63bf4b8713163e4
  manager: conda
  name: readline
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/readline-8.1-h46c0cb4_0.tar.bz2
  version: '8.1'
- category: main
  dependencies:
    libgcc-ng: '>=9.4.0'
    libzlib: '>=1.2.11,<1.3.0a0'
  hash:
    md5: 5b8c42eb62e9fc961af70bdd6a26e168
    sha256: 032fd769aad9d4cad40ba261ab222675acb7ec951a8832455fce18ef33fa8df0
  manager: conda
  name: tk
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/tk-8.6.12-h27826a3_0.tar.bz2
  version: 8.6.12
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
    libzlib: 1.2.11 h166bdaf_1014
  hash:
    md5: def3b82d1a03aa695bb38ac1dd072ff2
    sha256: ccfdb4dcceae8b191ddd4703e7be84eff2ba82b53788d6bb9298e531bae4eaf9
  manager: conda
  name: zlib
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/zlib-1.2.11-h166bdaf_1014.tar.bz2
  version: 1.2.11
- category: main
  dependencies:
    libgcc-ng: '>=10.3.0'
    libzlib: '>=1.2.11,<1.3.0a0'
    ncurses: '>=6.3,<7.0a0'
    readline: '>=8.1,<9.0a0'
    zlib: '>=1.2.11,<1.3.0a0'
  hash:
    md5: 8057ac02d6d10a162d7eb4b0ca7ed291
    sha256: 5b1f7e51e6f6453c295cd911b826327b7eba4785b0366cf63cf6f828ec346076
  manager: conda
  name: sqlite
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/sqlite-3.37.1-h4ff8645_0.tar.bz2
  version: 3.37.1
- category: main
  dependencies:
    bzip2: '>=1.0.8,<2.0a0'
    ld_impl_linux-64: '>=2.36.1'
    libffi: '>=3.4.2,<3.5.0a0'
    libgcc-ng: '>=10.3.0'
    libnsl: '>=2.0.0,<2.1.0a0'
    libuuid: '>=2.32.1,<3.0a0'
    libzlib: '>=1.2.11,<1.3.0a0'
    ncurses: '>=6.3,<7.0a0'
    openssl: '>=3.0.2,<4.0a0'
    readline: '>=8.1,<9.0a0'
    sqlite: '>=3.37.1,<4.0a0'
    tk: '>=8.6.12,<8.7.0a0'
    tzdata: ''
    xz: '>=5.2.5,<5.3.0a0'
  hash:
    md5: 0f72b088a5471e97309031e1636e7b3f
    sha256: 70eb462c28c5467c6e4860d5f574d240350b6ac718990b23cb0cc144d1dbea3f
  manager: conda
  name: python
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/python-3.10.4-h2660328_0_cpython.tar.bz2
  version: 3.10.4
- category: main
  dependencies:
    python: 3.10.*
  hash:
    md5: 9e7160cd0d865e98f6803f1fe15c8b61
    sha256: e7e52aaec7cba6e17e45d731f9d38ede007aea0d72aee66670ab71016f5783ed
  manager: conda
  name: python_abi
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.10-2_cp310.tar.bz2
  version: '3.10'
- category: main
  dependencies:
    python: '!=3.0,!=3.1,!=3.2,!=3.3,!=3.4'
  hash:
    md5: 1ca02aaf78d9c70d9a81a3bed5752022
    sha256: aede66e6370f3b936164a703e48362f9080d7162234058fb2ee63cc84d528afc
  manager: conda
  name: wheel
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/wheel-0.37.1-pyhd8ed1ab_0.tar.bz2
  version: 0.37.1
- category: main
  dependencies:
    python: '>=3.10,<3.11.0a0'
    python_abi: 3.10.* *_cp310
  hash:
    md5: 2bf50027b62c5e607310c1755c27e482
    sha256: 2d5aba1f98b586b637e66bd1593424f4d5530cbd73b06883b460f2947abc244e
  manager: conda
  name: setuptools
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/linux-64/setuptools-61.2.0-py310hff52083_3.tar.bz2
  version: 61.2.0
- category: main
  dependencies:
    python: '>=3.7'
    setuptools: ''
    wheel: ''
  hash:
    md5: b1239ce8ef2a1eec485c398a683c5bff
    sha256: d36bb23fa250be2d6a21cafe1760a7ae434318fb397c85223dd6a0c8e6e5562b
  manager: conda
  name: pip
  optional: false
  platform: linux-64
  url: https://conda.anaconda.org/conda-forge/noarch/pip-22.0.4-pyhd8ed1ab_0.tar.bz2
  version: 22.0.4
version: 1

Proposed conda-vendor changes + improvements:

Remove conda-vendor's meta-manifest generation

I propose that we remove the meta-manifest generation, as conda-lock's new lockfile format now includes all of the necessary information we use to vendor dependencies into a local channel.

• [x] Remove intermediary step of generating a meta-manifest in favor of conda-lock's 1.0.x new API
• [x] Add in vendor command as the primary
• [x] Add a subcommand to generate formatted output for IronBank's hardening_manifest.yaml resources block, using conda-lock's 1.0.x FetchAction object.

Remove conda-vendor's combined manifest functionality

conda-lock now has compound specification for lockfile generation, where you can create a conda-lock.yaml from one or more environment files.

• [ ] Remove intermediary step of generating a meta-manifest in favor of using conda-lock's 1.0.x compound-specification API NOTE: this would be best tracked as it's own issue

  Signing and Verification

  I propose that we introduce signing and verification of the vendored dependencies within a local channel (and/or the local channel itself), and generate a SLSA compliant in-toto spec attestation. NOTE: this would be best tracked as it's own issue

• [ ] Define attestation + SBOM format (see notes below)
• [ ] sigstore digital signing

[rigzba21]

Possible integration point for SBOM generation? https://github.com/anchore/syft/issues/932

[rigzba21]

New Usage:

# use conda as the solver for linux-64
conda-vendor vendor --file environment.yaml --solver conda --platform linux-64

# use mamba as the solver for osx-64
conda-vendor vendor --file environment.yaml --solver mamba --platform osx-64

# use micromamba as the solver for the host platform
conda-vendor vendor --file environment.yaml --solver micromamba

Now supports conda, mamba, and micromamba solvers.

Attest a Vendored Channel

Generate a SLSA spec compliant attestation

# "attest" as a subcommand
conda-vendor attest --vendored-channel path/to/my/vendored-channel/

which would produce an attestation file attestation.yaml.

Attestation Format (WIP):

Components (from in-toto attestation spec):

• Envelope: Handles authentication and serialization
• Statement: Binds the attestation to a particular subject and unambiguously identifies the types of the predicate
• Predicate: Contains arbitrary metadata about the subject, with a type-specific schema
• Bundle: Defines a method of grouping multiple attestations together

(WIP) for conda-vendor:

• Envelope: TBD
  • DSSE?
  • other?
• Statement: in-toto/attestation
• Predicate: TBD
  • SLSA Provenance?
  • SPDX Spec?
  • what works best for conda channels? packages? environments?
  • Bundle: TBD
  • vendored conda channels

References, Notes, and Links:

• SLSA Software Attestations
• SLSA Attestation Recommended Suite
• in-toto Attestation Spec