search

prefix-dev / rattler-build

rattler-build is a universal package builder for Windows, macOS and Linux
https://prefix-dev.github.io/rattler-build
BSD 3-Clause "New" or "Revised" License
218 stars 48 forks source link

race condition with `pin_subpackage` #984

Open pavelzw opened 4 months ago

pavelzw commented 4 months ago 
# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json

context:
  name: my-package

recipe:
  name: ${{ name }}
  version: '1.0.0'

outputs:
  - package:
      name: ${{ name }}-${{ variant }}

  - package:
      name: ${{ name }}
    requirements:
      run:
        - ${{ pin_subpackage('my-package-a', exact=true) if variant == 'a' else pin_subpackage('my-package-b', exact=true) }}
variant:
  - a
  - b

Sometimes, it works:

❯ rattler-build build -r recipe/recipe.yaml -m recipe/variants.yaml

 ╭─ Finding outputs from recipe
 │ Found 4 variants
 │ Build variant: my-package--1.0.0-hcdbb9e0_0
 │ 
 │ ╭─────────────────┬───────────╮
 │ │ Variant         ┆ Version   │
 │ ╞═════════════════╪═══════════╡
 │ │ target_platform ┆ osx-arm64 │
 │ │ variant         ┆ a         │
 │ ╰─────────────────┴───────────╯
 │ Build variant: my-package-1.0.0-hbb13421_0
 │ 
 │ ╭─────────────────┬──────────────────╮
 │ │ Variant         ┆ Version          │
 │ ╞═════════════════╪══════════════════╡
 │ │ my-package-a    ┆ 1.0.0 hcdbb9e0_0 │
 │ │ target_platform ┆ osx-arm64        │
 │ │ variant         ┆ a                │
 │ ╰─────────────────┴──────────────────╯
 │ Build variant: my-package--1.0.0-h89e5c08_0
 │ 
 │ ╭─────────────────┬───────────╮
 │ │ Variant         ┆ Version   │
 │ ╞═════════════════╪═══════════╡
 │ │ target_platform ┆ osx-arm64 │
 │ │ variant         ┆ b         │
 │ ╰─────────────────┴───────────╯
 │ Build variant: my-package-1.0.0-h41b9c1b_0
 │ 
 │ ╭─────────────────┬──────────────────╮
 │ │ Variant         ┆ Version          │
 │ ╞═════════════════╪══════════════════╡
 │ │ my-package-b    ┆ 1.0.0 h89e5c08_0 │
 │ │ target_platform ┆ osx-arm64        │
 │ │ variant         ┆ b                │
 │ ╰─────────────────┴──────────────────╯
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Checking existing builds
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: my-package-a-1.0.0-hcdbb9e0_0
 │
 │ ╭─ Fetching source code
 │ │ No sources to fetch
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running cache build
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Resolving environments
 │ │ 
 │ │ Finalized run dependencies: this output has no run dependencies
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running build script
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Packaging new files
 │ │ Copying done!
 │ │ Post-processing done!
 │ │ Writing test files
 │ │ Writing metadata for package
 │ │ Copying license files
 │ │ Copying recipe files
 │ │ Creating entry points
 │ │ 
 │ │ Files in package:
 │ │   - info/about.json
 │ │   - info/hash_input.json
 │ │   - info/index.json
 │ │   - info/paths.json
 │ │   - info/recipe/recipe.yaml
 │ │   - info/recipe/rendered_recipe.yaml
 │ │   - info/recipe/variant_config.yaml
 │ │   - info/recipe/variants.yaml
 │ │   - info/tests/tests.yaml
 │ │ Creating target folder "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64"
 │ │ Compressing archive...
 │ │ Archive written to "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-a-1.0.0-hcdbb9e0_0.conda"
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running package tests
 │ │ Removing previously cached package "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-a-1.0.0-hcdbb9e0_0"
 │ │ Creating test environment in "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/bld/rattler-build_my-package-a_1721894206/work/test"
 │ │ Collecting tests from "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-a-1.0.0-hcdbb9e0_0"
 │ │ ✔ all tests passed!
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: my-package-1.0.0-hbb13421_0
 │
 │ ╭─ Fetching source code
 │ │ No sources to fetch
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running cache build
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Resolving environments
 │ │ 
 │ │ Finalized run dependencies:
 │ │ ╭──────────────────┬─────────────────────────╮
 │ │ │ Name             ┆ Spec                    │
 │ │ ╞══════════════════╪═════════════════════════╡
 │ │ │ Run dependencies ┆                         │
 │ │ │ my-package-a     ┆ ==1.0.0 hcdbb9e0_0 (PS) │
 │ │ ╰──────────────────┴─────────────────────────╯
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running build script
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Packaging new files
 │ │ Copying done!
 │ │ ⚠ warning Overdepending against my-package-a
 │ │ Post-processing done!
 │ │ Writing test files
 │ │ Writing metadata for package
 │ │ Copying license files
 │ │ Copying recipe files
 │ │ Creating entry points
 │ │ 
 │ │ Files in package:
 │ │   - info/about.json
 │ │   - info/hash_input.json
 │ │   - info/index.json
 │ │   - info/paths.json
 │ │   - info/recipe/recipe.yaml
 │ │   - info/recipe/rendered_recipe.yaml
 │ │   - info/recipe/variant_config.yaml
 │ │   - info/recipe/variants.yaml
 │ │   - info/tests/tests.yaml
 │ │ Creating target folder "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64"
 │ │ Compressing archive...
 │ │ Archive written to "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-1.0.0-hbb13421_0.conda"
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running package tests
 │ │ Removing previously cached package "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-1.0.0-hbb13421_0"
 │ │ Creating test environment in "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/bld/rattler-build_my-package_1721894206/work/test"
 │ │ Collecting tests from "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-1.0.0-hbb13421_0"
 │ │ ✔ all tests passed!
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: my-package-b-1.0.0-h89e5c08_0
 │
 │ ╭─ Fetching source code
 │ │ No sources to fetch
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running cache build
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Resolving environments
 │ │ 
 │ │ Finalized run dependencies: this output has no run dependencies
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running build script
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Packaging new files
 │ │ Copying done!
 │ │ Post-processing done!
 │ │ Writing test files
 │ │ Writing metadata for package
 │ │ Copying license files
 │ │ Copying recipe files
 │ │ Creating entry points
 │ │ 
 │ │ Files in package:
 │ │   - info/about.json
 │ │   - info/hash_input.json
 │ │   - info/index.json
 │ │   - info/paths.json
 │ │   - info/recipe/recipe.yaml
 │ │   - info/recipe/rendered_recipe.yaml
 │ │   - info/recipe/variant_config.yaml
 │ │   - info/recipe/variants.yaml
 │ │   - info/tests/tests.yaml
 │ │ Creating target folder "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64"
 │ │ Compressing archive...
 │ │ Archive written to "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-b-1.0.0-h89e5c08_0.conda"
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running package tests
 │ │ Removing previously cached package "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-b-1.0.0-h89e5c08_0"
 │ │ Creating test environment in "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/bld/rattler-build_my-package-b_1721894206/work/test"
 │ │ Collecting tests from "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-b-1.0.0-h89e5c08_0"
 │ │ ✔ all tests passed!
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Running build for recipe: my-package-1.0.0-h41b9c1b_0
 │
 │ ╭─ Fetching source code
 │ │ No sources to fetch
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running cache build
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Resolving environments
 │ │ 
 │ │ Finalized run dependencies:
 │ │ ╭──────────────────┬─────────────────────────╮
 │ │ │ Name             ┆ Spec                    │
 │ │ ╞══════════════════╪═════════════════════════╡
 │ │ │ Run dependencies ┆                         │
 │ │ │ my-package-b     ┆ ==1.0.0 h89e5c08_0 (PS) │
 │ │ ╰──────────────────┴─────────────────────────╯
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running build script
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Packaging new files
 │ │ Copying done!
 │ │ ⚠ warning Overdepending against my-package-b
 │ │ Post-processing done!
 │ │ Writing test files
 │ │ Writing metadata for package
 │ │ Copying license files
 │ │ Copying recipe files
 │ │ Creating entry points
 │ │ 
 │ │ Files in package:
 │ │   - info/about.json
 │ │   - info/hash_input.json
 │ │   - info/index.json
 │ │   - info/paths.json
 │ │   - info/recipe/recipe.yaml
 │ │   - info/recipe/rendered_recipe.yaml
 │ │   - info/recipe/variant_config.yaml
 │ │   - info/recipe/variants.yaml
 │ │   - info/tests/tests.yaml
 │ │ Creating target folder "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64"
 │ │ Compressing archive...
 │ │ Archive written to "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-1.0.0-h41b9c1b_0.conda"
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Running package tests
 │ │ Removing previously cached package "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-1.0.0-h41b9c1b_0"
 │ │ Creating test environment in "/private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/bld/rattler-build_my-package_1721894206/work/test"
 │ │ Collecting tests from "/Users/pavel/Library/Caches/rattler/cache/pkgs/my-package-1.0.0-h41b9c1b_0"
 │ │ ✔ all tests passed!
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

 ╭─ Build summary
 │
 │ ╭─ Build summary for recipe: my-package-a-1.0.0-hcdbb9e0_0
 │ │ Artifact: /private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-a-1.0.0-hcdbb9e0_0.conda (1.60 KiB)
 │ │ Variant configuration (hash: hcdbb9e0_0):
 │ │ ╭─────────────────┬───────────╮
 │ │ │ target_platform ┆ osx-arm64 │
 │ │ │ variant         ┆ a         │
 │ │ ╰─────────────────┴───────────╯
 │ │ 
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Build summary for recipe: my-package-1.0.0-hbb13421_0
 │ │ Artifact: /private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-1.0.0-hbb13421_0.conda (1.67 KiB)
 │ │ Variant configuration (hash: hbb13421_0):
 │ │ ╭─────────────────┬──────────────────╮
 │ │ │ my-package-a    ┆ 1.0.0 hcdbb9e0_0 │
 │ │ │ target_platform ┆ osx-arm64        │
 │ │ │ variant         ┆ a                │
 │ │ ╰─────────────────┴──────────────────╯
 │ │ 
 │ │ Run dependencies:
 │ │ ╭──────────────────┬─────────────────────────╮
 │ │ │ Name             ┆ Spec                    │
 │ │ ╞══════════════════╪═════════════════════════╡
 │ │ │ Run dependencies ┆                         │
 │ │ │ my-package-a     ┆ ==1.0.0 hcdbb9e0_0 (PS) │
 │ │ ╰──────────────────┴─────────────────────────╯
 │ │ 
 │ │ ⚠ warning Warnings:
 │ │ ⚠ warning Overdepending against my-package-a
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Build summary for recipe: my-package-b-1.0.0-h89e5c08_0
 │ │ Artifact: /private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-b-1.0.0-h89e5c08_0.conda (1.63 KiB)
 │ │ Variant configuration (hash: h89e5c08_0):
 │ │ ╭─────────────────┬───────────╮
 │ │ │ target_platform ┆ osx-arm64 │
 │ │ │ variant         ┆ b         │
 │ │ ╰─────────────────┴───────────╯
 │ │ 
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 │ ╭─ Build summary for recipe: my-package-1.0.0-h41b9c1b_0
 │ │ Artifact: /private/var/folders/rz/q1r4tv0n6ll38q18fbk419mc0000gn/T/tmp.qDneok4ZpX/output/osx-arm64/my-package-1.0.0-h41b9c1b_0.conda (1.69 KiB)
 │ │ Variant configuration (hash: h41b9c1b_0):
 │ │ ╭─────────────────┬──────────────────╮
 │ │ │ my-package-b    ┆ 1.0.0 h89e5c08_0 │
 │ │ │ target_platform ┆ osx-arm64        │
 │ │ │ variant         ┆ b                │
 │ │ ╰─────────────────┴──────────────────╯
 │ │ 
 │ │ Run dependencies:
 │ │ ╭──────────────────┬─────────────────────────╮
 │ │ │ Name             ┆ Spec                    │
 │ │ ╞══════════════════╪═════════════════════════╡
 │ │ │ Run dependencies ┆                         │
 │ │ │ my-package-b     ┆ ==1.0.0 h89e5c08_0 (PS) │
 │ │ ╰──────────────────┴─────────────────────────╯
 │ │ 
 │ │ ⚠ warning Warnings:
 │ │ ⚠ warning Overdepending against my-package-b
 │ │
 │ ╰─────────────────── (took 0 seconds)
 │
 ╰─────────────────── (took 0 seconds)

Sometimes it doesn't:

❯ rattler-build build -r recipe/recipe.yaml -m recipe/variants.yaml

 ╭─ Finding outputs from recipe
 │
 ╰─────────────────── (took 0 seconds)
Error:   × Missing output: my-package-a (used in pin_subpackage)
wolfv commented 3 months ago

Hm, interesting! Thanks for the reproducer, too! Unsure why this fails.

wolfv commented 3 months ago

I think this one is related to #985 - because we migth not get the "right" name when ordering the packages this fails randomly. So I do think that we should retry this one when we fix #985.

pavelzw commented 1 month ago

found a smaller reproducer

# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json

recipe:
  name: my-package
  version: '1.0.0'

outputs:

  - package:
      name: my-package
    requirements:
      run:
        - ${{ pin_subpackage('my-package-a') }}

  - package:
      name: my-package-${{ variant }}
variant:
  - a