Tests failing on main locally is there some cache or state I need to update?

[schneems]

Tests are failing on one of my prs and locally using main. Any ideas on what cache state might need to be invalidated as it seems the last PR merged into main was successful.

Here's my output:

$ rofl
## All commands

$ dockboot
$ cargo fmt
$ cargo clippy --all-targets --all-features -- --deny warnings
$ dockwait
$ time cargo test -- --ignored
$ git status
$ git status --porcelain

## Running

$ dockboot
Docker is booted

$ cargo fmt

$ cargo clippy --all-targets --all-features -- --deny warnings
    Finished dev [unoptimized + debuginfo] target(s) in 0.12s

$ dockwait
Docker is booted
Docker is booted

$ time cargo test -- --ignored
    Finished test [unoptimized + debuginfo] target(s) in 0.14s
     Running unittests src/main.rs (target/debug/deps/examples_basics-69676ce3b7891042)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/bin/dice_roller.rs (target/debug/deps/dice_roller-ea7e3e19c49a74d1)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/examples_execd-f208932e89a29f98)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running tests/integration_test.rs (target/debug/deps/integration_test-b5dad89e5ec79fb3)

running 1 test
warning: variable does not need to be mutable
   --> libcnb/src/runtime.rs:142:9
    |
142 |     let mut trace_error = |_: &dyn std::error::Error| {};
    |         ----^^^^^^^^^^^
    |         |
    |         help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default

warning: variable does not need to be mutable
   --> libcnb/src/runtime.rs:210:9
    |
210 |     let mut trace_error = |_: &dyn std::error::Error| {};
    |         ----^^^^^^^^^^^
    |         |
    |         help: remove this `mut`

warning: `libcnb` (lib) generated 2 warnings (run `cargo fix --lib -p libcnb` to apply 2 suggestions)
    Finished dev [unoptimized] target(s) in 0.05s
warning: variable does not need to be mutable
   --> libcnb/src/runtime.rs:142:9
    |
142 |     let mut trace_error = |_: &dyn std::error::Error| {};
    |         ----^^^^^^^^^^^
    |         |
    |         help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default

warning: variable does not need to be mutable
   --> libcnb/src/runtime.rs:210:9
    |
210 |     let mut trace_error = |_: &dyn std::error::Error| {};
    |         ----^^^^^^^^^^^
    |         |
    |         help: remove this `mut`

warning: `libcnb` (lib) generated 2 warnings (run `cargo fix --lib -p libcnb` to apply 2 suggestions)
    Finished dev [unoptimized] target(s) in 0.03s
test basic ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.40s

     Running unittests src/lib.rs (target/debug/deps/libcnb-7393cb66f879b495)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 46 filtered out; finished in 0.00s

     Running unittests src/main.rs (target/debug/deps/cargo_libcnb-59f9fdf43e381a0e)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.00s

     Running tests/integration_test.rs (target/debug/deps/integration_test-5520423bbdfb9ae0)

running 7 tests
test package_command_error_when_run_in_project_with_no_buildpacks ... ok
test package_command_respects_ignore_files ... ok
test package_non_libcnb_buildpack_in_composite_buildpack_project ... ok
test package_buildpack_in_single_buildpack_project ... ok
test package_single_buildpack_in_monorepo_buildpack_project ... ok
test package_single_composite_buildpack_in_monorepo_buildpack_project ... ok
test package_all_buildpacks_in_monorepo_buildpack_project ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.96s

     Running unittests src/lib.rs (target/debug/deps/libcnb_common-bd01f56e08bb9c6b)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/libcnb_data-e228e3a5c3233618)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 53 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/libcnb_package-f1d7db05e98a720b)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/libcnb_proc_macros-88c9d7ec7a153986)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/libcnb_test-7d8d14f5e53a92b2)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 33 filtered out; finished in 0.00s

     Running tests/integration_test.rs (target/debug/deps/integration_test-c5e4ef686be33fb5)

running 27 tests
test expected_pack_failure_still_panics_for_non_pack_failure - should panic ... ok
test app_dir_invalid_path ... ok
test expected_pack_failure ... ok
    Finished dev [unoptimized] target(s) in 0.00s
    Finished dev [unoptimized] target(s) in 0.00s
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on package cache
    Blocking waiting for file lock on build directory
    Finished dev [unoptimized] target(s) in 0.00s
    Finished dev [unoptimized] target(s) in 0.00s
   Compiling compile-error v0.0.0 (/Users/rschneeman/Documents/projects/work/libcnb.rs/libcnb-test/tests/fixtures/buildpacks/compile-error)
test download_sbom_files_failure - should panic ... okile-error(bin)
error: Example compile error
 --> src/main.rs:2:5
  |
2 |     compile_error!("Example compile error");
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: could not compile `compile-error` (bin "compile-error") due to 1 previous error
test packaging_failure_composite_buildpack_missing_package_toml - should panic ... ok
test packaging_failure_compile_error - should panic ... ok
    Finished dev [unoptimized] target(s) in 0.00s
test packaging_failure_missing_buildpack_toml ... ok
test packaging_failure_invalid_buildpack_toml - should panic ... ok
test packaging_failure_invalid_cargo_toml - should panic ... ok
test packaging_failure_non_existent_workspace_buildpack ... ok
test app_dir_absolute_path ... ok
test build_other_buildpack ... ok
test build_workspace_component_buildpack ... ok
test build_workspace_composite_buildpack ... ok
test unexpected_pack_failure - should panic ... ok
test build_multiple_buildpacks ... ok
test address_for_port_when_port_not_exposed - should panic ... ok
test run_shell_command_nonzero_exit_status - should panic ... ok
test logs_work_after_container_crashed ... ok
test address_for_port_when_container_crashed ... ok
test rebuild ... ok
test unexpected_pack_success - should panic ... ok
test shell_exec_nonzero_exit_status - should panic ... ok
test start_container_spawn_failure - should panic ... ok
test shell_exec_when_container_has_crashed - should panic ... ok
test app_dir_preprocessor ... ok
test starting_containers ... FAILED

failures:

---- starting_containers stdout ----
thread 'starting_containers' panicked at libcnb-test/tests/integration_test.rs:502:21:
assertion failed: `(is empty)`
value (unescaped):
python3 -u -m http.server ${PORT:+"${PORT}"}: line 1: python3 -u -m http.server : command not found

value (escaped): `"python3 -u -m http.server ${PORT:+\"${PORT}\"}: line 1: python3 -u -m http.server : command not found\n"`

failures:
    starting_containers

test result: FAILED. 26 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 8.11s

error: test failed, to rerun pass `-p libcnb-test --test integration_test`
  6.70s user 2.51s system 78% cpu 11.703 total
⛄️ 3.1.4 🚀 /Users/rschneeman/Documents/projects/work/libcnb.rs (main)
$ git log
commit f0983996fd4e8a0cf5bb916ed4608b731fb6474c (HEAD -> main, origin/main, origin/HEAD)
Author: Ed Morley <501702+edmorley@users.noreply.github.com>
Date:   Wed Feb 28 13:42:05 2024 +0000

    Fix `starting_containers` test after release of Procfile CNB v3.0.0 (#801)

    Procfile CNB v3.0.0 was just released with an intentional change to the
    way that `command` vs `args` are handled in the CNB process type
    definition. (Before the `Procfile` file entry would be set as the
    process `command`, but now it's set as `args`.)

    That change improves the overall UX of running images that use the
    Procfile CNB, but is breaking in some lesser used scenarios that
    happened to be tested via the `starting_containers` test in this repo.

    See:
    - https://github.com/heroku/procfile-cnb/pull/205#discussion_r1495090822
    - https://github.com/heroku/procfile-cnb/compare/v2.0.2...v3.0.0#diff-782521a81713992d3a07e85975d367cfac60afc78583133551efcddc2026bd3eL19-R20

    The tests have been updated to account for the new behaviour, and an
    additional test added for the "overriding command only" scenario (that
    wasn't possible to easily test before due to the way the Procfile CNB
    was previously implemented).

    Fixes #800.
    GUS-W-15139634.

[edmorley]

Running docker pull heroku/builder:22 locally to pick up the latest builder image (which includes Procfile CNB v3.0.0) should resolve the failures.

The reason running the libcnb-test tests locally don't pick up this new version is because we intentionally use --pull-policy if-not-present when libcnb-test invokes pack build, which means if the image exists locally a newer one isn't pulled to (a) reduce the chance of hitting Docker rate limits when running lots of tests, (b) to improve test performance.

This is obviously a trade-off, however, it's only in fairly rare circumstances that this will cause an issue (in this instance we intentionally made a breaking change in Procfile CNB which affects some lesser used scenarios + the start_container tests in this repo happen to test those lesser used scenarios for completeness, though in reality it's not something many users will really hit.)

[edmorley]

Also worth noting that using pack build locally in its default config will cause an image pull too (since it defaults to --pull-policy always), so over the long term the builder image on buildpack maintainers machines will stay up to date even though libcnb-test itself doesn't pull newer versions.

[schneems]

Thanks, that did it.