GitLab CI/CD: Jobs with needs do not download dotenv artifacts by default, causing new trace IDs generated

[dnsmichi]

Problem to solve

2 of my pipeline jobs always generated a new trace ID which looks pretty odd in Jaeger.

After identifying the pattern being the use of needs for asynchronous pipeline execution, I have found searched for needs requires artifacts because I remembered that needs explicitly requires to specify artifact dependencies or similar discussions.

Found

• https://forum.gitlab.com/t/why-needs-is-not-transitive-at-least-for-artifacts/76496
• https://gitlab.com/gitlab-org/gitlab/-/issues/389478
• https://docs.gitlab.com/ee/ci/yaml/#needsartifacts

Proposal

Document the workaround with needs:artifacts

generate-trace-id:
  stage: preparation
  before_script: []
  after_script: []
  script:
    - echo "main_trace_id=$(openssl rand -hex 16)" >> variables.env
    - echo "main_span_id=$(openssl rand -hex 8)" >> variables.env
    - echo "main_trace_start_time=$(date +%s)" >> variables.env
  artifacts:
    # Debug: Expose variables.env as artifact 
    paths:
      - "variables.env"
    reports:
      dotenv: variables.env              # Use artifacts:reports:dotenv to expose the variables to other jobs
  # Note: Asynchronous jobs with `needs` do not download artifacts from previous stages by default.
  # https://docs.gitlab.com/ee/ci/yaml/#needsartifacts
  # All jobs using needs will need to explicitly specify this job.
  # Example:
  # needs:
  #    - job: generate-trace-id
  #      artifacts: true 

test-cpp:
  stage: test
  # TODO: Verify if the heavy job needs to run for tests 
  needs:  
    - build-cpp
    - build-cpp-heavy
    # Require the tracepusher job artifacts explicitly 
    - job: generate-trace-id
      artifacts: true        
  script:
    - ci/test_cpp.sh

deploy-cloudnative:
  stage: deploy
# TODO: Why is the deploy-rhel8 dependency needed?
  needs: 
    - deploy-rhel8
    - build-go
    - build-cpp
    # Require the tracepusher job artifacts explicitly 
    - job: generate-trace-id
      artifacts: true           
  script:
    - ci/deploy/prepare.sh
    - ci/deploy/prepare_env.sh
    - ci/deploy/provision_env.sh
    - ci/deploy/deploy_app.sh
    - ci/deploy/observe_app.sh
    - ci/deploy/collect_profiling.sh

Tests in this MR https://gitlab.com/gitlab-de/use-cases/observability/devsecops-efficiency/slow-pipeline-for-analysis/-/merge_requests/1 and pipeline https://gitlab.com/gitlab-de/use-cases/observability/devsecops-efficiency/slow-pipeline-for-analysis/-/pipelines/920935030

[agardnerIT]

Am I right in thinking that, if the feeback for the self-contained binaries (see release candidate for v0.8.0) goes well, this will probably be redundant as we don't need any of it, just include the binary and call it?

[agardnerIT]

Since https://gitlab.com/gitlab-org/gitlab/-/issues/389478 is closed, I'm going to close this as well. Please re-open if it's still relevant.