search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.9k stars 1.21k forks source link

do not expect usm event stream if unsupported #31462

Closed brycekahle closed 3 days ago

brycekahle commented 3 days ago

What does this PR do?

Fix config test when process event data stream is not supported

Motivation

Failing tests on main

Describe how to test/QA your changes

Fixed tests.

Possible Drawbacks / Trade-offs

Additional Notes

Added cmd/system-probe paths to trigger KMT. KMT tests did not run in #31420, which caused the original author to miss this.

chouquette commented 3 days ago

/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on

dd-devflow[bot] commented 3 days ago

Devflow running: /trigger-ci --variable RUN_ALL_BUILDS=true --varia...

View all feedbacks in Devflow UI.

2024-11-26 07:15:13 UTC :information_source: Gitlab pipeline started

Started pipeline #49935418

agent-platform-auto-pr[bot] commented 3 days ago

Gitlab CI Configuration Changes

Modified Jobs
.kmt_run_sysprobe_tests ```diff .kmt_run_sysprobe_tests: artifacts: expire_in: 2 weeks paths: - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz - $CI_PROJECT_DIR/logs - $CI_PROJECT_DIR/pcaps - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log reports: annotations: - $EXTERNAL_LINKS_PATH when: always before_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH - PLATFORMS_FOR_COMPLEXITY_COLLECTION="amazon_5.4 debian_10 ubuntu_18.04 centos_8 opensuse_15.3 suse_12.5 fedora_38" - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\ \ | grep -qw \"${TAG}\" ; then\n export COLLECT_COMPLEXITY=yes\nfi\n" - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}" retry: exit_codes: 42 max: 2 when: - job_execution_timeout - runner_system_failure - stuck_or_timeout_failure - unknown_failure - api_failure - scheduler_failure - stale_schedule - data_integrity_failure rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output) - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP" --output text --query $QUERY_INSTANCE_IDS | wc -l ) - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n echo \"These jobs do not permit\ \ retries. The go tests are retried a user-specified number of times automatically.\ \ In order to re-run the tests, you must trigger the pipeline again\"\n 'false'\n\ fi\n" - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET))) | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output) - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET))) | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output) - GO_VERSION=$(inv go-version) - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt" - ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt ${MICRO_VM_IP}:/job_env.txt" - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file /home/kernel-version-testing/ddvm_rsa -vm-cmd 'CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY} /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'" - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE -vm-cmd "${NESTED_VM_CMD}" -send-env-vars DD_API_KEY - ssh metal_instance "ssh ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'" - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log' stage: kernel_matrix_testing_system_probe timeout: 1h 30m variables: AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json RETRY: 2 TEST_COMPONENT: system-probe ```
.kmt_sysprobe_cleanup ```diff .kmt_sysprobe_cleanup: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job before_script: - GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN read_api) || exit $?; export GITLAB_TOKEN - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - inv kmt.wait-for-setup-job --pipeline-id $CI_PIPELINE_ID --arch $ARCH --component $TEST_COMPONENT - aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_PIPELINE $FILTER_ARCH $FILTER_INSTANCE_TYPE $FILTER_TEST_COMPONENT --output json --query $QUERY_INSTANCE_IDS | tee -a instance.json - INSTANCE_ID="$(jq -r '.[0][0]' < instance.json)" - echo ${INSTANCE_ID} - "if [[ \"${INSTANCE_ID}\" != \"\" ]] && [[ \"${INSTANCE_ID}\" != \"null\" ]];\ \ then\n aws ec2 terminate-instances --instance-ids \"${INSTANCE_ID}\"\nfi\n" stage: kernel_matrix_testing_cleanup tags: - arch:amd64 variables: TEST_COMPONENT: system-probe ```
.on_system_probe_or_e2e_changes_or_manual ```diff .on_system_probe_or_e2e_changes_or_manual: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* ```
.pull_test_dockers ```diff .pull_test_dockers: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job artifacts: expire_in: 1 day paths: - $KMT_DOCKERS image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES needs: [] rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - DOCKER_LOGIN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DOCKER_REGISTRY_RO user) || exit $? - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DOCKER_REGISTRY_RO token | crane auth login --username "$DOCKER_LOGIN" --password-stdin "$DOCKER_REGISTRY_URL" - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials needs gitlab runner restart"; exit $EXIT; fi - mkdir $KMT_DOCKERS - inv -e system-probe.save-test-dockers --use-crane --output-dir $KMT_DOCKERS --arch $ARCH stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: KMT_DOCKERS: $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH ```
.system_probe_change_paths ```diff .system_probe_change_paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* ```
.upload_sysprobe_tests ```diff .upload_sysprobe_tests: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job artifacts: paths: - $CI_PROJECT_DIR/connector-${ARCH} when: always before_script: - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache - rm -f modcache.tar.xz - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH - tar czvf $TEST_ARCHIVE_NAME opt - popd - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\ \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\ \ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\ \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\ ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\ \ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\ \ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\ \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n" - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/ - pushd $CI_PROJECT_DIR/test/new-e2e - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go - popd - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector stage: kernel_matrix_testing_prepare variables: AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz TEST_COMPONENT: system-probe ```
cleanup_kitchen_functional_test ```diff cleanup_kitchen_functional_test: allow_failure: true before_script: null dependencies: [] image: registry.ddbuild.io/ci/datadog-agent-buildimages/dd-agent-testing$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - cd $DD_AGENT_TESTING_DIR - tasks/clean.sh stage: functional_test_cleanup tags: - arch:amd64 variables: DD_PIPELINE_ID: $CI_PIPELINE_ID-fnct ```
kmt_run_sysprobe_tests_arm64 ```diff kmt_run_sysprobe_tests_arm64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET))) | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output) - mkdir -p $CI_PROJECT_DIR/logs - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" > $CI_PROJECT_DIR/logs/setup-ddvm.log || true - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\"" | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" || true - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/ || true - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" || true - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/ || true - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true - PLATFORMS_FOR_COMPLEXITY_COLLECTION="amazon_5.4 debian_10 ubuntu_18.04 centos_8 opensuse_15.3 suse_12.5 fedora_38" - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\ \ | grep -qw \"${TAG}\" ; then\n export COLLECT_COMPLEXITY=yes\nfi\n" - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}" - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n ssh metal_instance \"scp\ \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\ \ || true\n scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\ \ $DD_AGENT_TESTING_DIR/ || true\nfi\n" - inv -e kmt.tag-ci-job - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz" artifacts: expire_in: 2 weeks paths: - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz - $CI_PROJECT_DIR/logs - $CI_PROJECT_DIR/pcaps - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log reports: annotations: - $EXTERNAL_LINKS_PATH when: always before_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH - PLATFORMS_FOR_COMPLEXITY_COLLECTION="amazon_5.4 debian_10 ubuntu_18.04 centos_8 opensuse_15.3 suse_12.5 fedora_38" - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\ \ | grep -qw \"${TAG}\" ; then\n export COLLECT_COMPLEXITY=yes\nfi\n" - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}" image: registry.ddbuild.io/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES needs: - kmt_setup_env_sysprobe_arm64 - upload_dependencies_sysprobe_arm64 - upload_sysprobe_tests_arm64 - upload_minimized_btfs_sysprobe_arm64 parallel: matrix: - TAG: - ubuntu_18.04 - ubuntu_20.04 - ubuntu_22.04 - ubuntu_24.04 - amazon_4.14 - amazon_5.4 - amazon_5.10 - amazon_2023 - fedora_37 - fedora_38 - debian_10 - debian_11 - debian_12 - centos_7.9 - centos_8 TEST_SET: - only_usm - no_usm retry: exit_codes: - 42 max: 2 when: - job_execution_timeout - runner_system_failure - stuck_or_timeout_failure - unknown_failure - api_failure - scheduler_failure - stale_schedule - data_integrity_failure rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output) - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP" --output text --query $QUERY_INSTANCE_IDS | wc -l ) - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n echo \"These jobs do not permit\ \ retries. The go tests are retried a user-specified number of times automatically.\ \ In order to re-run the tests, you must trigger the pipeline again\"\n 'false'\n\ fi\n" - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET))) | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output) - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET))) | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output) - GO_VERSION=$(inv go-version) - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt" - ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt ${MICRO_VM_IP}:/job_env.txt" - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file /home/kernel-version-testing/ddvm_rsa -vm-cmd 'CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY} /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'" - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE -vm-cmd "${NESTED_VM_CMD}" -send-env-vars DD_API_KEY - ssh metal_instance "ssh ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'" - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log' stage: kernel_matrix_testing_system_probe tags: - arch:arm64 timeout: 1h 30m variables: ARCH: arm64 AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json RETRY: 2 TEST_COMPONENT: system-probe ```
kmt_run_sysprobe_tests_x64 ```diff kmt_run_sysprobe_tests_x64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - export MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET))) | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output) - mkdir -p $CI_PROJECT_DIR/logs - ssh metal_instance "ssh ${MICRO_VM_IP} \"journalctl -u setup-ddvm.service\"" > $CI_PROJECT_DIR/logs/setup-ddvm.log || true - cat $CI_PROJECT_DIR/logs/setup-ddvm.log || true - ssh metal_instance "ssh ${MICRO_VM_IP} \"systemctl is-active setup-ddvm.service\"" | tee $CI_PROJECT_DIR/logs/setup-ddvm.status || true - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/junit.tar.gz /home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" || true - scp "metal_instance:/home/ubuntu/junit-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/ || true - ssh metal_instance "scp ${MICRO_VM_IP}:/ci-visibility/testjson.tar.gz /home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" || true - scp "metal_instance:/home/ubuntu/testjson-${ARCH}-${TAG}-${TEST_SET}.tar.gz" $DD_AGENT_TESTING_DIR/ || true - ssh metal_instance "scp -r ${MICRO_VM_IP}:/tmp/test_pcaps /home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true - mkdir -p "$CI_PROJECT_DIR/pcaps" && scp -r "metal_instance:/home/ubuntu/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" "$CI_PROJECT_DIR/pcaps/test_pcaps-${ARCH}-${TAG}-${TEST_SET}" || true - PLATFORMS_FOR_COMPLEXITY_COLLECTION="amazon_5.4 debian_10 ubuntu_18.04 centos_8 opensuse_15.3 suse_12.5 fedora_38" - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\ \ | grep -qw \"${TAG}\" ; then\n export COLLECT_COMPLEXITY=yes\nfi\n" - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}" - "if [ \"${COLLECT_COMPLEXITY}\" = \"yes\" ]; then\n ssh metal_instance \"scp\ \ ${MICRO_VM_IP}:/verifier-complexity.tar.gz /home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\ \ || true\n scp \"metal_instance:/home/ubuntu/verifier-complexity-${ARCH}-${TAG}-${TEST_COMPONENT}.tar.gz\"\ \ $DD_AGENT_TESTING_DIR/ || true\nfi\n" - inv -e kmt.tag-ci-job - $CI_PROJECT_DIR/tools/ci/junit_upload.sh "$DD_AGENT_TESTING_DIR/junit-*.tar.gz" artifacts: expire_in: 2 weeks paths: - $DD_AGENT_TESTING_DIR/junit-$ARCH-$TAG-$TEST_SET.tar.gz - $DD_AGENT_TESTING_DIR/testjson-$ARCH-$TAG-$TEST_SET.tar.gz - $DD_AGENT_TESTING_DIR/verifier-complexity-$ARCH-$TAG-${TEST_COMPONENT}.tar.gz - $CI_PROJECT_DIR/logs - $CI_PROJECT_DIR/pcaps - $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log reports: annotations: - $EXTERNAL_LINKS_PATH when: always before_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE - echo "CI_JOB_URL=${CI_JOB_URL}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_ID=${CI_JOB_ID}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_NAME=${CI_JOB_NAME}" >> $DD_AGENT_TESTING_DIR/job_env.txt - echo "CI_JOB_STAGE=${CI_JOB_STAGE}" >> $DD_AGENT_TESTING_DIR/job_env.txt - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH - PLATFORMS_FOR_COMPLEXITY_COLLECTION="amazon_5.4 debian_10 ubuntu_18.04 centos_8 opensuse_15.3 suse_12.5 fedora_38" - "if [ \"${TEST_SET}\" = \"no_usm\" ] && echo \"${PLATFORMS_FOR_COMPLEXITY_COLLECTION}\"\ \ | grep -qw \"${TAG}\" ; then\n export COLLECT_COMPLEXITY=yes\nfi\n" - echo "COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY}" image: registry.ddbuild.io/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES needs: - kmt_setup_env_sysprobe_x64 - upload_dependencies_sysprobe_x64 - upload_sysprobe_tests_x64 - upload_minimized_btfs_sysprobe_x64 parallel: matrix: - TAG: - ubuntu_16.04 - ubuntu_18.04 - ubuntu_20.04 - ubuntu_22.04 - ubuntu_24.04 - amazon_4.14 - amazon_5.4 - amazon_5.10 - amazon_2023 - fedora_37 - fedora_38 - debian_9 - debian_10 - debian_11 - debian_12 - centos_7.9 - centos_8 TEST_SET: - only_usm - no_usm retry: exit_codes: - 42 max: 2 when: - job_execution_timeout - runner_system_failure - stuck_or_timeout_failure - unknown_failure - api_failure - scheduler_failure - stale_schedule - data_integrity_failure rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - INSTANCE_IP=$(jq --exit-status --arg ARCH $ARCH -r '.[$ARCH].ip' $CI_PROJECT_DIR/stack.output) - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - RUNNING_INSTANCES=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_PIPELINE $FILTER_TEST_COMPONENT "Name=private-ip-address,Values=$INSTANCE_IP" --output text --query $QUERY_INSTANCE_IDS | wc -l ) - "if [ $RUNNING_INSTANCES -eq \"0\" ]; then\n echo \"These jobs do not permit\ \ retries. The go tests are retried a user-specified number of times automatically.\ \ In order to re-run the tests, you must trigger the pipeline again\"\n 'false'\n\ fi\n" - MICRO_VM_IP=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET))) | map(select(.tag==$TAG)) | .[].ip' $CI_PROJECT_DIR/stack.output) - MICRO_VM_NAME=$(jq --exit-status --arg TAG $TAG --arg ARCH $ARCH --arg TEST_SET $TEST_SET -r '.[$ARCH].microvms | map(select(."vmset-tags"| index($TEST_SET))) | map(select(.tag==$TAG)) | .[].id' $CI_PROJECT_DIR/stack.output) - GO_VERSION=$(inv go-version) - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - scp "$DD_AGENT_TESTING_DIR/job_env.txt" "metal_instance:/home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt" - ssh metal_instance "scp /home/ubuntu/job_env-${ARCH}-${TAG}-${TEST_SET}.txt ${MICRO_VM_IP}:/job_env.txt" - NESTED_VM_CMD="/home/ubuntu/connector -host ${MICRO_VM_IP} -user root -ssh-file /home/kernel-version-testing/ddvm_rsa -vm-cmd 'CI=true /root/fetch_dependencies.sh ${ARCH} && COLLECT_COMPLEXITY=${COLLECT_COMPLEXITY} /opt/micro-vm-init.sh -test-tools /opt/testing-tools -retry ${RETRY} -test-root /opt/${TEST_COMPONENT}-tests -packages-run-config /opt/${TEST_SET}.json'" - $CI_PROJECT_DIR/connector-$ARCH -host $INSTANCE_IP -user ubuntu -ssh-file $AWS_EC2_SSH_KEY_FILE -vm-cmd "${NESTED_VM_CMD}" -send-env-vars DD_API_KEY - ssh metal_instance "ssh ${MICRO_VM_IP} '/opt/testing-tools/test-json-review -flakes /opt/testing-tools/flakes.yaml -codeowners /opt/testing-tools/CODEOWNERS -test-root /opt/${TEST_COMPONENT}-tests'" - '[ ! -f $CI_PROJECT_DIR/daemon-${ARCH}.log ] && scp metal_instance:/home/ubuntu/daemon.log $CI_PROJECT_DIR/vm-metrics-daemon-${ARCH}.log' stage: kernel_matrix_testing_system_probe tags: - arch:amd64 timeout: 1h 30m variables: ARCH: x86_64 AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json RETRY: 2 TEST_COMPONENT: system-probe ```
kmt_setup_env_sysprobe_arm64 ```diff kmt_setup_env_sysprobe_arm64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - export AWS_PROFILE=agent-qa-ci - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu $CI_PROJECT_DIR/libvirt/dnsmasq - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP" "sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml '{}' > /tmp/ddvm-xml-'{}'.txt\"" - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP" "sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log /tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\"" - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP" "mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo chown 1000:1000 /tmp/dnsmasq/*" - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log" $CI_PROJECT_DIR/libvirt/log - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*" $CI_PROJECT_DIR/libvirt/xml - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log" $CI_PROJECT_DIR/libvirt/qemu - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*" $CI_PROJECT_DIR/libvirt/dnsmasq - "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n GO_ARCH=amd64\nfi\n" - cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics "ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics" - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP" "/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro --tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\" --daemon -log-file /home/ubuntu/daemon.log" - inv -e kmt.tag-ci-job artifacts: paths: - $CI_PROJECT_DIR/stack.output - $CI_PROJECT_DIR/libvirt - $VMCONFIG_FILE reports: annotations: - $EXTERNAL_LINKS_PATH when: always before_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache - rm -f modcache.tar.xz - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH || true image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES needs: - go_deps - go_tools_deps rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" > $STACK_DIR - pulumi login $(cat $STACK_DIR | tr -d '\n') - inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --vmconfig-template=$TEST_COMPONENT --memory=12288 - inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE $INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE --infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID} --run-agent - jq "." $CI_PROJECT_DIR/stack.output - pulumi logout stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: AMI_ID_ARG: --arm-ami-id=$KERNEL_MATRIX_TESTING_ARM_AMI_ID ARCH: arm64 AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key AWS_EC2_SSH_KEY_NAME: datadog-agent-ci AWS_REGION: us-east-1 EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json INFRA_ENV: aws/agent-qa INSTANCE_TYPE: m6gd.metal INSTANCE_TYPE_ARG: --instance-type-arm=$INSTANCE_TYPE KITCHEN_EC2_REGION: us-east-1 KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7 KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811 KUBERNETES_MEMORY_LIMIT: 16Gi KUBERNETES_MEMORY_REQUEST: 12Gi LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-arm PIPELINE_ID: $CI_PIPELINE_ID RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME} STACK_DIR: $CI_PROJECT_DIR/stack.dir TEAM: ebpf-platform TEST_COMPONENT: system-probe VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json ```
kmt_setup_env_sysprobe_x64 ```diff kmt_setup_env_sysprobe_x64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - export AWS_PROFILE=agent-qa-ci - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu $CI_PROJECT_DIR/libvirt/dnsmasq - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP" "sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml '{}' > /tmp/ddvm-xml-'{}'.txt\"" - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP" "sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log /tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\"" - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP" "mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo chown 1000:1000 /tmp/dnsmasq/*" - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log" $CI_PROJECT_DIR/libvirt/log - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*" $CI_PROJECT_DIR/libvirt/xml - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log" $CI_PROJECT_DIR/libvirt/qemu - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*" $CI_PROJECT_DIR/libvirt/dnsmasq - "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n GO_ARCH=amd64\nfi\n" - cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go - scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics "ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics" - ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP" "/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro --tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\" --daemon -log-file /home/ubuntu/daemon.log" - inv -e kmt.tag-ci-job artifacts: paths: - $CI_PROJECT_DIR/stack.output - $CI_PROJECT_DIR/libvirt - $VMCONFIG_FILE reports: annotations: - $EXTERNAL_LINKS_PATH when: always before_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache - rm -f modcache.tar.xz - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE - inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH || true image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES needs: - go_deps - go_tools_deps rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" > $STACK_DIR - pulumi login $(cat $STACK_DIR | tr -d '\n') - inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --vmconfig-template=$TEST_COMPONENT --memory=12288 - inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE $INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE --infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID} --run-agent - jq "." $CI_PROJECT_DIR/stack.output - pulumi logout stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: AMI_ID_ARG: --x86-ami-id=$KERNEL_MATRIX_TESTING_X86_AMI_ID ARCH: x86_64 AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key AWS_EC2_SSH_KEY_NAME: datadog-agent-ci AWS_REGION: us-east-1 EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json INFRA_ENV: aws/agent-qa INSTANCE_TYPE: m5d.metal INSTANCE_TYPE_ARG: --instance-type-x86=$INSTANCE_TYPE KITCHEN_EC2_REGION: us-east-1 KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7 KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811 KUBERNETES_MEMORY_LIMIT: 16Gi KUBERNETES_MEMORY_REQUEST: 12Gi LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-x86 PIPELINE_ID: $CI_PIPELINE_ID RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME} STACK_DIR: $CI_PROJECT_DIR/stack.dir TEAM: ebpf-platform TEST_COMPONENT: system-probe VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json ```
kmt_sysprobe_cleanup_arm64 ```diff kmt_sysprobe_cleanup_arm64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job before_script: - GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN read_api) || exit $?; export GITLAB_TOKEN - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES needs: - kmt_setup_env_sysprobe_arm64 - kmt_run_sysprobe_tests_arm64 - upload_minimized_btfs_sysprobe_arm64 - upload_sysprobe_tests_arm64 - upload_dependencies_sysprobe_arm64 rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - inv kmt.wait-for-setup-job --pipeline-id $CI_PIPELINE_ID --arch $ARCH --component $TEST_COMPONENT - aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_PIPELINE $FILTER_ARCH $FILTER_INSTANCE_TYPE $FILTER_TEST_COMPONENT --output json --query $QUERY_INSTANCE_IDS | tee -a instance.json - INSTANCE_ID="$(jq -r '.[0][0]' < instance.json)" - echo ${INSTANCE_ID} - "if [[ \"${INSTANCE_ID}\" != \"\" ]] && [[ \"${INSTANCE_ID}\" != \"null\" ]];\ \ then\n aws ec2 terminate-instances --instance-ids \"${INSTANCE_ID}\"\nfi\n" stage: kernel_matrix_testing_cleanup tags: - arch:amd64 variables: ARCH: arm64 INSTANCE_TYPE: m6gd.metal TEST_COMPONENT: system-probe when: always ```
kmt_sysprobe_cleanup_arm64_manual ```diff kmt_sysprobe_cleanup_arm64_manual: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job allow_failure: true before_script: - GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN read_api) || exit $?; export GITLAB_TOKEN - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES needs: [] rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - inv kmt.wait-for-setup-job --pipeline-id $CI_PIPELINE_ID --arch $ARCH --component $TEST_COMPONENT - aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_PIPELINE $FILTER_ARCH $FILTER_INSTANCE_TYPE $FILTER_TEST_COMPONENT --output json --query $QUERY_INSTANCE_IDS | tee -a instance.json - INSTANCE_ID="$(jq -r '.[0][0]' < instance.json)" - echo ${INSTANCE_ID} - "if [[ \"${INSTANCE_ID}\" != \"\" ]] && [[ \"${INSTANCE_ID}\" != \"null\" ]];\ \ then\n aws ec2 terminate-instances --instance-ids \"${INSTANCE_ID}\"\nfi\n" stage: kernel_matrix_testing_cleanup tags: - arch:amd64 variables: ARCH: arm64 INSTANCE_TYPE: m6gd.metal TEST_COMPONENT: system-probe when: manual ```
kmt_sysprobe_cleanup_x64 ```diff kmt_sysprobe_cleanup_x64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job before_script: - GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN read_api) || exit $?; export GITLAB_TOKEN - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES needs: - kmt_setup_env_sysprobe_x64 - kmt_run_sysprobe_tests_x64 - upload_minimized_btfs_sysprobe_x64 - upload_sysprobe_tests_x64 - upload_dependencies_sysprobe_x64 rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - inv kmt.wait-for-setup-job --pipeline-id $CI_PIPELINE_ID --arch $ARCH --component $TEST_COMPONENT - aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_PIPELINE $FILTER_ARCH $FILTER_INSTANCE_TYPE $FILTER_TEST_COMPONENT --output json --query $QUERY_INSTANCE_IDS | tee -a instance.json - INSTANCE_ID="$(jq -r '.[0][0]' < instance.json)" - echo ${INSTANCE_ID} - "if [[ \"${INSTANCE_ID}\" != \"\" ]] && [[ \"${INSTANCE_ID}\" != \"null\" ]];\ \ then\n aws ec2 terminate-instances --instance-ids \"${INSTANCE_ID}\"\nfi\n" stage: kernel_matrix_testing_cleanup tags: - arch:amd64 variables: ARCH: x86_64 INSTANCE_TYPE: m5d.metal TEST_COMPONENT: system-probe when: always ```
kmt_sysprobe_cleanup_x64_manual ```diff kmt_sysprobe_cleanup_x64_manual: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job allow_failure: true before_script: - GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN read_api) || exit $?; export GITLAB_TOKEN - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci image: registry.ddbuild.io/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES needs: [] rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - inv kmt.wait-for-setup-job --pipeline-id $CI_PIPELINE_ID --arch $ARCH --component $TEST_COMPONENT - aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_PIPELINE $FILTER_ARCH $FILTER_INSTANCE_TYPE $FILTER_TEST_COMPONENT --output json --query $QUERY_INSTANCE_IDS | tee -a instance.json - INSTANCE_ID="$(jq -r '.[0][0]' < instance.json)" - echo ${INSTANCE_ID} - "if [[ \"${INSTANCE_ID}\" != \"\" ]] && [[ \"${INSTANCE_ID}\" != \"null\" ]];\ \ then\n aws ec2 terminate-instances --instance-ids \"${INSTANCE_ID}\"\nfi\n" stage: kernel_matrix_testing_cleanup tags: - arch:amd64 variables: ARCH: x86_64 INSTANCE_TYPE: m5d.metal TEST_COMPONENT: system-probe when: manual ```
notify_ebpf_complexity_changes ```diff notify_ebpf_complexity_changes: allow_failure: true before_script: - python3 -m pip install -r tasks/kernel_matrix_testing/requirements-ci.txt --break-system-packages - python3 -m pip install -r tasks/libs/requirements-github.txt --break-system-packages - GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_GITHUB_APP key_b64) || exit $?; export GITHUB_KEY_B64 - GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_GITHUB_APP app_id) || exit $?; export GITHUB_APP_ID - GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_GITHUB_APP installation_id) || exit $?; export GITHUB_INSTALLATION_ID - echo "Using agent GitHub App" - GITLAB_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $GITLAB_TOKEN read_api) || exit $?; export GITLAB_TOKEN image: registry.ddbuild.io/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES needs: - job: kmt_run_sysprobe_tests_x64 optional: true parallel: matrix: - TAG: - amazon_5.4 - debian_10 - ubuntu_18.04 - centos_8 - opensuse_15.3 - suse_12.5 - fedora_38 TEST_SET: no_usm - job: kmt_run_sysprobe_tests_arm64 optional: true parallel: matrix: - TAG: - amazon_5.4 - debian_10 - ubuntu_18.04 - centos_8 - opensuse_15.3 - suse_12.5 - fedora_38 TEST_SET: no_usm rules: - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $CI_COMMIT_BRANCH == "main" when: never - if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/ when: never - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $DEPLOY_AGENT == "false" && $DDR_WORKFLOW_ID == null && $RUN_E2E_TESTS == "off" when: never - if: $DEPLOY_AGENT == "true" || $DDR_WORKFLOW_ID != null when: never - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* - allow_failure: true if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: - pkg/ebpf/**/* - pkg/security/**/* - pkg/eventmonitor/**/* - .gitlab/kernel_matrix_testing/security_agent.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/tests/cws/**/* - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/security_agent.py - tasks/kmt.py - tasks/kernel_matrix_testing/* - allow_failure: true when: manual script: - inv -e ebpf.generate-complexity-summary-for-pr stage: notify tags: - arch:amd64 ```
pull_test_dockers_arm64 ```diff pull_test_dockers_arm64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job artifacts: expire_in: 1 day paths: - $KMT_DOCKERS image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES needs: [] rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - DOCKER_LOGIN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DOCKER_REGISTRY_RO user) || exit $? - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DOCKER_REGISTRY_RO token | crane auth login --username "$DOCKER_LOGIN" --password-stdin "$DOCKER_REGISTRY_URL" - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials needs gitlab runner restart"; exit $EXIT; fi - mkdir $KMT_DOCKERS - inv -e system-probe.save-test-dockers --use-crane --output-dir $KMT_DOCKERS --arch $ARCH stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: ARCH: arm64 KMT_DOCKERS: $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH ```
pull_test_dockers_x64 ```diff pull_test_dockers_x64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job artifacts: expire_in: 1 day paths: - $KMT_DOCKERS image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES needs: [] rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - DOCKER_LOGIN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DOCKER_REGISTRY_RO user) || exit $? - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DOCKER_REGISTRY_RO token | crane auth login --username "$DOCKER_LOGIN" --password-stdin "$DOCKER_REGISTRY_URL" - EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials needs gitlab runner restart"; exit $EXIT; fi - mkdir $KMT_DOCKERS - inv -e system-probe.save-test-dockers --use-crane --output-dir $KMT_DOCKERS --arch $ARCH stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: ARCH: x86_64 KMT_DOCKERS: $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH ```
upload_dependencies_sysprobe_arm64 ```diff upload_dependencies_sysprobe_arm64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job before_script: - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE image: registry.ddbuild.io/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES needs: - pull_test_dockers_arm64 - go_deps - go_tools_deps rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\ \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\ \ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\ \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\ ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\ \ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\ \ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\ \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n" - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - "if [ -d $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH ]; then\n tar czvf $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH.tar.gz\ \ -C $DD_AGENT_TESTING_DIR kmt-dockers-$ARCH\n scp $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH.tar.gz\ \ metal_instance:/opt/kernel-version-testing\nfi\n" stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: ARCH: arm64 AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key INSTANCE_TYPE: m6gd.metal TEST_COMPONENT: system-probe ```
upload_dependencies_sysprobe_x64 ```diff upload_dependencies_sysprobe_x64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job before_script: - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE image: registry.ddbuild.io/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES needs: - pull_test_dockers_x64 - go_deps - go_tools_deps rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\ \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\ \ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\ \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\ ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\ \ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\ \ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\ \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n" - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - "if [ -d $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH ]; then\n tar czvf $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH.tar.gz\ \ -C $DD_AGENT_TESTING_DIR kmt-dockers-$ARCH\n scp $DD_AGENT_TESTING_DIR/kmt-dockers-$ARCH.tar.gz\ \ metal_instance:/opt/kernel-version-testing\nfi\n" stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: ARCH: x86_64 AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key INSTANCE_TYPE: m5d.metal TEST_COMPONENT: system-probe ```
upload_minimized_btfs_sysprobe_arm64 ```diff upload_minimized_btfs_sysprobe_arm64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job image: registry.ddbuild.io/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES needs: - generate_minimized_btfs_arm64 rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - mkdir -p $DEPENDENCIES - pushd $DEPENDENCIES - mkdir -p $BTF_DIR - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz $BTF_DIR/minimized-btfs.tar.xz - popd - ls -la $DEPENDENCIES - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH - tar czvf $ARCHIVE_NAME btfs - popd - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\ \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\ \ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\ \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\ ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\ \ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\ \ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\ \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n" - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/ stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: ARCH: arm64 ARCHIVE_NAME: btfs-arm64.tar.gz AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key BTF_DIR: opt/kmt-ramfs/system-probe-tests/pkg/ebpf/bytecode/build/${ARCH}/co-re/btf DEPENDENCIES: $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/btfs INSTANCE_TYPE: m6gd.metal TEST_COMPONENT: system-probe ```
upload_minimized_btfs_sysprobe_x64 ```diff upload_minimized_btfs_sysprobe_x64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job image: registry.ddbuild.io/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES needs: - generate_minimized_btfs_x64 rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - mkdir -p $DEPENDENCIES - pushd $DEPENDENCIES - mkdir -p $BTF_DIR - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz $BTF_DIR/minimized-btfs.tar.xz - popd - ls -la $DEPENDENCIES - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH - tar czvf $ARCHIVE_NAME btfs - popd - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\ \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\ \ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\ \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\ ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\ \ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\ \ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\ \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n" - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/ stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: ARCH: x86_64 ARCHIVE_NAME: btfs-x86_64.tar.gz AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key BTF_DIR: opt/kmt-ramfs/system-probe-tests/pkg/ebpf/bytecode/build/${ARCH}/co-re/btf DEPENDENCIES: $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/btfs INSTANCE_TYPE: m5d.metal TEST_COMPONENT: system-probe ```
upload_sysprobe_tests_arm64 ```diff upload_sysprobe_tests_arm64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job artifacts: paths: - $CI_PROJECT_DIR/connector-${ARCH} when: always before_script: - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache - rm -f modcache.tar.xz - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE image: registry.ddbuild.io/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES needs: - go_deps - prepare_sysprobe_ebpf_functional_tests_arm64 - tests_ebpf_arm64 rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH - tar czvf $TEST_ARCHIVE_NAME opt - popd - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\ \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\ \ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\ \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\ ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\ \ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\ \ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\ \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n" - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/ - pushd $CI_PROJECT_DIR/test/new-e2e - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go - popd - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector stage: kernel_matrix_testing_prepare tags: - arch:arm64 variables: ARCH: arm64 AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key INSTANCE_TYPE: m6gd.metal TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz TEST_COMPONENT: system-probe ```
upload_sysprobe_tests_x64 ```diff upload_sysprobe_tests_x64: after_script: - DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_API_KEY_ORG2 token) || exit $?; export DD_API_KEY - inv -e kmt.tag-ci-job artifacts: paths: - $CI_PROJECT_DIR/connector-${ARCH} when: always before_script: - mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache - rm -f modcache.tar.xz - mkdir -p ~/.aws - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E profile >> ~/.aws/config || exit $? - export AWS_PROFILE=agent-qa-ci - touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE - $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_E2E ssh_key > $AWS_EC2_SSH_KEY_FILE || exit $? - echo "" >> $AWS_EC2_SSH_KEY_FILE - chmod 600 $AWS_EC2_SSH_KEY_FILE image: registry.ddbuild.io/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES needs: - go_deps - prepare_sysprobe_ebpf_functional_tests_x64 - tests_ebpf_x64 rules: - if: $CI_COMMIT_BRANCH == "main" - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/ when: never - if: $RUN_KMT_TESTS == 'on' - changes: compare_to: main paths: + - cmd/system-probe/**/* - pkg/collector/corechecks/ebpf/**/* - pkg/collector/corechecks/servicediscovery/module/* - pkg/ebpf/**/* - pkg/network/**/* - pkg/process/monitor/* - pkg/util/kernel/**/* - pkg/dynamicinstrumentation/**/* - pkg/gpu/**/* - .gitlab/kernel_matrix_testing/system_probe.yml - .gitlab/kernel_matrix_testing/common.yml - .gitlab/source_test/ebpf.yml - test/new-e2e/system-probe/**/* - test/new-e2e/scenarios/system-probe/**/* - test/new-e2e/pkg/runner/**/* - test/new-e2e/pkg/utils/**/* - test/new-e2e/go.mod - tasks/system_probe.py - tasks/kmt.py - tasks/kernel_matrix_testing/* script: - pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH - tar czvf $TEST_ARCHIVE_NAME opt - popd - FILTER_TEAM="Name=tag:team,Values=ebpf-platform" - FILTER_MANAGED="Name=tag:managed-by,Values=pulumi" - FILTER_STATE="Name=instance-state-name,Values=running" - FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}" - FILTER_ARCH="Name=tag:arch,Values=${ARCH}" - FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}" - FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}" - QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId' - QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress' - "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\ \ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\ \ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\ \ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\ ; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\ \ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\ \ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\ \ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\ \ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\ \ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n" - INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output text --query $QUERY_PRIVATE_IPS) - echo "$ARCH-instance-ip" $INSTANCE_IP - mkdir -p ~/.ssh && chmod 700 ~/.ssh - echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config - chmod 600 ~/.ssh/config - scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/ - pushd $CI_PROJECT_DIR/test/new-e2e - go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go - popd - scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector stage: kernel_matrix_testing_prepare tags: - arch:amd64 variables: ARCH: x86_64 AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key INSTANCE_TYPE: m5d.metal TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz TEST_COMPONENT: system-probe ```

Changes Summary

Removed Modified Added Renamed
0 24 0 0

:information_source: Diff available in the job log.

cit-pr-commenter[bot] commented 3 days ago

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: e9ec519e-40de-42d3-b00b-0cb5ff4df966

Baseline: 88703d8e3b462deac3f889d4e8efac46567d0e7e Comparison: 84668634e8c7c98fb6cc84771b8026ed15431bf1 Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links | |------|----------------------------------------------|--------------------|----------|----------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ➖ | basic_py_check | % cpu utilization | +4.11 | [+0.14, +8.09] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Abasic_py_check%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | quality_gate_idle_all_features | memory utilization | +0.62 | [+0.52, +0.72] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle_all_features%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle_all_features&tpl_var_job_id%5B0%5D=e9ec519e-40de-42d3-b00b-0cb5ff4df966&tpl_var_run-id%5B0%5D=e9ec519e-40de-42d3-b00b-0cb5ff4df966&view=spans&from_ts=1732607131000&to_ts=1732607731000&live=false) | | ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | +0.36 | [-0.38, +1.09] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api_cpu%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | +0.23 | [-0.23, +0.69] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency_linear_load%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | tcp_syslog_to_blackhole | ingress throughput | +0.11 | [+0.04, +0.18] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_syslog_to_blackhole%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.04 | [-0.73, +0.81] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_0ms_latency%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.11, +0.11] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Auds_dogstatsd_to_api%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.01, +0.01] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Atcp_dd_logs_filter_exclude%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | file_to_blackhole_500ms_latency | egress throughput | -0.01 | [-0.78, +0.77] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_500ms_latency%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | file_to_blackhole_100ms_latency | egress throughput | -0.01 | [-0.76, +0.73] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_100ms_latency%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | file_to_blackhole_300ms_latency | egress throughput | -0.02 | [-0.65, +0.61] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_300ms_latency%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | otel_to_otel_logs | ingress throughput | -0.30 | [-0.97, +0.37] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aotel_to_otel_logs%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | file_tree | memory utilization | -0.34 | [-0.48, -0.20] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_tree%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.36 | [-1.14, +0.42] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Afile_to_blackhole_1000ms_latency%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) | | ➖ | quality_gate_idle | memory utilization | -0.76 | [-0.80, -0.71] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Aquality_gate_idle%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) [bounds checks dashboard](https://app.datadoghq.com/dashboard/vz3-jd5-bdi?fromUser=true&refresh_mode=paused&tpl_var_experiment%5B0%5D=quality_gate_idle&tpl_var_job_id%5B0%5D=e9ec519e-40de-42d3-b00b-0cb5ff4df966&tpl_var_run-id%5B0%5D=e9ec519e-40de-42d3-b00b-0cb5ff4df966&view=spans&from_ts=1732607131000&to_ts=1732607731000&live=false) | | ➖ | pycheck_lots_of_tags | % cpu utilization | -1.92 | [-5.34, +1.51] | 1 | [Logs](https://app.datadoghq.com/logs?query=experiment%3Apycheck_lots_of_tags%20run_id%3Ae9ec519e-40de-42d3-b00b-0cb5ff4df966&agg_m=count&agg_m_source=base&agg_q=%40span.url&agg_q_source=base&agg_t=count&fromUser=true&index=single-machine-performance-target-logs&messageDisplay=inline&refresh_mode=paused&storage=hot&stream_sort=time%2Cdesc&top_n=100&top_o=top&viz=stream&x_missing=true&from_ts=1732599931000&to_ts=1732611331000&live=false) |

Bounds Checks: ❌ Failed

perf experiment bounds_check_name replicates_passed links
❌ file_to_blackhole_300ms_latency lost_bytes 9/10
✅ file_to_blackhole_0ms_latency lost_bytes 10/10
✅ file_to_blackhole_0ms_latency memory_usage 10/10
✅ file_to_blackhole_1000ms_latency memory_usage 10/10
✅ file_to_blackhole_1000ms_latency_linear_load memory_usage 10/10
✅ file_to_blackhole_100ms_latency lost_bytes 10/10
✅ file_to_blackhole_100ms_latency memory_usage 10/10
✅ file_to_blackhole_300ms_latency memory_usage 10/10
✅ file_to_blackhole_500ms_latency lost_bytes 10/10
✅ file_to_blackhole_500ms_latency memory_usage 10/10
✅ quality_gate_idle memory_usage 10/10 bounds checks dashboard
✅ quality_gate_idle_all_features memory_usage 10/10 bounds checks dashboard

Explanation

**Confidence level:** 90.00% **Effect size tolerance:** |Δ mean %| ≥ 5.00% Performance changes are noted in the **perf** column of each table: * ✅ = significantly better comparison variant performance * ❌ = significantly worse comparison variant performance * ➖ = no significant change in performance A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI". For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true: 1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look. 2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that *if our statistical model is accurate*, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants. 3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.