search

intel / ai-containers

This repository contains Dockerfiles, scripts, yaml files, Helm charts, etc. used to scale out AI containers with versions of TensorFlow and PyTorch that have been optimized for Intel platforms. Scaling is done with python, Docker, kubernetes, kubeflow, cnvrg.io, Helm, and other container orchestration frameworks for use in the cloud and on-premise
https://intel.github.io/ai-containers/
Apache License 2.0
23 stars 17 forks source link

Test Reporter for Container CI #213

Closed cholocate closed 2 months ago

cholocate commented 3 months ago

Description

The following changes in the .yaml files worked with the previous forked repository, and will likely require some additional changes as we migrate to the new ai-containers format.

Changes Made

Validation

image

github-actions[bot] commented 3 months ago

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

PackageVersionScoreDetails
actions/actions/download-artifact fa0a91b85d4f404e444e00e005971372dc801d16 :green_circle: 6.6
Details
CheckScoreReason
Code-Review:green_circle: 10all changesets reviewed
Maintained:green_circle: 45 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 4
CII-Best-Practices:warning: 0no effort to earn an OpenSSF best practices badge detected
License:green_circle: 10license file detected
Signed-Releases:warning: -1no releases found
Branch-Protection:warning: -1internal error: error during branchesHandler.setup: internal error: githubv4.Query: Resource not accessible by integration
Dangerous-Workflow:green_circle: 10no dangerous workflow patterns detected
Packaging:warning: -1packaging workflow not detected
Binary-Artifacts:green_circle: 10no binaries found in the repo
Fuzzing:warning: 0project is not fuzzed
Token-Permissions:warning: 0detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies:warning: 1dependency not pinned by hash detected -- score normalized to 1
Security-Policy:green_circle: 9security policy file detected
SAST:green_circle: 10SAST tool is run on all commits
Vulnerabilities:green_circle: 100 existing vulnerabilities detected
actions/marocchino/sticky-pull-request-comment 331f8f5b4215f0445d3c07b4967662a32a2d3e31 :green_circle: 5.4
Details
CheckScoreReason
Maintained:green_circle: 1030 commit(s) and 2 issue activity found in the last 90 days -- score normalized to 10
Code-Review:warning: 2Found 1/5 approved changesets -- score normalized to 2
CII-Best-Practices:warning: 0no effort to earn an OpenSSF best practices badge detected
License:green_circle: 10license file detected
Signed-Releases:warning: -1no releases found
Packaging:warning: -1packaging workflow not detected
Dangerous-Workflow:green_circle: 10no dangerous workflow patterns detected
Binary-Artifacts:green_circle: 10no binaries found in the repo
Token-Permissions:green_circle: 9detected GitHub workflow tokens with excessive permissions
Branch-Protection:warning: 0branch protection not enabled on development/release branches
Pinned-Dependencies:warning: 0dependency not pinned by hash detected -- score normalized to 0
Fuzzing:warning: 0project is not fuzzed
Security-Policy:warning: 0security policy file not detected
SAST:warning: 0SAST tool is not run on all commits -- score normalized to 0
Vulnerabilities:green_circle: 100 existing vulnerabilities detected

Scanned Manifest Files

.github/workflows/integration-test.yaml
  • actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16
  • marocchino/sticky-pull-request-comment@331f8f5b4215f0445d3c07b4967662a32a2d3e31
github-advanced-security[bot] commented 3 months ago

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

cholocate commented 2 months ago

Test-Group Test Status python/tests import-core PASS python/tests perf-full PASS python/tests perf-core PASS python/tests perf-stock PASS python/tests1 import-core PASS python/tests1 perf-full PASS python/tests1 perf-core PASS python/tests1 perf-stock PASS

recreate: true is not deleting the previous comment, will investigate to see what additional input is necessary for the following fix. Also there is a destination mismatch, looking at the following log, we can see that all three were downloaded to the same directory:

Run actions/download-artifact@v4
  with:
    pattern: my-artifact*json
    merge-multiple: true
    repository: intel/ai-containers
    run-id: 9982280984
Found [1](https://github.com/intel/ai-containers/actions/runs/9982280984/job/27588636581?pr=213#step:2:1)0 artifact(s)
Filtering artifacts by pattern 'my-artifact*json'
Preparing to download the following artifacts:
- my-artifact classical-ml-tests json (REDACTED)
- my-artifact python-tests json (REDACTED)
- my-artifact python-tests1 json (REDACTED)
Redirecting to blob download url: ...
Starting download of artifact to: /home/runner/work/ai-containers/ai-containers
Redirecting to blob download url: ...
Starting download of artifact to: /home/runner/work/ai-containers/ai-containers
Redirecting to blob download url: ...

But when running ls . for the next job we see that there's only: 

Run ls .
tests-test-runner.json
tests1-test-runner.json

/home/runner/work/ai-containers/ai-containers looks fixed, will make the following fix by navigating to the path and see whether this is fixed.

cholocate commented 2 months ago

The most recent comment, running with recreate: true. Testing whether this comment is deleted or the one previously posted with the same header.

cholocate commented 2 months ago

Test-Group Test Status preset/data-analytics dataset-librarian-3.9 PASS preset/data-analytics dataset-librarian-3.10 PASS preset/data-analytics modin-3.9 PASS preset/data-analytics modin-3.10 PASS preset/data-analytics modin-notebook-3.9 PASS preset/data-analytics modin-notebook-3.10 PASS python/tests import-core PASS python/tests perf-full PASS python/tests perf-core PASS python/tests perf-stock PASS python/tests1 import-core PASS python/tests1 perf-full PASS python/tests1 perf-core PASS python/tests1 perf-stock PASS

Conclusion: Confirmed expected behavior from across triggered containers in group-diff job, gathering the summaries of all test-runner jobs and combined together successfully.

coveralls-official[bot] commented 2 months ago

Pull Request Test Coverage Report for Build 10114270103

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

Totals Coverage Status
Change from base Build 10114012271: 0.0%
Covered Lines: 146
Relevant Lines: 165
💛 - Coveralls
github-actions[bot] commented 2 months ago
Test-Group Test Status
python-tests-logs import-core PASS
python-tests-logs perf-full PASS
python-tests-logs perf-core PASS
python-tests-logs perf-stock PASS
cholocate commented 2 months ago

Resolved case where tests.yaml is not present when group-diff triggered the container. Expected behavior: Skip job. Validated (please see the following PR run) Also validated regular case where test-runner is supposed to run. Please see this PR run

Challenges and notes to self:

Make sure to always use [[ ]] when running bash conditional if statements, and also have matching quotes.