Closed Onokaev closed 1 year ago
microsoft-github-policy-service[bot]
commented
1 year ago This issue requires additional investigation by the Accessibility Insights team. When the issue is ready to be triaged again, we will update the issue with the investigation result and add "status: ready for triage". Thank you for contributing to Accessibility Insights!
JGibson2019
commented
1 year ago Hi @Onokaev a couple of things:
Onokaev
commented
1 year ago Onokaev
commented
1 year ago Here is the yaml:
name: $(date:yyyyMMdd)$(rev:.r)
pool: 1es-windows-latest
pr: branches: include:
variables: isMaster: $[eq(variables['Build.SourceBranch'], 'refs/heads/master')] isDev: $[eq(variables['Build.SourceBranch'], 'refs/heads/dev')]
trigger: branches: include:
jobs:
job: One displayName: "Build and test validation" steps:
task: CredScan@3
task: ComponentGovernanceComponentDetection@0 inputs: scanType: "Register" verbosity: "Verbose" alertWarningLevel: "High" ignoreDirectories: ".github,.vscode/"
task: ComponentGovernanceComponentDetection@0 inputs: scanType: "Register" verbosity: "Verbose" alertWarningLevel: "High" ignoreDirectories: ".github,.vscode/"
task: NodeTool@0 inputs: versionSpec: "14.20.x" displayName: "Install Node.js"
script: | npm install displayName: "npm install"
script: | npm run lint displayName: "Runs linting checks"
script: | npm test displayName: "Runs Unit tests"
task: PublishTestResults@2 condition: succeededOrFailed() inputs: testResultsFormat: "JUnit" testResultsFiles: "test-report.xml" displayName: "Publish Test Results"
script: | npm run build displayName: "Run build"
job: Two displayName: "Run playwright tests" condition: eq(variables['isDev'], 'true') steps:
task: NodeTool@0 inputs: versionSpec: "14.20.x" displayName: "Install npm"
script: | npm install displayName: "npm install"
script: | npm install -D @playwright/test npx playwright install-deps displayName: "Install playwright dependancies"
script: | npx playwright install displayName: "Install playwright"
script: | npm run test-playwright-accessibility displayName: "Run accessibility tests"
script: | set CI=true npm run test-playwright
env: REACT_APP_CLIENT_ID: $(REACT_APP_PLAYWRIGHT_TEST_CLIENT_ID) REACT_APP_INSTRUMENTATION_KEY: $(REACT_APP_STAGING_INSTRUMENTATION_KEY) REACT_APP_FEEDBACK_CAMPAIGN_ID: $(REACT_APP_STAGING_FEEDBACK_CAMPAIGN_ID) REACT_APP_NPS_FEEDBACK_CAMPAIGN_ID: $(REACT_APP_NPS_FEEDBACK_CAMPAIGN_ID) REACT_APP_NOMINATION_PERIOD: $(REACT_APP_NOMINATION_PERIOD) REACT_APP_COOLDOWN_PERIOD: $(REACT_APP_COOLDOWN_PERIOD) REACT_APP_USAGE_TIME: $(REACT_APP_USAGE_TIME) PLAYWRIGHT_TESTS_BASE_URL: $(PLAYWRIGHT_TESTS_BASE_URL) PLAYWRIGHT_TESTS_USERNAME: $(PLAYWRIGHT_TESTS_USERNAME) PLAYWRIGHT_TESTS_PASSWORD: $(PLAYWRIGHT_TESTS_PASSWORD) displayName: "Run playwright tests"
publish: $(System.DefaultWorkingDirectory)/playwright-report artifact: playwright-report condition: always() displayName: "Publish test results"
job: Three displayName: "Publish artifacts" dependsOn: One condition: and(succeeded(), or(eq(variables['isMaster'], 'true'), eq(variables['isDev'], 'true'))) steps:
task: NodeTool@0 inputs: versionSpec: "14.20.x" displayName: "Install Node.js"
script: | npm install displayName: "npm install"
script: | npm run build env: REACT_APP_CLIENT_ID: $(REACT_APP_STAGING_CLIENT_ID) REACT_APP_INSTRUMENTATION_KEY: $(REACT_APP_STAGING_INSTRUMENTATION_KEY) REACT_APP_FEEDBACK_CAMPAIGN_ID: $(REACT_APP_STAGING_FEEDBACK_CAMPAIGN_ID) REACT_APP_NPS_FEEDBACK_CAMPAIGN_ID: $(REACT_APP_NPS_FEEDBACK_CAMPAIGN_ID) REACT_APP_NOMINATION_PERIOD: $(REACT_APP_NOMINATION_PERIOD) REACT_APP_COOLDOWN_PERIOD: $(REACT_APP_COOLDOWN_PERIOD) REACT_APP_USAGE_TIME: $(REACT_APP_USAGE_TIME) displayName: "Build static assets for staging"
task: PowerShell@2 displayName: "Set version-number" condition: and(succeeded(), eq(variables['isMaster'], 'true')) inputs: targetType: "inline" script: | $lernaPath = "$Env:BUILD_SOURCESDIRECTORY\package.json" $json = Get-Content "$lernaPath" | Out-String | ConvertFrom-Json $versionNumber = $json.version; Write-Host "Setting the build number variable to '$versionNumber'." Write-Host "##vso[build.updatebuildnumber]$versionNumber"
script: | npm run build condition: and(succeeded(), eq(variables['isMaster'], 'true')) env: REACT_APP_CLIENT_ID: $(REACT_APP_PROD_CLIENT_ID) REACT_APP_INSTRUMENTATION_KEY: $(REACT_APP_INSTRUMENTATION_KEY) REACT_APP_FEEDBACK_CAMPAIGN_ID: $(REACT_APP_FEEDBACK_CAMPAIGN_ID) REACT_APP_NPS_FEEDBACK_CAMPAIGN_ID: $(REACT_APP_NPS_FEEDBACK_CAMPAIGN_ID) REACT_APP_NOMINATION_PERIOD: $(REACT_APP_NOMINATION_PERIOD) REACT_APP_COOLDOWN_PERIOD: $(REACT_APP_COOLDOWN_PERIOD) REACT_APP_USAGE_TIME: $(REACT_APP_USAGE_TIME) displayName: "Build static assets for prod"
task: PowerShell@2 displayName: "Create _manifest directory" inputs: targetType: "inline" script: | $manifestPath = "$Env:SYSTEM_DEFAULTWORKINGDIRECTORY/_manifest" mkdir -p $manifestPath Write-Host "Created directory '$manifestPath'."
task: ManifestGeneratorTask@0 inputs: ManifestDirPath: "$(System.DefaultWorkingDirectory)/_manifest" BuildComponentPath: "$(System.DefaultWorkingDirectory)/build" PackageName: "graph-explorer-v2" PackageVersion: "$(Build.UpdateBuildNumber)"
task: CopyFiles@1 displayName: "Copy manifest files to artifact staging dir: manifest" inputs: SourceFolder: "$(System.DefaultWorkingDirectory)/_manifest" TargetFolder: "$(Build.ArtifactStagingDirectory)/manifest"
task: PublishBuildArtifacts@1 displayName: "Publish Artifact: manifest" inputs: PathtoPublish: "$(Build.ArtifactStagingDirectory)/manifest" ArtifactName: manifest
task: CopyFiles@1 displayName: "Copy files to: $(Build.ArtifactStagingDirectory)/build" inputs: SourceFolder: "$(System.DefaultWorkingDirectory)/build" TargetFolder: "$(Build.ArtifactStagingDirectory)/build"
task: PublishBuildArtifacts@1 displayName: "Publish Artifact: drop" inputs: PathtoPublish: "$(build.ArtifactStagingDirectory)/build" ArtifactName: drop
task: DeleteFiles@1 inputs: contents: node_modules displayName: "Delete node_modules"
job: Four displayName: Run Accessibility Insights tool steps:
task: NodeTool@0 displayName: "Use node 14" inputs: versionSpec: "14.20.x"
script: | npm install displayName: "Install dependencies"
script: | npm run build displayName: "Build app"
script: | npm install -g serve displayName: "Install serve"
script: | start /B serve -s build displayName: "Serve static files async" timeoutInMinutes: 25
task: accessibility-insights.prod.task.accessibility-insights@3 timeoutInMinutes: 25 inputs: url: "https://developer.microsoft.com/en-us/graph/graph-explorer" snapshot: true scanTimeout: 1200000
Onokaev
commented
1 year ago The accessibility job is the last one: job: Four
DaveTryon
commented
1 year ago @Onokaev, the error being thrown is coming from this line. This class wraps the scan of a single page. The value for AccessibilityScanOperation.axeScanTimeoutSec is hardcoded to 180 on line 17 and no plumbing currently exists to provide an override.
The extension's scanTimeout parameter influences the overall execution time, but has no impact on the hardcode per-page limitation. As a team, we'll discuss potential configuration options for the per-page timeout, then let you know where that lands.
In the meantime, you might be able to work around this by enabling the extension's singleWorker flag to minimize contention from scanning pages concurrently, or by breaking the job into multiple scans, potentially each starting with a different url under https://developer.microsoft.com/en-us/graph/graph-explorer.
microsoft-github-policy-service[bot]
commented
1 year ago This issue has been marked as ready for team triage; we will triage it in our weekly review and update the issue. Thank you for contributing to Accessibility Insights!
dbjorge
commented
1 year ago Hi @Onokaev , it looks like the page in question is embedding a copy of the Monaco editor. We think the issue you're encountering is a repro of a known performance issue in axe-core (the open source accessibility testing engine we're based on) which is triggered by an implementation detail of the Monaco editor, tracked by https://github.com/dequelabs/axe-core/issues/3923 and currently planned to be fixed in the next feature-level engine release. We're going to treat this issue as blocked pending that upstream fix.
DaveTryon
commented
1 year ago Update: https://github.com/dequelabs/axe-core/issues/3923 has been resolved and will be in axe-core 4.8. We don't yet have a timetable for consuming it.
DaveTryon
commented
1 year ago @nang4a11y, please add this to your notes for the future feature that consumes axe-core 4.8, so that we can test it and mention it in the release notes.
Your question here
Could this be related to the container issue that was raised before? I am running the tests using a self-hosted agent