Megalinter doesn't work with Azure Devops automated PR build runs

karl18 commented 3 weeks ago

Hi @nvuillam I run megalinter in Azure Devops. Everything looks fine except 3 questions\ issues:

When an automated PR runs a build (automatic run when PR is opened against main branch for e.g.), megalinter doesn't work at all and fails with the error which i pasted below.

However, If I ran the PR pipeline manually it works fine and this very strange to me !

Maybe a good check would be to move to LOG_LEVEL: DEBUG and see what are the difference between automated PRs run and manually triggered env vars passed to megalinter when the script runs! The only main\ obvious difference i see is (according to Diffchecker) between both env vars on both pipelines are:

2024-08-21T09:20:10.1486100Z BUILD_SOURCEBRANCH=refs/pull/60549/merge 2024-08-21T09:20:10.1486616Z BUILD_SOURCEBRANCHNAME=merge

on a manually triggered one I see:

2024-08-21T12:10:17.9246612Z BUILD_SOURCEBRANCH=refs/heads/test-megalinter-pipeline-ref 2024-08-21T12:10:17.9247552Z BUILD_SOURCEBRANCHNAME=test-megalinter-pipeline-ref

not sure if these could affect the megalinter.py script to run as expected as BUILD_SOURCEBRANCHNAME could be a wrong input or something similar??

Not sure that VALIDATE_ALL_CODE: false ever worked! although there are no errors shown in the pipeline, but its always set to true because I always see the lines below for all runs:

Listing updated files in [/tmp/lint] using git diff.
Unable to list updated files from git diff. Switch to VALIDATE_ALL_CODE_BASE=true
Listing all files in directory [/tmp/lint], then filter with:
- File extensions: .bash, .dash, .htm, .html, .json, .json5, .jsonc, .ksh, .markdown, .md, .mdx, .py, .rst, .sh, .txt, .yaml, .yml
- File names (regex): Containerfile, Dockerfile
- Excluding regex: ^(fp-cicd-shared-scripts/.*$)|^(fp-cicd-shared-pipelines/.*$)
Unable to list git ignored files (/tmp/lint)

this is whether its set to true or false in either "docker run cmd" or in the "megalinter.yaml" config file itself. I tried with several fetchDepth:0 & 1 & 2 for both self repo but nothing helped!

For FILTER_REGEX_EXCLUDE: | .*fp-cicd-shared-scripts/.* I want to exclude this folder "fp-cicd-shared-scripts" from ALL linters that might be used currently or in the furture, is it necessary to define it for each and every linter the regex exclude? or just use FILTER_REGEX_EXCLUDE: is sufficient for all linters? If yes, do i define the regex correctly?

Also last thing worth mentioning, that i do some merge for the linter configuration with ruamel.yamland at the end I get some file with the name: .merged-megalinter.yaml`which has the end results\ full merge for all linters to run along with the developers desired linters and some framework linters for the configuration file, but thats different story as it doesn't affects the pipeline, it works fine apart from the automated PR builds which the issue is about!

I really appreciate your look and further investigation! Sorry for pasting long configs & thanks in advance! :)

here's my configuration ".megalinter.yaml":

  VALIDATE_ALL_CODEBASE: true # Enable codebase scans
  FILTER_REGEX_EXCLUDE: |
    .*fp-cicd-shared-scripts/.*
  REPOSITORY_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  DOCKERFILE_HADOLINT_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  KUBERNETES_HELM_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  YAML_PRETTIER_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  YAML_YAMLLINT_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  YAML_V8R_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  JSON_JSONLINT_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  JSON_PRETTIER_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  BASH_EXEC_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  REPOSITORY_GITLEAKS_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  REPOSITORY_GRYPE_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  REPOSITORY_SECRETLINT_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  REPOSITORY_TRIVY_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  REPOSITORY_TRUFFLEHOG_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  SPELL_LYCHEE_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  PYTHON_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  PYTHON_RUFF_FILTER_REGEX_EXCLUDE: .*fp-cicd-shared-scripts/.*
  REPOSITORY_GITLEAKS_CONFIG_FILE: .gitleaks.toml
  REPOSITORY_TRIVY_ARGUMENTS: ["--skip-dirs", "./fp-cicd-shared-scripts", "--skip-dirs", "./fp-cicd-shared-pipelines/"]
  REPOSITORY_TRIVY_CONFIG_FILE: trivy.yaml
  REPOSITORY_TRUFFLEHOG_ARGUMENTS: "--exclude-paths=.trufflehogignore"
  REPOSITORY_CHECKOV_CONFIG_FILE: .checkov.yml
  REPOSITORY_GRYPE_CONFIG_FILE: .grype.yaml
  YAML_YAMLLINT_CONFIG_FILE: .yamllint.yml
  #YAML_V8R_CONFIG_FILE: .v8rrc.yml
  YAML_V8R_CONFIG_FILE: LINTER_DEFAULT
  YAML_PRETTIER_CONFIG_FILE: .prettierrc.yaml
  #JSON_PRETTIER_ARGUMENTS: ["--log-level", "debug"] # https://prettier.io/docs/en/cli#--list-different
  JSON_PRETTIER_CONFIG_FILE: .prettierrc.json
  REPOSITORY_GITLEAKS_PR_COMMITS_SCAN: true # enable for only pr commit scan and not all codebase
  PARALLEL: false
  PRINT_ALPACA: true
  SHOW_ELAPSED_TIME: true
  FLAVOR_SUGGESTIONS: false
  EMAIL_REPORTER: false
  FILEIO_REPORTER: false
  JSON_REPORTER: true
  GITHUB_STATUS_REPORTER: false
  AZURE_COMMENT_REPORTER: true
  PRINT_ALL_FILES: true
  MARKDOWN_DEFAULT_STYLE: markdownlint
  POST_COMMANDS:
    - command: echo "Make sure to Keep your Code Clean every PR ! :)"
      cwd: "workspace"   # Will be run at the root of the workspace (usually your repository root)
      continue_if_failed: True  # Will stop the process if command is failed (return code > 0)
  MEGALINTER_CONFIG: .megalinter-merged.yaml
  DEFAULT_BRANCH: main # Usually master or main
  LOG_LEVEL: DEBUG
  LINTER_RULES_PATH: .
  ENABLE_LINTERS:
    - DOCKERFILE_HADOLINT
    - KUBERNETES_HELM
    - YAML_YAMLLINT
    - JSON_JSONLINT
    - BASH_EXEC
    - REPOSITORY_GITLEAKS
    - REPOSITORY_GRYPE
    - REPOSITORY_SECRETLINT
    - REPOSITORY_TRIVY
    - REPOSITORY_TRUFFLEHOG
  DISABLE_ERRORS_LINTERS: None

here's my pipeline configuration "pr_docker_build.yml":

parameters:
  ...

stages:
  - stage: Linter
    displayName: Linter
    condition: eq('${{ parameters.enableLinters }}', 'true')
    pool: "Azure Pool - Linux - West Europe"
    dependsOn: []
    jobs:
      # MegaLinter Job for merge linter Config files & enable linting
      - job: MergeAndRunMegaLinterConfigs
        displayName: 'Merge And Run MegaLinter Configurations'
        continueOnError: true
        condition: eq('${{ parameters.enableLinters }}', 'true')
        steps:
        - script: |
            echo "Changing permissions for files and directories..."
            sudo find $(System.DefaultWorkingDirectory) -type d -exec chmod 755 {} \;
            sudo find $(System.DefaultWorkingDirectory) -type f -exec chmod 644 {} \;
            sudo chown -R $(id -u):$(id -g) $(System.DefaultWorkingDirectory)
            echo "Permissions changed."
          displayName: 'Change file permissions'

        - checkout: self
          fetchDepth: 0
          clean: true
          persistCredentials: true
          displayName: 'Git Checkout Self'

        # Call merge pipeline template
        - template: templates/ci_helm_merge_megalinter_configs.yml
          parameters:
            framework: ${{ parameters.framework }}
            enableLinters: ${{ parameters.enableLinters }}

        # MegaLinter docker run + checking exit code
        - script: |
            echo "Framework parameter value: ${{ parameters.framework }}"
            set +e
            docker run \
              -v $(System.DefaultWorkingDirectory):/tmp/lint \
              --env-file <(env | grep -e SYSTEM_ -e BUILD_ -e TF_ -e AGENT_) \
              -e SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
              -e GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \
              -e MEGALINTER_CONFIG=${{ parameters.mergedMegalinterFile }}\
              oxsecurity/megalinter:latest
            linter_exit_code=$?
            set -e
            if [ $linter_exit_code -ne 0 ]; then
              echo "##vso[task.complete result=SucceededWithIssues;]"
              echo "##vso[task.logissue type=warning]Linter found issues. Check the logs for details."
            fi
          displayName: 'Lint PR Changed Files'

        # Publish the Anchore report as an artifact to Azure Pipelines
        - task: PublishPipelineArtifact@1
          displayName: 'Publish Artifact: MegaLinter Report'
          condition: succeededOrFailed()
          inputs:
            targetPath: "$(System.DefaultWorkingDirectory)/megalinter-reports/"
            artifactName: MegaLinterReport

this pipeline runs a template to be able to merge the linter files as mentioned earlier, here is that file ci_helm_merge_megalinter_configs.yml:

parameters:
...

steps:
  # Checkout Shared Scripts repo
  - ${{ if eq(parameters.enableLinters, 'true') }}:
    - checkout: sharedScripts
      fetchDepth: 0
      clean: true
      persistCredentials: true
      displayName: 'Git Checkout sharedScripts' # Only when (enableLinters == true)

  # Create megalinter-reports directory
  - script: |
      mkdir -p $(System.DefaultWorkingDirectory)/megalinter-reports
    displayName: 'Create MegaLinter Reports Directory'

  # Adjust Ownership and Permissions
  - script: |
      sudo find $(System.DefaultWorkingDirectory) -type d -exec chmod 755 {} \;
      sudo find $(System.DefaultWorkingDirectory) -type f -exec chmod 644 {} \;
      sudo chown -R $(id -u):$(id -g) $(System.DefaultWorkingDirectory)
    displayName: 'Adjust Ownership and Permissions'

  - task: UsePythonVersion@0
    displayName: Use Python 3.8
    inputs:
        versionSpec: '3.8'
        addToPath: true
        architecture: 'x64'

  # install ruamel.yaml + HiYaPyCo
  - script: pip3 install -r fp-cicd-shared-scripts/Python/requirements.txt
    displayName: 'Install requirements'

  # Run merge script
  - task: PythonScript@0
    displayName: 'Merge MegaLinter Configs'
    inputs:
      scriptSource: 'filePath'
      scriptPath: 'fp-cicd-shared-scripts/Python/merge_megalinter_configs.py'
      failOnStderr: true

pipeline error (ONLY for PR automated builds) [check the vars i talked about earlier]):

2024-08-22T12:54:36.7362878Z ##[section]Starting: Lint PR Changed Files
2024-08-22T12:54:36.7367574Z ==============================================================================
2024-08-22T12:54:36.7367697Z Task         : Command line
2024-08-22T12:54:36.7367789Z Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
2024-08-22T12:54:36.7367903Z Version      : 2.237.1
2024-08-22T12:54:36.7367962Z Author       : Microsoft Corporation
2024-08-22T12:54:36.7368036Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
2024-08-22T12:54:36.7368175Z ==============================================================================
2024-08-22T12:54:38.9247992Z Generating script.
2024-08-22T12:54:38.9257691Z ========================== Starting Command Output ===========================
2024-08-22T12:54:38.9276215Z [command]/usr/bin/bash --noprofile --norc /agent/_work/_temp/9a292b66-a4b4-4421-8569-7e4dae723cfe.sh
2024-08-22T12:54:38.9323924Z Framework parameter value: net
2024-08-22T12:54:52.2296072Z Skipped setting git safe.directory DEFAULT_WORKSPACE:  ...
2024-08-22T12:54:52.2296556Z Setting git safe.directory default: /github/workspace ...
2024-08-22T12:54:53.1573668Z Setting git safe.directory to /tmp/lint ...
2024-08-22T12:54:53.1595187Z [MegaLinter init] ONE-SHOT RUN
2024-08-22T12:55:00.7924310Z [config] /tmp/lint/.megalinter-merged.yaml + Environment variables
2024-08-22T12:55:00.7926000Z 
2024-08-22T12:55:00.7926764Z     .:oool'                                  ,looo;                           
2024-08-22T12:55:00.7927155Z     .xNXNXl                                 .dXNNXo.                          
2024-08-22T12:55:00.7927743Z      lXXXX0c.                              'oKXXN0;                           
2024-08-22T12:55:00.7928213Z      .oKNXNX0kxdddddddoc,.    .;lodddddddxk0XXXX0c                            
2024-08-22T12:55:00.7928669Z       .:kKXXXXXXXXXXXXNXX0dllx0XXXXXXXXXXXXXXXKd,                             
2024-08-22T12:55:00.7929186Z         .,cdkOOOOOOOO0KXXXXXXXXXXK0OOOOOOOkxo:'                               
2024-08-22T12:55:00.7929859Z                       'ckKXNNNXkc'                                            
2024-08-22T12:55:00.7930525Z               ':::::;.  .c0XX0l.  .;::::;.                                    
2024-08-22T12:55:00.7931189Z               'xXXXXXx'   :kx:   ;OXXXXKd.                                    
2024-08-22T12:55:00.7931626Z                .dKNNXXO;   ..   :0XXXXKl.                                     
2024-08-22T12:55:00.7931987Z                 .lKXXXX0:     .lKXXXX0:                                       
2024-08-22T12:55:00.7932417Z                   :0XXXXKl.  .dXXXXXk,                                        
2024-08-22T12:55:00.7933085Z                    ;kXXXXKd:cxXXXXXx'                                         
2024-08-22T12:55:00.7933657Z                     'xXNXXXXXXXXXKo.                                          
2024-08-22T12:55:00.7934463Z                      .oKXXXXNXXX0l.                                           
2024-08-22T12:55:00.7934866Z                       .lKNNXNNXO:                                             
2024-08-22T12:55:00.7935466Z                         ,looool'                                              
2024-08-22T12:55:00.7935613Z 
2024-08-22T12:55:00.7935812Z ==========================================================
2024-08-22T12:55:00.7936056Z =============   MegaLinter, by OX.security   =============
2024-08-22T12:55:00.7936274Z =========  https://ox.security?ref=megalinter  ===========
2024-08-22T12:55:00.7936523Z ==========================================================
2024-08-22T12:55:00.7936626Z 
2024-08-22T12:55:00.7937000Z ----------------------------------------------------------------------------------------------------
2024-08-22T12:55:00.7937461Z ------------------------------------ MegaLinter, by OX Security ------------------------------------
2024-08-22T12:55:00.7937906Z ----------------------------------------------------------------------------------------------------
2024-08-22T12:55:00.7938768Z  - Image Creation Date: 2024-01-21T10:56:17Z
2024-08-22T12:55:00.7939131Z  - Image Revision: 688bc7466d7ab4faa83d614c2e6f9acf42b674dc
2024-08-22T12:55:00.7939453Z  - Image Version: v7.8.0
2024-08-22T12:55:00.7939845Z ----------------------------------------------------------------------------------------------------
2024-08-22T12:55:00.7940124Z The MegaLinter documentation can be found at:
2024-08-22T12:55:00.7940372Z  - https://megalinter.io/7.8.0
2024-08-22T12:55:00.7940766Z ----------------------------------------------------------------------------------------------------
2024-08-22T12:55:00.7947139Z ##[group]MegaLinter initialization (expand for details)
2024-08-22T12:55:00.7947579Z MegaLinter will analyze workspace [/tmp/lint]
2024-08-22T12:55:00.7954613Z AGENT_ACCEPTTEEEULA=True
2024-08-22T12:55:00.7955130Z AGENT_BUILDDIRECTORY=/agent/_work/1
2024-08-22T12:55:00.7955509Z AGENT_CLOUDID=
2024-08-22T12:55:00.7960782Z AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN=true
2024-08-22T12:55:00.7961372Z AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN=true
2024-08-22T12:55:00.7961834Z AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE=true
2024-08-22T12:55:00.7962258Z AGENT_HOMEDIRECTORY=/agent
2024-08-22T12:55:00.7962650Z AGENT_ID=14649
2024-08-22T12:55:00.7963003Z AGENT_ISSELFHOSTED=1
2024-08-22T12:55:00.7963311Z AGENT_JOBNAME=Merge And Run MegaLinter Configurations
2024-08-22T12:55:00.7963670Z AGENT_JOBSTATUS=Succeeded
2024-08-22T12:55:00.7963957Z AGENT_LOGTOBLOBSTORAGESERVICE=true
2024-08-22T12:55:00.7966858Z AGENT_MACHINENAME=devops-agent-linux-we0002ZD
2024-08-22T12:55:00.7967640Z AGENT_NAME=devops-agent-linux-we0002ZD
2024-08-22T12:55:00.7967988Z AGENT_OS=Linux
2024-08-22T12:55:00.7968364Z AGENT_OSARCHITECTURE=X64
2024-08-22T12:55:00.7968858Z AGENT_READONLYVARIABLES=true
2024-08-22T12:55:00.7969242Z AGENT_RETAINDEFAULTENCODING=false
2024-08-22T12:55:00.7969707Z AGENT_ROOTDIRECTORY=/agent/_work
2024-08-22T12:55:00.7970134Z AGENT_TASKRESTRICTIONSENFORCEMENTMODE=Enabled
2024-08-22T12:55:00.7970392Z AGENT_TEMPDIRECTORY=/agent/_work/_temp
2024-08-22T12:55:00.7970728Z AGENT_TOOLSDIRECTORY=/agent/_work/_tool
2024-08-22T12:55:00.7971057Z AGENT_USEWORKSPACEID=true
2024-08-22T12:55:00.7971390Z AGENT_USE_FETCH_FILTER_IN_CHECKOUT_TASK=true
2024-08-22T12:55:00.7971707Z AGENT_VERSION=3.243.0
2024-08-22T12:55:00.7971984Z AGENT_WORKFOLDER=/agent/_work
2024-08-22T12:55:00.7972291Z APPLY_FIXES=none
2024-08-22T12:55:00.7972884Z ARM_TTK_PSD1=/opt/microsoft/arm-ttk-master/arm-ttk/arm-ttk.psd1
2024-08-22T12:55:00.7973327Z AZP_AGENT_CHECK_FOR_TASK_DEPRECATION=true
2024-08-22T12:55:00.7975870Z AZP_AGENT_IGNORE_VSTSTASKLIB=true
2024-08-22T12:55:00.7976217Z AZP_AGENT_LOG_TASKNAME_IN_USERAGENT=true
2024-08-22T12:55:00.7976468Z AZP_AGENT_MOUNT_WORKSPACE=true
2024-08-22T12:55:00.7976744Z AZURE_COMMENT_REPORTER=True
2024-08-22T12:55:00.7977450Z AZURE_GUEST_AGENT_EXTENSION_PATH=/var/lib/waagent/Microsoft.VisualStudio.Services.TeamServicesAgentLinux-1.23.0.1
2024-08-22T12:55:00.7978198Z AZURE_GUEST_AGENT_EXTENSION_SUPPORTED_FEATURES=[{"Key": "ExtensionTelemetryPipeline", "Value": "1.0"}]
2024-08-22T12:55:00.7978946Z AZURE_GUEST_AGENT_EXTENSION_VERSION=1.23.0.1
2024-08-22T12:55:00.7979334Z AZURE_GUEST_AGENT_UNINSTALL_CMD_EXIT_CODE=NOT_RUN
2024-08-22T12:55:00.7979671Z AZURE_GUEST_AGENT_WIRE_PROTOCOL_ADDRESS=168.63.129.16
2024-08-22T12:55:00.7980298Z BASH_EXEC_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.7980670Z BUILD_ARTIFACTSTAGINGDIRECTORY=/agent/_work/1/a
2024-08-22T12:55:00.7980931Z BUILD_BINARIESDIRECTORY=/agent/_work/1/b
2024-08-22T12:55:00.7981206Z BUILD_BUILDID=221343
2024-08-22T12:55:00.7981666Z BUILD_BUILDNUMBER=0.1.31-PullRequest60549.7
2024-08-22T12:55:00.7982049Z BUILD_BUILDURI=vstfs:///Build/Build/221343
2024-08-22T12:55:00.7982370Z BUILD_CONTAINERID=33682780
2024-08-22T12:55:00.7982831Z BUILD_DATE=2024-01-21T10:56:17Z
2024-08-22T12:55:00.7983131Z BUILD_DEFINITIONFOLDERPATH=\
2024-08-22T12:55:00.7983584Z BUILD_DEFINITIONNAME=AMC-C4MODEL-SUPERVISOR-API - PR
2024-08-22T12:55:00.7983869Z BUILD_DEFINITIONVERSION=5
2024-08-22T12:55:00.7984438Z BUILD_QUEUEDBY=Microsoft.VisualStudio.Services.TFS
2024-08-22T12:55:00.7985064Z BUILD_QUEUEDBYID=00000002-0000-8888-8000-000000000000
2024-08-22T12:55:00.7985360Z BUILD_REASON=PullRequest
2024-08-22T12:55:00.7985649Z BUILD_REPOSITORY_CLEAN=True
2024-08-22T12:55:00.7986087Z BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT=False
2024-08-22T12:55:00.7986646Z BUILD_REPOSITORY_ID=44881d33-92f2-4835-9cdf-1fb808012f88
2024-08-22T12:55:00.7987018Z BUILD_REPOSITORY_LOCALPATH=/agent/_work/1/s
2024-08-22T12:55:00.7987488Z BUILD_REPOSITORY_NAME=amc-c4model-supervisor-api
2024-08-22T12:55:00.7987790Z BUILD_REPOSITORY_PROVIDER=TfsGit
2024-08-22T12:55:00.7988438Z BUILD_REPOSITORY_URI=https://FlaschenpostSE@dev.azure.com/FlaschenpostSE/AMC/_git/amc-c4model-supervisor-api
2024-08-22T12:55:00.7988858Z BUILD_REQUESTEDFOR=Kareem Mostafa
2024-08-22T12:55:00.7989180Z BUILD_REQUESTEDFOREMAIL=Kareem.Mostafa@flaschenpost.de
2024-08-22T12:55:00.7989734Z BUILD_REQUESTEDFORID=3e12393c-897f-65ae-8e42-9e483edfe1cf
2024-08-22T12:55:00.7990254Z BUILD_REVISION=688bc7466d7ab4faa83d614c2e6f9acf42b674dc
2024-08-22T12:55:00.7990672Z BUILD_SOURCEBRANCH=refs/pull/60549/merge
2024-08-22T12:55:00.7990939Z BUILD_SOURCEBRANCHNAME=merge
2024-08-22T12:55:00.7991227Z BUILD_SOURCESDIRECTORY=/agent/_work/1/s
2024-08-22T12:55:00.7991558Z BUILD_SOURCEVERSION=3041a532350c4e80fff6b3fccba0ca9629909854
2024-08-22T12:55:00.7991919Z BUILD_SOURCEVERSIONAUTHOR=Kareem Mostafa
2024-08-22T12:55:00.7992608Z BUILD_SOURCEVERSIONMESSAGE=Merge pull request 60549 from test-megalinter-pipeline-ref into main
2024-08-22T12:55:00.7993026Z BUILD_STAGINGDIRECTORY=/agent/_work/1/a
2024-08-22T12:55:00.7993441Z BUILD_VERSION=v7.8.0
2024-08-22T12:55:00.7993810Z CONFIG_SOURCE=TEMPORARY VAL THAT SHOULD NOT REMAIN
2024-08-22T12:55:00.7994174Z DEFAULT_BRANCH=main
2024-08-22T12:55:00.7994399Z DISABLE_ERRORS_LINTERS=None
2024-08-22T12:55:00.7994872Z DISABLE_LINTERS=['COPYPASTE_JSCPD', 'CSHARP_DOTNET_FORMAT']
2024-08-22T12:55:00.7995320Z DISTRIBUTEDTASK_AGENT_AGENTENABLEPIPELINEARTIFACTLARGECHUNKSIZE=True
2024-08-22T12:55:00.7995743Z DISTRIBUTEDTASK_AGENT_CONTINUEAFTERCANCELPROCESSTREEKILLATTEMPT=True
2024-08-22T12:55:00.7996129Z DISTRIBUTEDTASK_AGENT_DOCKERACTIONRETRIES=True
2024-08-22T12:55:00.7996493Z DISTRIBUTEDTASK_AGENT_ENABLEADDITIONALMASKINGREGEXES=True
2024-08-22T12:55:00.7996883Z DISTRIBUTEDTASK_AGENT_ENABLEISSUESOURCEVALIDATION=True
2024-08-22T12:55:00.7997348Z DISTRIBUTEDTASK_AGENT_ENABLERESOURCEUTILIZATIONWARNINGS=False
2024-08-22T12:55:00.7997815Z DISTRIBUTEDTASK_AGENT_FAILDEPRECATEDBUILDTASK=True
2024-08-22T12:55:00.7998160Z DISTRIBUTEDTASK_AGENT_FAILDEPRECATEDTASK=True
2024-08-22T12:55:00.7998512Z DISTRIBUTEDTASK_AGENT_FAILJOBWHENAGENTDIES=True
2024-08-22T12:55:00.7998999Z DISTRIBUTEDTASK_AGENT_FIXPOSSIBLEGITOUTOFMEMORYPROBLEM=False
2024-08-22T12:55:00.7999509Z DISTRIBUTEDTASK_AGENT_FORCEUPDATETOLATEST2VERSION=False
2024-08-22T12:55:00.8000090Z DISTRIBUTEDTASK_AGENT_IGNOREVSTSTASKLIB=True
2024-08-22T12:55:00.8000466Z DISTRIBUTEDTASK_AGENT_LOGTASKNAMEINUSERAGENT=True
2024-08-22T12:55:00.8000826Z DISTRIBUTEDTASK_AGENT_LOGTOBLOBSTORAGESERVICE=True
2024-08-22T12:55:00.8001153Z DISTRIBUTEDTASK_AGENT_MOUNTWORKSPACE=True
2024-08-22T12:55:00.8001499Z DISTRIBUTEDTASK_AGENT_READONLYVARIABLES=True
2024-08-22T12:55:00.8001865Z DISTRIBUTEDTASK_AGENT_ROSETTA2WARNING=True
2024-08-22T12:55:00.8002310Z DISTRIBUTEDTASK_AGENT_USEDOCKERCOMPOSEV2COMPATIBLEMODE=False
2024-08-22T12:55:00.8002756Z DISTRIBUTEDTASK_AGENT_USEFETCHFILTERINCHECKOUTTASK=True
2024-08-22T12:55:00.8003088Z DISTRIBUTEDTASK_AGENT_USEGITLONGPATHS=True
2024-08-22T12:55:00.8003431Z DISTRIBUTEDTASK_AGENT_USELATESTGITVERSION=True
2024-08-22T12:55:00.8003913Z DISTRIBUTEDTASK_AGENT_USEMSALLIBRARY=True
2024-08-22T12:55:00.8004301Z DISTRIBUTEDTASK_AGENT_USEMSDEPLOYTOKENAUTH=True
2024-08-22T12:55:00.8004672Z DISTRIBUTEDTASK_AGENT_USENEWNODEHANDLERTELEMETRY=True
2024-08-22T12:55:00.8005057Z DISTRIBUTEDTASK_AGENT_USEWORKSPACEID=True
2024-08-22T12:55:00.8005690Z DOCKERFILE_HADOLINT_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8006252Z EMAIL_REPORTER=False
2024-08-22T12:55:00.8007378Z ENABLE_LINTERS=['DOCKERFILE_HADOLINT', 'KUBERNETES_HELM', 'YAML_YAMLLINT', 'JSON_JSONLINT', 'BASH_EXEC', 'REPOSITORY_GITLEAKS', 'REPOSITORY_GRYPE', 'REPOSITORY_SECRETLINT', 'REPOSITORY_TRIVY', 'REPOSITORY_TRUFFLEHOG']
2024-08-22T12:55:00.8007961Z FAIL_DEPRECATED_BUILD_TASK=true
2024-08-22T12:55:00.8008303Z FAIL_JOB_WHEN_AGENT_DIES=true
2024-08-22T12:55:00.8008673Z FILEIO_REPORTER=False
2024-08-22T12:55:00.8009179Z FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8009367Z 
2024-08-22T12:55:00.8009682Z FLAVOR_SUGGESTIONS=False
2024-08-22T12:55:00.8009972Z GITHUB_STATUS_REPORTER=False
2024-08-22T12:55:00.8010299Z GIT_AUTHORIZATION_BEARER=HIDDEN_BY_MEGALINTER
2024-08-22T12:55:00.8010590Z GOPATH=/go
2024-08-22T12:55:00.8010833Z GOROOT=/usr/lib/go
2024-08-22T12:55:00.8011125Z GPG_KEY=7169605F62C751356D054A26A821E680E5FA6305
2024-08-22T12:55:00.8011450Z HOME=/root
2024-08-22T12:55:00.8011773Z HOSTNAME=9596f1251b0e
2024-08-22T12:55:00.8012303Z JAVA_HOME=/usr/lib/jvm/java-17-openjdk
2024-08-22T12:55:00.8012819Z JSON_JSONLINT_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8013268Z JSON_PRETTIER_CONFIG_FILE=.prettierrc.json
2024-08-22T12:55:00.8013902Z JSON_PRETTIER_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8014219Z JSON_REPORTER=True
2024-08-22T12:55:00.8014553Z KICS_LIBRARIES_PATH=/usr/bin/assets/libraries
2024-08-22T12:55:00.8014858Z KICS_QUERIES_PATH=/usr/bin/assets/queries
2024-08-22T12:55:00.8015460Z KUBERNETES_HELM_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8015779Z LANG=C.UTF-8
2024-08-22T12:55:00.8016010Z LINTER_RULES_PATH=.
2024-08-22T12:55:00.8016297Z LOG_LEVEL=DEBUG
2024-08-22T12:55:00.8016630Z MARKDOWN_DEFAULT_STYLE=markdownlint
2024-08-22T12:55:00.8017172Z MEGALINTER_CONFIG=.megalinter-merged.yaml
2024-08-22T12:55:00.8017489Z MEGALINTER_FLAVOR=all
2024-08-22T12:55:00.8017707Z MYPY_CACHE_DIR=/tmp
2024-08-22T12:55:00.8017978Z NODE_ENV=production
2024-08-22T12:55:00.8022974Z NODE_OPTIONS=--max-old-space-size=8192
2024-08-22T12:55:00.8024779Z NODE_PATH=/node-deps/node_modules
2024-08-22T12:55:00.8025091Z PARALLEL=False
2024-08-22T12:55:00.8028590Z PATH=~/.raku/bin:/opt/rakudo-pkg/bin:/opt/rakudo-pkg/share/perl6/site/bin:/usr/lib/jvm/java-17-openjdk/bin:/node-deps/node_modules/.bin:/root/.cargo/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/go/bin:/go/bin:/venvs/ansible-lint/bin:/venvs/cpplint/bin:/venvs/cfn-lint/bin:/venvs/djlint/bin:/venvs/pylint/bin:/venvs/black/bin:/venvs/flake8/bin:/venvs/isort/bin:/venvs/bandit/bin:/venvs/mypy/bin:/venvs/pyright/bin:/venvs/ruff/bin:/venvs/checkov/bin:/venvs/rst-lint/bin:/venvs/rstcheck/bin:/venvs/rstfmt/bin:/venvs/snakemake/bin:/venvs/snakefmt/bin:/venvs/proselint/bin:/venvs/sqlfluff/bin:/venvs/yamllint/bin:/root/.dotnet/tools
2024-08-22T12:55:00.8033056Z POST_COMMANDS=[{'command': 'echo "Make sure to Keep your Code Clean every PR ! :)"', 'continue_if_failed': True, 'cwd': 'workspace'}]
2024-08-22T12:55:00.8033539Z PRINT_ALL_FILES=True
2024-08-22T12:55:00.8033798Z PRINT_ALPACA=True
2024-08-22T12:55:00.8035260Z PWD=/
2024-08-22T12:55:00.8035474Z PYTHONPATH=:/
2024-08-22T12:55:00.8035968Z PYTHON_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8036398Z PYTHON_GET_PIP_SHA256=7cfd4bdc4d475ea971f1c0710a5953bcc704d171f83c797b9529d9974502fcc6
2024-08-22T12:55:00.8038370Z PYTHON_GET_PIP_URL=https://github.com/pypa/get-pip/raw/049c52c665e8c5fd1751f942316e0a5c777d304f/public/get-pip.py
2024-08-22T12:55:00.8038808Z PYTHON_PIP_VERSION=23.2.1
2024-08-22T12:55:00.8039280Z PYTHON_RUFF_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8040771Z PYTHON_VERSION=3.12.1
2024-08-22T12:55:00.8042303Z REPOSITORY_CHECKOV_CONFIG_FILE=.checkov.yml
2024-08-22T12:55:00.8042841Z REPOSITORY_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8043226Z REPOSITORY_GITLEAKS_CONFIG_FILE=.gitleaks.toml
2024-08-22T12:55:00.8045217Z REPOSITORY_GITLEAKS_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8045847Z REPOSITORY_GITLEAKS_PR_COMMITS_SCAN=True
2024-08-22T12:55:00.8046129Z REPOSITORY_GRYPE_CONFIG_FILE=.grype.yaml
2024-08-22T12:55:00.8048068Z REPOSITORY_GRYPE_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8048674Z REPOSITORY_SECRETLINT_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8050606Z REPOSITORY_TRIVY_ARGUMENTS=['--skip-dirs', './fp-cicd-shared-scripts', '--skip-dirs', './fp-cicd-shared-pipelines/']
2024-08-22T12:55:00.8051068Z REPOSITORY_TRIVY_CONFIG_FILE=trivy.yaml
2024-08-22T12:55:00.8051597Z REPOSITORY_TRIVY_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8053425Z REPOSITORY_TRUFFLEHOG_ARGUMENTS=--exclude-paths=.trufflehogignore
2024-08-22T12:55:00.8054007Z REPOSITORY_TRUFFLEHOG_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8055586Z SHLVL=1
2024-08-22T12:55:00.8055800Z SHOW_ELAPSED_TIME=True
2024-08-22T12:55:00.8056316Z SPELL_LYCHEE_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8056651Z SYSTEM_ACCESSTOKEN=HIDDEN_BY_MEGALINTER
2024-08-22T12:55:00.8058236Z SYSTEM_ARTIFACTSDIRECTORY=/agent/_work/1/a
2024-08-22T12:55:00.8058929Z SYSTEM_COLLECTIONID=c92d2a14-efd3-4ab5-a095-cb41ce6bb725
2024-08-22T12:55:00.8059285Z SYSTEM_COLLECTIONURI=https://dev.azure.com/FlaschenpostSE/
2024-08-22T12:55:00.8059597Z SYSTEM_CULTURE=en-US
2024-08-22T12:55:00.8061188Z SYSTEM_DEFAULTWORKINGDIRECTORY=/agent/_work/1/s
2024-08-22T12:55:00.8061464Z SYSTEM_DEFINITIONID=2724
2024-08-22T12:55:00.8061970Z SYSTEM_DEFINITIONNAME=AMC-C4MODEL-SUPERVISOR-API - PR
2024-08-22T12:55:00.8063563Z SYSTEM_ENABLEACCESSTOKEN=SecretVariable
2024-08-22T12:55:00.8063840Z SYSTEM_HOSTTYPE=build
2024-08-22T12:55:00.8064088Z SYSTEM_ISAZUREVM=1
2024-08-22T12:55:00.8064327Z SYSTEM_ISDOCKERCONTAINER=0
2024-08-22T12:55:00.8064536Z SYSTEM_ISSCHEDULED=False
2024-08-22T12:55:00.8064707Z SYSTEM_JOBATTEMPT=1
2024-08-22T12:55:00.8064900Z SYSTEM_JOBDISPLAYNAME=Merge And Run MegaLinter Configurations
2024-08-22T12:55:00.8066236Z SYSTEM_JOBID=2296aff7-412d-500f-0d72-add5a9c59add
2024-08-22T12:55:00.8066481Z SYSTEM_JOBIDENTIFIER=Linter.MergeAndRunMegaLinterConfigs.__default
2024-08-22T12:55:00.8066662Z SYSTEM_JOBNAME=__default
2024-08-22T12:55:00.8066831Z SYSTEM_JOBPARALLELISMTAG=Private
2024-08-22T12:55:00.8067004Z SYSTEM_JOBPOSITIONINPHASE=1
2024-08-22T12:55:00.8067145Z SYSTEM_JOBTIMEOUT=60
2024-08-22T12:55:00.8067740Z SYSTEM_OIDCREQUESTURI=https://dev.azure.com/FlaschenpostSE/8963000c-4b81-4844-adf8-1d14570c7b8f/_apis/distributedtask/hubs/build/plans/4c3bd827-cfef-461c-9aef-d2da05a4b782/jobs/2296aff7-412d-500f-0d72-add5a9c59add/oidctoken
2024-08-22T12:55:00.8068071Z SYSTEM_PHASEATTEMPT=1
2024-08-22T12:55:00.8068247Z SYSTEM_PHASEDISPLAYNAME=Merge And Run MegaLinter Configurations
2024-08-22T12:55:00.8068553Z SYSTEM_PHASEID=ed1c1d2d-2c57-5782-3c77-ae8cf4b8ba15
2024-08-22T12:55:00.8068989Z SYSTEM_PHASENAME=MergeAndRunMegaLinterConfigs
2024-08-22T12:55:00.8069280Z SYSTEM_PIPELINESTARTTIME=2024-08-22 14:53:25+02:00
2024-08-22T12:55:00.8069594Z SYSTEM_PLANID=4c3bd827-cfef-461c-9aef-d2da05a4b782
2024-08-22T12:55:00.8069783Z SYSTEM_POSTLINESSPEED=500
2024-08-22T12:55:00.8069934Z SYSTEM_PULLREQUEST_ISFORK=False
2024-08-22T12:55:00.8070120Z SYSTEM_PULLREQUEST_PULLREQUESTID=60549
2024-08-22T12:55:00.8070315Z SYSTEM_PULLREQUEST_PULLREQUESTITERATION=4
2024-08-22T12:55:00.8070604Z SYSTEM_PULLREQUEST_SOURCEBRANCH=refs/heads/test-megalinter-pipeline-ref
2024-08-22T12:55:00.8070858Z SYSTEM_PULLREQUEST_SOURCECOMMITID=578842c38d86e70e662ca1f6d37bc62ad81f04ba
2024-08-22T12:55:00.8071276Z SYSTEM_PULLREQUEST_SOURCEREPOSITORYURI=https://FlaschenpostSE@dev.azure.com/FlaschenpostSE/AMC/_git/amc-c4model-supervisor-api
2024-08-22T12:55:00.8071549Z SYSTEM_PULLREQUEST_TARGETBRANCH=refs/heads/main
2024-08-22T12:55:00.8071746Z SYSTEM_PULLREQUEST_TARGETBRANCHNAME=main
2024-08-22T12:55:00.8071906Z SYSTEM_SERVERTYPE=Hosted
2024-08-22T12:55:00.8072066Z SYSTEM_STAGEATTEMPT=1
2024-08-22T12:55:00.8072234Z SYSTEM_STAGEDISPLAYNAME=Linter
2024-08-22T12:55:00.8072569Z SYSTEM_STAGEID=d972221d-4d9b-52c8-0e65-52d13c08acda
2024-08-22T12:55:00.8072755Z SYSTEM_STAGENAME=Linter
2024-08-22T12:55:00.8072952Z SYSTEM_TASKDEFINITIONSURI=https://dev.azure.com/FlaschenpostSE/
2024-08-22T12:55:00.8073143Z SYSTEM_TASKDISPLAYNAME=Lint PR Changed Files
2024-08-22T12:55:00.8073444Z SYSTEM_TASKINSTANCEID=7e54ee85-e014-5069-08a5-5e21c37ad285
2024-08-22T12:55:00.8073645Z SYSTEM_TASKINSTANCENAME=CmdLine5
2024-08-22T12:55:00.8073839Z SYSTEM_TEAMFOUNDATIONCOLLECTIONURI=https://dev.azure.com/FlaschenpostSE/
2024-08-22T12:55:00.8074084Z SYSTEM_TEAMFOUNDATIONSERVERURI=https://dev.azure.com/FlaschenpostSE/
2024-08-22T12:55:00.8074283Z SYSTEM_TEAMPROJECT=AMC
2024-08-22T12:55:00.8074543Z SYSTEM_TEAMPROJECTID=8963000c-4b81-4844-adf8-1d14570c7b8f
2024-08-22T12:55:00.8074855Z SYSTEM_TIMELINEID=4c3bd827-cfef-461c-9aef-d2da05a4b782
2024-08-22T12:55:00.8075031Z SYSTEM_TOTALJOBSINPHASE=1
2024-08-22T12:55:00.8075199Z SYSTEM_WORKFOLDER=/agent/_work
2024-08-22T12:55:00.8075361Z TF_BUILD=True
2024-08-22T12:55:00.8075494Z VALIDATE_ALL_CODEBASE=True
2024-08-22T12:55:00.8075696Z VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT=true
2024-08-22T12:55:00.8075906Z VSTSAGENT_DOCKER_ACTION_RETRIES=true
2024-08-22T12:55:00.8076133Z YAML_PRETTIER_CONFIG_FILE=.prettierrc.yaml
2024-08-22T12:55:00.8076431Z YAML_PRETTIER_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8076640Z YAML_V8R_CONFIG_FILE=LINTER_DEFAULT
2024-08-22T12:55:00.8076901Z YAML_V8R_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8077106Z YAML_YAMLLINT_CONFIG_FILE=.yamllint.yml
2024-08-22T12:55:00.8077403Z YAML_YAMLLINT_FILTER_REGEX_EXCLUDE=.*fp-cicd-shared-scripts/.*
2024-08-22T12:55:00.8077574Z _=/usr/local/bin/python
2024-08-22T12:55:00.8077753Z _AZURE_GUEST_AGENT_DAEMON_VERSION_=2.8.0.11
2024-08-22T12:55:00.8077898Z cli=True
2024-08-22T12:55:00.8078152Z request_id=be156be4-6085-11ef-a036-0242ac110002
2024-08-22T12:55:00.8078523Z ----------------------------------------------------------------------------------------------------
2024-08-22T12:55:00.8078661Z 
2024-08-22T12:55:00.8171544Z [Pre] No commands declared in user configuration
2024-08-22T12:55:05.1881805Z [Activation] KUBERNETES_HELM has been set inactive, as none of these files has been found: ['Chart.yml', 'Chart.yaml']
2024-08-22T12:55:05.9497979Z Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')
2024-08-22T12:55:05.9499128Z Traceback (most recent call last):
2024-08-22T12:55:05.9499569Z   File "<frozen runpy>", line 198, in _run_module_as_main
2024-08-22T12:55:05.9500048Z   File "<frozen runpy>", line 88, in _run_code
2024-08-22T12:55:05.9559631Z   File "/megalinter/run.py", line 11, in <module>
2024-08-22T12:55:05.9560166Z     linter = megalinter.Megalinter({"cli": True, "request_id": str(uuid.uuid1())})
2024-08-22T12:55:05.9560905Z              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-08-22T12:55:05.9561264Z   File "/megalinter/MegaLinter.py", line 181, in __init__
2024-08-22T12:55:05.9561590Z     self.load_linters()
2024-08-22T12:55:05.9561964Z   File "/megalinter/MegaLinter.py", line 566, in load_linters
2024-08-22T12:55:05.9562461Z     all_linters = linter_factory.list_all_linters(linter_init_params)
2024-08-22T12:55:05.9562771Z                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-08-22T12:55:05.9563041Z   File "/megalinter/linter_factory.py", line 30, in list_all_linters
2024-08-22T12:55:05.9563494Z     descriptor_linters = build_descriptor_linters(
2024-08-22T12:55:05.9563884Z                          ^^^^^^^^^^^^^^^^^^^^^^^^^
2024-08-22T12:55:05.9564292Z   File "/megalinter/linter_factory.py", line 117, in build_descriptor_linters
2024-08-22T12:55:05.9564862Z     linter_instance = linter_class(linter_init_params, instance_attributes)
2024-08-22T12:55:05.9565318Z                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-08-22T12:55:05.9565723Z   File "/megalinter/linters/GitleaksLinter.py", line 19, in __init__
2024-08-22T12:55:05.9566445Z     self.pr_source_sha, self.pr_target_sha = self.get_pr_data()
2024-08-22T12:55:05.9619552Z                                              ^^^^^^^^^^^^^^^^^^
2024-08-22T12:55:05.9619936Z   File "/megalinter/linters/GitleaksLinter.py", line 40, in get_pr_data
2024-08-22T12:55:05.9620406Z     pr_target_sha = self.get_azure_devops_pr_target_sha(
2024-08-22T12:55:05.9622400Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-08-22T12:55:05.9622780Z   File "/megalinter/linters/GitleaksLinter.py", line 67, in get_azure_devops_pr_target_sha
2024-08-22T12:55:05.9623113Z     repo = utils.git.Repo(os.path.realpath(self.workspace))
2024-08-22T12:55:05.9623387Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-08-22T12:55:05.9624138Z   File "/usr/local/lib/python3.12/site-packages/GitPython-3.1.41-py3.12.egg/git/repo/base.py", line 276, in __init__
2024-08-22T12:55:05.9624496Z     raise InvalidGitRepositoryError(epath)
2024-08-22T12:55:05.9624796Z git.exc.InvalidGitRepositoryError: /tmp/lint
2024-08-22T12:55:06.9246789Z ##[warning]Linter found issues. Check the logs for details.
2024-08-22T12:55:06.9255939Z 
2024-08-22T12:55:06.9350103Z ##[section]Finishing: Lint PR Changed Files

nvuillam commented 3 weeks ago

@karl18 wow, that's quite a setup, you are probably much more expert about Azure than I am :)

VALIDATE_ALL_CODEBASE=false is probably the cause of the issue (I'm pretty sure than if you set it to true you won't have an issue anymore)

You can see the discussion on this issue https://github.com/oxsecurity/megalinter/issues/1125

When you run manually a pipeline, it is done with your user, that probably has more rights than Azure Build Services user, that could explain why it does not crash

Can you try to give more rights to the Azure build service ?

nvuillam commented 3 weeks ago

You might also try to make a new git checkout step just before MegaLinter, because it seems to not find a repo according to the error

        - checkout: self
          fetchDepth: 0
          clean: true
          persistCredentials: true
          displayName: 'Git Checkout Self'

karl18 commented 3 weeks ago

@nvuillam Thanks for the compliment, I've been working on this project for quite a while now! :)

Your doubts about why we see the error and the troubleshooting steps are lovely and make much sense to me!

But unluckily, at the end of the day,

VALIDATE_ALL_CODEBASE is always set to true in my configuration, check the logs I sent … I was asking if I want to use it later and set it to false, how it looks like on Azure Pipelines!
Also, Unfortunately, giving even all permissions to Azure build service didn't solve the problem :(
I tried to add checkout: self right before the MegaLinter step, but it didn't help! Even in other different parts of the pipeline, the error doesn't change!

I was asking if any of the ENV vars in DEBUG mode rang the bell to you, e.g. I see BUILD_SOURCEBRANCHNAME=merge on the automated PR builds, for me, it might be an issue as the correct branch name won't be passed properly to the script, or? Couldn't think of other possibilities :(

Appreciate it .. we could also engage any Azure Pipelines expert that you know ! ;)

nvuillam commented 3 weeks ago

I think that BUILD_SOURCEBRANCHNAME should be the source branch of the PR :)

About experts... I think @EtienneDeneuve @DariuszPorowski @riosengineer @lumarel are quite experts in Azure... maybe they can help ? ^^

riosengineer commented 2 weeks ago

I only ever run MegaLinter in VALIDATE_ALL_CODEBASE=false mode because I only want it to lint the files that differ from main as part of the PR.

In ADO, this does work for me and has done across many different orgs/projects - it will only pickup the changed files which is great - plus it means it's quicker to run as the full repo scan can take sometime depending on repo/file sizes and linters enabled.

I notice that the MegaLinter task is running off a custom ADO agent pool called: Azure Pool - Linux - West Europe - is the task running on a self hosted agent or Microsoft hosted? The only difference I can think of currently. I usually only ever use Microsoft hosted agents for my PR pipeline which has MegaLinter included, so I don't have any issues when the build validation pipeline triggers on PR creation. I am wondering if that is causing an odd permission problem somehow.

Has it ever worked for you?

karl18 commented 2 weeks ago

Hi @riosengineer, Basically these Agents are self-hosted ones used as a resort for the MegaLinter Image, instead of pulling the image each time a build run ! I tried to use Microsoft hosted Agents + checking the project permissions to the Agents _(BuildService +Project granted Administrator on these Agents) and still the issue is not solved!

If it's related to a permission issue, the Job won't be running initially! Of course MegaLinter is working fine except when it's a PR automated (maybe I suggest checking my question one more time) e.g. when I open a pull request and the Policy queue the pipeline, here's my problem!

I always see Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')

Regarding VALIDATE_ALL_CODEBASE=false what else do you need to identify in the pipeline? Only adding it megalinter.yml config OR something else needed to be done e.g. Docker env var , etc ...?

echoix commented 2 weeks ago

I didn't follow closely the whole thread, but doesn't it seem similar to the limitation on GitHub Actions where when an automated action is done by the GITHUB_TOKEN, it doesn't trigger any more CI (to prevent an infinite loop and probably security too). The way around it is to use a Personal Access Token, where the actions are done as if it was that user.

riosengineer commented 2 weeks ago

Hi @riosengineer, Basically these Agents are self-hosted ones used as a resort for the MegaLinter Image, instead of pulling the image each time a build run ! I tried to use Microsoft hosted Agents + checking the project permissions to the Agents _(BuildService +Project granted Administrator on these Agents) and still the issue is not solved!

If it's related to a permission issue, the Job won't be running initially! Of course MegaLinter is working fine except when it's a PR automated (maybe I suggest checking my question one more time) e.g. when I open a pull request and the Policy queue the pipeline, here's my problem!

I always see Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')

Regarding VALIDATE_ALL_CODEBASE=false what else do you need to identify in the pipeline? Only adding it megalinter.yml config OR something else needed to be done e.g. Docker env var , etc ...?

You can use VALIDATE_ALL_CODEBASE: false in your megalinter.yaml config, or if you're using the docker defaults you can override it with -e VALIDATE_ALL_CODEBASE=false \

Like:

trigger: none

pool:
  vmImage: ubuntu-latest

jobs:
- job: MegaLinter
  displayName: MegaLinter 
  steps:
  - checkout: self
  - script: docker pull oxsecurity/megalinter:v8
    displayName: Pull MegaLinter
  - script: |
      docker run -v $(System.DefaultWorkingDirectory):/tmp/lint \
      --env-file <(env | grep -e SYSTEM_ -e BUILD_ -e TF_ -e AGENT_) \
      -e VALIDATE_ALL_CODEBASE=false \
      -e DISABLE_LINTERS=REPOSITORY_CHECKOV,SPELL_CSPELL,YAML_YAMLLINT,COPYPASTE_JSCPD,ARM_ARM_TTK,REPOSITORY_KICS,REPOSITORY_DEVSKIM \
      -e SYSTEM_ACCESSTOKEN=$(System.AccessToken) \
      -e GIT_AUTHORIZATION_BEARER=$(System.AccessToken) \
      oxsecurity/megalinter:v8
    displayName: Run MegaLinter

  - task: PublishPipelineArtifact@1
    condition: succeededOrFailed()
    displayName: Upload MegaLinter reports
    inputs:
      targetPath: $(System.DefaultWorkingDirectory)/megalinter-reports/
      artifactName: MegaLinterReport

I actually don't need to set any special permissions for this to work. You don't or shouldn't have to amend the build service perms, at least on an MS Hosted pipeline run anyway. The above saved in a .yaml file, then assigned as the repos build validation pipeline is enough for megalinter to run successfully in ADO.

As for your actual issue, I am not sure, sorry, but hopefully the above helps. Maybe try the above yaml with a hosted agent, and temporarily set it as the build validation pipeline to test it works ok?

karl18 commented 1 day ago

Hi @riosengineer -e VALIDATE_ALL_CODEBASE=false \ doesn't work for me at all, do you have any other special checkout for the repos you do which you can show me?

I tried several fetchdepth: 0,1,2 but none of them worked, I always see:

Listing updated files in [/tmp/lint] using git diff.
Unable to list updated files from git diff. Switch to VALIDATE_ALL_CODE_BASE=true

And somehow I had the same issue for APPLY_FIXES: all or for e.g. APPLY_FIXES: YAML_YAMLLINT, doesn't work at all on azure pipelines as well, I tried to set both env var in docker cmd and the conf. File but none of them worked :(

I appreciate any other Idea which I could try out on Azure pipelines, which I don't know! Appreciate your response !

Note: I also checked #1125 , #3421

@nvuillam @echoix @riosengineer Regarding the main thread issue's, Yay! I figured out how to solve it 2 weeks ago ;) : So I only needed to remove this line from the conf file REPOSITORY_GITLEAKS_PR_COMMITS_SCAN=True It seems conflicting with the PR automation and once I commented this line out, it worked like a charm!

riosengineer commented 1 day ago

Hi @riosengineer -e VALIDATE_ALL_CODEBASE=false \ doesn't work for me at all, do you have any other special checkout for the repos you do which you can show me?

I tried several fetchdepth: 0,1,2 but none of them worked, I always see:
Listing updated files in [/tmp/lint] using git diff.
Unable to list updated files from git diff. Switch to VALIDATE_ALL_CODE_BASE=true
And somehow I had the same issue for APPLY_FIXES: all or for e.g. APPLY_FIXES: YAML_YAMLLINT, doesn't work at all on azure pipelines as well, I tried to set both env var in docker cmd and the conf. File but none of them worked :(

I appreciate any other Idea which I could try out on Azure pipelines, which I don't know! Appreciate your response !

Note: I also checked #1125 , #3421

@nvuillam @echoix @riosengineer Regarding the main thread issue's, Yay! I figured out how to solve it 2 weeks ago ;) : So I only needed to remove this line from the conf file REPOSITORY_GITLEAKS_PR_COMMITS_SCAN=True It seems conflicting with the PR automation and once I commented this line out, it worked like a charm!

Nope, the snippet I pasted above is all I have on a completely fresh ADO project with no other changes (besides making the build validation policy to trigger on PR). Maybe worth trying in a fresh project to rule out permission problems.

karl18 commented 1 day ago

Hi @riosengineer @nvuillam @echoix, I managed to make it working by changing the location where megalinter checks for the updated files in [/tmp/lint] using git diff. I just realized that /tmp/lint doesn't contain the checked-out repo files to do any git diffs there 😀

I changed this line from:

docker run \ -v $(System.DefaultWorkingDirectory):/tmp/lint \ to

docker run \ -v $(System.DefaultWorkingDirectory)/$(Build.Repository.Name)/:/tmp/lint \

Only one thing now left is APPLY_FIXES 😀 .. I'm trying to find out what's might be the issue..

nvuillam commented 11 hours ago

@karl18 amazing finding ! :)

Please can you make a PR to update the doc once you unlocked MegaLinter's greatest mystery with Azure ? ^^

It's around here :) https://github.com/oxsecurity/megalinter/blob/main/README.md#azure-pipelines ( don't bother propagating in /docs, it's done by automated build scripts)

oxsecurity / megalinter

Megalinter doesn't work with Azure Devops automated PR build runs #3921