Devops - Analysis execute in seconds and nothing seems to be done. No errors but no results

[tlogik]

Bug Description

I am testing your devops extension and i have discovered that i cannot actually get it to analyze anything. It is unclear to me why.

I have setup a couple of pipelines, one for a C# based project and another with React (js, ts, html, css etc)

When the PR is triggers the pipeline the extension runs with success. However the analysis take 1 second and then it is done. There are no error messages or anything indicating a problem, but regardless of how many changes i add to a PR it does not seem to analyze them.

Can you asssist in figuring out why?

I have added a couple of screendumps from devops to show the result of the analysis as well as the YAML used.

C# related pipeline

YAML

      - job: CodeReview
        displayName: OPENAI - PR ONLY - Code Analysis
        condition: and(succeeded(), eq(variables.IS_PR, 'true'))
        pool:
          vmImage: 'ubuntu-latest'
        steps:
        - checkout: self
          persistCredentials: true
        - task: OpenAICodeReview@1
          inputs:
            api_key: $(OPENAI_APIKEY)
            ai_model: 'gpt-3.5-turbo'
            bugs: true
            performance: true
            best_practices: true
            file_extensions: 'cs,js,csproj'
            additional_prompts: 'Fix variable naming, Ensure consistent indentation, Review error handling approach'

Result (observe analysis is 1 second and nothing is added to the PR comments)

REACT related pipeline

YAML

          - job: CodeReview
            displayName: OPENAI - PR ONLY - Code Analysis
            condition: and(succeeded(), eq(variables.IS_PR, 'true'))
            pool:
              vmImage: 'ubuntu-latest'
            steps:
            - checkout: self
              persistCredentials: true
            - task: OpenAICodeReview@1
              inputs:
                api_key: $(OPENAI_APIKEY)
                ai_model: 'gpt-3.5-turbo'
                bugs: true
                performance: true
                best_practices: true
                file_extensions: 'cs,js,html,css,ts'
                additional_prompts: 'Fix variable naming, Ensure consistent indentation, Review error handling approach'

Result (observe analysis is 1 second and nothing is added to the PR comments)

Extension version

• Version: 1.0.0 (Latest) Last updated 11 Dec 2023 at 14.58 CET

[a1dancole]

HI,

Thanks for trying out the extension.

I think it's because you've followed the documentation correctly! The extensions should actually be listed with a '.'

I'll get the tooltip updated, but that should get you working in the meantime. If not please let me know and I'll try and reproduce.

For example:

        displayName: OPENAI - PR ONLY - Code Analysis
        condition: and(succeeded(), eq(variables.IS_PR, 'true'))
        pool:
          vmImage: 'ubuntu-latest'
        steps:
        - checkout: self
          persistCredentials: true
        - task: OpenAICodeReview@1
          inputs:
            api_key: $(OPENAI_APIKEY)
            ai_model: 'gpt-3.5-turbo'
            bugs: true
            performance: true
            best_practices: true
            file_extensions: '.cs,.js,.csproj'
            additional_prompts: 'Fix variable naming, Ensure consistent indentation, Review error handling approach ```

[tlogik]

Hi @a1dancole Thanks - that makes sense - I will test. Then to follow up on the that area. Does the EXCLUDE file list support wildcards? example: '*.txt,*.css,*.bicep,*.yaml'

[tlogik]

Hi @a1dancole

I updated my pipeline task as you suggested and it works :-) Awesome - thanks for the support. One comment to you documentation. Maybe add the part of ensuring access to the AUTH token for YAML pipelines is achieved by this task:

        - checkout: self
          persistCredentials: true

For brevity the updated YAML that works.

      - job: CodeReview
        displayName: OPENAI - PR ONLY - Code Analysis
        condition: and(succeeded(), eq(variables.IS_PR, 'true'))
        pool:
          vmImage: 'ubuntu-latest'
        steps:
        - checkout: self
          persistCredentials: true
        - task: OpenAICodeReview@1
          inputs:
            api_key: '$(OPENAI_APIKEY)'
            ai_model: 'gpt-3.5-turbo'
            bugs: true
            performance: true
            best_practices: true
            file_extensions: '.cs,.js,.csproj'
            additional_prompts: 'Fix variable naming, Ensure consistent indentation, Review error handling approach'

[a1dancole]

Hi @a1dancole Thanks - that makes sense - I will test. Then to follow up on the that area. Does the EXCLUDE file list support wildcards? example: '*.txt,*.css,*.bicep,*.yaml'

No it does not, the exclude list in our use case was just to exclude specific files that we never wanted to send to Open AI. If you think having a list for wildcard excludes would be beneficial it's definitely something easy enough to add.

[a1dancole]

Hi @a1dancole

I updated my pipeline task as you suggested and it works :-) Awesome - thanks for the support. One comment to you documentation. Maybe add the part of ensuring access to the AUTH token for YAML pipelines is achieved by this task:

        - checkout: self
          persistCredentials: true

For brevity the updated YAML that works.

      - job: CodeReview
        displayName: OPENAI - PR ONLY - Code Analysis
        condition: and(succeeded(), eq(variables.IS_PR, 'true'))
        pool:
          vmImage: 'ubuntu-latest'
        steps:
        - checkout: self
          persistCredentials: true
        - task: OpenAICodeReview@1
          inputs:
            api_key: '$(OPENAI_APIKEY)'
            ai_model: 'gpt-3.5-turbo'
            bugs: true
            performance: true
            best_practices: true
            file_extensions: '.cs,.js,.csproj'
            additional_prompts: 'Fix variable naming, Ensure consistent indentation, Review error handling approach'

I'm glad it's working for you.

And thanks, I will update the documentation with this.

[tlogik]

Hi @a1dancole

If you think having a list for wildcard excludes would be beneficial it's definitely something easy enough to add.

I think it would be quite beneficial to broadly being able to ignore filetypes etc. broadly. Would make it very simple to ensure testfiles, pipelines etc. be ignored broadly.

Thanks for the tooling - it is really cool :-D

[csharpengineer]

I also noticed that the api_key parameter is quoted.

api_key: '$(OPENAI_APIKEY)' vs api_key: $(OPENAI_APIKEY)

That made a difference in the extension working or not for me.