search

errata-ai / vale-action

:octocat: The official GitHub Action for Vale -- install, manage, and run Vale with ease.
MIT License
195 stars 46 forks source link

Warning causes workflow to fail when `fail_on_error` is `true` #84

Open chadfawcett opened 1 year ago

chadfawcett commented 1 year ago

Based on the docs, I would only expect the workflow to fail if there was an error. I'm seeing it fail when there is only a Warning. Is this the expected behaviour?

According to the reviewdog docs, a Warning should result in a "neutral" GitHub Status.

jdkato commented 1 year ago

Warnings should indeed result in a neutral status -- see, for example, this job with multiple warnings and a neutral status.

Can you share your .vale.ini and workflow config?

chadfawcett commented 1 year ago

Thanks for the response @jdkato. Here's my config setup

.vale.ini

MinAlertLevel = suggestion
StylesPath = .github/styles

[*.md]
BasedOnStyles = proselint, inclusive-language
proselint.Typography = NO
proselint.Cliches = NO
proselint.Very = NO
proselint.Cursing = NO
proselint.Currency = NO
proselint.CorporateSpeak = NO
proselint.Skunked = NO
proselint.DateSpacing = NO
proselint.DateCase = NO
proselint.Diacritical = NO
proselint.Hyperbole = NO

workflow

name: Vale Lint
on:
  pull_request:

jobs:
  prose:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      - name: Vale
        uses: errata-ai/vale-action@e9cd17e2bb87ed772b939286b7b141d7ba64ed54
        env:
          GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}
        with:
          fail_on_error: true

Then the warning was for starting a paragraph with "but". Which is defined like so: .github/styles/proselint/But.yml

extends: existence
message: "Do not start a paragraph with a 'but'."
level: warning
scope: paragraph
action:
  name: remove
tokens:
  - ^But
afscrome commented 1 year ago

I'm encountering this too - I've reproduced it at https://github.com/afscrome/vale-test/actions/runs/3349416267/jobs/5549402510 .

Whilst there are no error annotations, there are errors in the "Filtered Findings". I wonder if those are resulting in the final error status?

From my testing, the problem seems to have nothing to do with fail_on_error - even when that's false, the check itself still reporting back as "failed" - see https://github.com/afscrome/vale-test/actions/runs/3349406635 . All fail_on_error seems to do is decide if the error gets propagated up to the main job.

jdkato commented 1 year ago

While it's possible that I'm misconfiguring reviewdog, it seems that this isn't easy to get right: