PR commenting not working for me (Help?)

[bhatvikrant]

My Github workflow looks like this:

name: Lighthouse Check
on:
  deployment_status:
    branches:
      - '*'
jobs:
  lighthouse-check:
    runs-on: ubuntu-latest
    if: github.event.deployment_status.state == 'success'
    steps:
      - uses: actions/checkout@master
      - run: mkdir /tmp/artifacts
      - name: Run Lighthouse
        uses: foo-software/lighthouse-check-action@master
        with:
          accessToken: ${{ secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN }}
          outputDirectory: /tmp/artifacts
          urls: '${{ github.event.deployment_status.target_url }}'
      - name: Upload artifacts
        uses: actions/upload-artifact@master
        with:
          name: Lighthouse reports
          path: /tmp/artifacts

Here I have added LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN as my Github personal access token. But for some reason, it is still not commenting the lighthouse results on any PRs. Check this PR for example.

Also, can I use the secrets.GITHUB_TOKEN (default that github provides with every repo) instead of secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN ?

PS: I have also asked this question to a wider audience on stackoverflow here.

[adamhenson]

Hi @bhatvikrant. Sorry for the troubles and thanks for providing all those details. Our documentation isn't that great, especially on this subject. There are plans to improve. The intention of this feature was to comment on PRs, however I feel this may need to be revisited to be more robust.

My theory is that because you are not triggering this on pull_request it isn't working correctly (even though you have a PR open). Unless you explicitly specify commentUrl, then this GitHub Action will use context.payload.pull_request.url, which I'm guessing will be empty if not triggered by a pull_request. Your example is triggered by deployment_status.

To test my theory, could you make this change? Yes, we're hard-coding the PR, but this will at least isolate the issue (hopefully), and then we can brainstorm on next steps.

with:
  accessToken: ${{ secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN }}
+  commentUrl: https://api.github.com/repos/bhatvikrant/prep-station/pulls/2/reviews
  outputDirectory: /tmp/artifacts
  urls: '${{ github.event.deployment_status.target_url }}'

[bhatvikrant]

Thank you for replying promptly @adamhenson, I hope you are doing good! Sure I'll update that in the PR, please check back in 2 minutes from now.

PS: I can help you with the documentation part if you have anything in mind. 😄

[bhatvikrant]

Your theory was right, it did comment this time around

[adamhenson]

Thanks @bhatvikrant. Regarding the docs, that's something we'll have to do because it will require in-depth knowledge of this project. It's on the roadmap though... we'll get there.

Okay, well that's interesting. I think this is a clever way of populating the URLs and I'm not so sure how we can get this to work.

I wonder if you do the following - it will show up on PRs 🤔 . Want to give it a shot?

with:
  accessToken: ${{ secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN }}
-  commentUrl: https://api.github.com/repos/bhatvikrant/prep-station/pulls/2/reviews
+  commentUrl: https://api.github.com/repos/bhatvikrant/prep-station/commits/${{ github.sha }}/comments
  outputDirectory: /tmp/artifacts
  urls: '${{ github.event.deployment_status.target_url }}'

[bhatvikrant]

Alrighty, I ve added that.

[bhatvikrant]

Interestingly, that does seem to work!

[adamhenson]

Okay, awesome - looks like it worked @bhatvikrant

Can you do 2 more things.

1. Remove that 1st comment with the scores, but keep the last?
2. Push up an empty commit to trigger the action again.

In theory there should only be one comment that is edited when additional commits are pushed. I want to see if this works as expected.

[bhatvikrant]

I can't delete the autogenerated 1st comment, but I have pushed an empty commit.

[adamhenson]

Great, no prob - I just want to see if the 2nd gets edited or if a new comment is created 🤞

[bhatvikrant]

Yes, I understood that, hehe :)

[bhatvikrant]

So, it created a new commit

[adamhenson]

Yep, okay looks like it will just keep creating new comments. So, this method will work, but your PR will get flooded.

If you were to do this on a PR triggered action, then only one comment would show and it would keep editing it.

I don't know if there is a better solution at this time, but I'll take some more time on it and get back to you.

[bhatvikrant]

Sure! I think I can make do with it for now, but I will be looking forward to new updates to the action and suggestions from you! Thankyou once again for your time :) @adamhenson

[adamhenson]

No problem - and thanks for reporting this one.

[bhatvikrant]

Hey @adamhenson, so I was implementing the same as we discussed above at my work repo which is private. But for some reason I am getting this octakit deprecated warning, and no commit found for SHA.

Any idea what can be going wrong? I have implemented it exactly as we have discussed above.

[adamhenson]

Hey @bhatvikrant - I wouldn't worry about the Octokit warning. I've seen that one before and it's related to a dependency we probably need to bump.

My guess is that there is something wrong with the commentUrl. Make sure it reflects the repo correctly. If it's a public repo, you can send the code my way... or really just the URL for the repo and what you have for commentUrl.

[bhatvikrant]

Hey @adamhenson, the repo is private, but I'll share the context that should help. So before the lighthouse-check workflow, I have this dummy-commit.yml workflow that is a workaround for deploying to Vercel for preview.

this is how dummy-commit.yml looks like

name: Vercel force trigger build
on:
    push:
        branches:
            - '*'
jobs:
    trigger_vercel_build:
        runs-on: ubuntu-latest
        if: github.actor != 'mehulmpt'
        steps:
            - name: Set branch name
              id: vars
              run: echo ::set-output name=short_ref::${GITHUB_REF#refs/*/}

            - name: Dummy commit
              run: |
                  curl -X PUT \
                  --url https://api.github.com/repos/${{ github.repository }}/contents/.VERCEL_TRIGGER_BUILD \
                  --header 'authorization: Bearer ${{ secrets.CODEDAMN_GH_DEPLOYMENT_TOKEN }}' \
                  --header 'content-type: application/json' \
                  --data '{
                  "sha":"e69de29bb2d1d6434b8b29ae775ad8c2e48c5391",
                  "branch": "${{ steps.vars.outputs.short_ref }}",
                  "message": "trigger vercel build",
                  "content": ""
                  }'

and after this initiates a build for vercel preview on every commit, then the lighthouse-check workflow us run. I have a feeling this might be creating this issue. What are your thoughts?

[bhatvikrant]

I echoed the comment url in my lighthouse workflow

name: Lighthouse Check
on:
    deployment_status:
        branches:
            - '*'
            - '!master'

jobs:
    lighthouse-check:
        runs-on: ubuntu-latest
        if: github.event.deployment_status.state == 'success'
        steps:
            - uses: actions/checkout@master
            - run: mkdir /tmp/artifacts
+           - run: echo https://api.github.com/repos/bhatvikrant/prep-station/commits/${{ github.sha }}/comments
+           - run: echo ${{ github.sha }}
            - name: Run Lighthouse
              uses: foo-software/lighthouse-check-action@master
              with:
                  commentUrl: https://api.github.com/repos/bhatvikrant/prep-station/commits/${{ github.sha }}/comments
                  accessToken: ${{ secrets.LIGHTHOUSE_CHECK_GITHUB_ACCESS_TOKEN }}
                  outputDirectory: /tmp/artifacts
                  urls: '${{ github.event.deployment_status.target_url }}'
            - name: Upload artifacts
              uses: actions/upload-artifact@master
              with:
                  name: Lighthouse reports
                  path: /tmp/artifacts

And the following was the output in actions:

[adamhenson]

Hey @bhatvikrant - but if you're using a different repo than before then why do you still have bhatvikrant/prep-station as part of the commentUrl. Maybe you should just change it to this:

commentUrl: https://api.github.com/repos/${{ github.repository }}/commits/${{ github.sha }}/comments

[bhatvikrant]

Oh shoot, midnight hours got me hallucinating that I couldn't even see this blunder. Thank you once again @adamhenson.

[adamhenson]

No problem @bhatvikrant - glad I could kinda help at least. I did some research and it seems like using a comment URL to be associated with commit is the only straightforward option (for now at least) when using the deployment_status event. The reason is that the deployment_status event doesn't associate a PR. With that said, I'll need to close this and hopefully the concluding thoughts below can help others.

Conclusion

When using the deployment_status event to trigger a workflow, Lighthouse Check Action PR comments will not work out of the box because under the hood we construct use github.context.payload.pull_request.url to post comments unless specified by the user. deployment_status event doesn't provide pull request data.

Solution with Caveat

In order to enable comments, you would need to populate commentUrl, similar to our example, to comment on the associated commit. There is a caveat to this in that if you have an open PR, you will have a new comment with Lighthouse results for every commit, which can get noisy. This is different from using the pull_request event, which by default only posts one comment and updates it for each new commit after.

Possible Hacky Solutions for PR Comments

• Use GitHub API to get associated PRs to the commit as explained here. The issue with that is you could have multiple PRs and you would somehow need to choose one if so. This could work for small projects in a case that you only have one PR open at a time. You would then need to use this PR # to construct a URL per the PR review API and populate commentsUrl with it.
• I haven't really solidified this idea, but apparently there is a way to share artifacts between workflows. Maybe there could be a way to somehow share a corresponding PR, but I'm not really sure how that could work... just a half-baked idea. Also, my source is Stack Overflow, so probably not the best idea.