search

Drieam / rspec-github

Formatter for RSpec to show errors in GitHub action annotations
https://drieam.github.io/rspec-github/
MIT License
60 stars 10 forks source link

Not displaying all failures #7

Closed matthewford closed 2 years ago

matthewford commented 3 years ago

Thanks for this gem I was wondering if its meant to display all failures or if there is a limit?

While doing a Rails upgrade I see around 300 spec failures in the logs but only 34 reported in the UI.

Cheers, Matt

StefSchenkelaars commented 3 years ago

Hi @matthewford, thanks for the issue.

Well, there is no intentional limit to the number of failures. But It would be great if you could provide a bit more information. Is the repo where you saw this behaviour public or could you share the logs? Of course I also understand if they contain sensitive information but I first need to get this reproduced.

I do see that this issue received +1 as well so it looks like more people have this issue. So if any of you (@trafium / @enjaku4) could help me out here, I could investigate this issue.

StefSchenkelaars commented 3 years ago

Ok, that was easy to reproduce. I've just added many failing specs (500) to a commit and looked at the result. The raw logs look ok so I think it's really GitHub that's malfunctioning ➡️ I'll send them a message.

An alternative would maybe be to use the checks API to send the errors to but that seems overkill to me.

Ticket text:

Hi GitHub,

I've created a formatter for the ruby testing framework RSpec that outputs the errors as annotations in GitHub Actions. It uses the format ::error file={name},line={line},col={col}::{message} as described in the documentation.

This actually works quite well, however, a user opened an issue that not all errors are shown as annotations. So to reproduce, I created a commit with loads of failing tests (>500). But if you then look at the actual action run, it only shows 15 annotations.

But if you have a look at the raw logs of the e2e job of the actions, it does show all the errors in the ::error file={name},line={line},col={col}::{message} format. So in some way, there is something not working with the scanning of the logs.

Hopefully you can figure out why not all annotations are show.

Thanks,

Stef

Link to ticket (needs authentication)

matthewford commented 3 years ago

@StefSchenkelaars not sure if it helps but we're an enterprise user :)

StefSchenkelaars commented 3 years ago

I still haven't heard anything from GitHub... Maybe since you are an enterprise user, you can send them a message as well, maybe they listen to you 🙈

StefSchenkelaars commented 3 years ago

In another project that does not use this formatter I've also noticed that not all annotiations are shown. So then I started googling and it seems that there are some GitHub community tickets on this.

Summary of their answer:

StefSchenkelaars commented 3 years ago

Similar issue for swiftlint: https://github.com/norio-nomura/action-swiftlint/issues/17

StefSchenkelaars commented 3 years ago

@matthewford I've been playing with the checks API in PR https://github.com/Drieam/rspec-github/pull/10. It's still a draft but maybe you want to try it out as well to see if it fixes your problems.

matthewford commented 3 years ago

Thanks will do, I raised a ticket and they are looking into it

StefSchenkelaars commented 3 years ago

This was the reply from GitHub Support:

My colleague did some testing and found the following:

Currently, GitHub Actions does not support so many annotations in a workflow run, the below is a summary of the limitation:

10 warning annotations and 10 error annotations per step 50 annotations per job (sum of annotations from all the steps) 50 annotations per run (separate from the job annotations, these annotations aren’t created by users https://github.community/t/annotation-limitation/17998/2

We do have an internal issue open for the Documentation team to review so that we can add these annotation limits on our official docs. As it turns out it is still under review but hopefully it will be published in the near future.

If you have any further questions, please do not hesitate to ask!

Cheers, GitHub Support

So it looks like they are only going to update the documentation and not 'just fix it'...