search

hercules-ci / support

User feedback, questions and our public roadmap. help@hercules-ci.com
5 stars 1 forks source link

Some way to mark "expected failures" #43

Closed michaelpj closed 4 years ago

michaelpj commented 4 years ago

It's not uncommon to have tests which don't work, and you know that, but you'd like to keep running in case they start working.

At the moment you just have to filter them out and then occasionally try re-enabling them.

This could be expensive, though, since Nix doesn't currently cache failures, so we'd retry them every time.

jbboehr commented 4 years ago

Would be nice to have it work with evaluation failures as well, so I can test against nixpkgs-unstable without every other build failing.

roberth commented 4 years ago

@michaelpj This seems to be a duplicate of #34 which will be available in the upcoming release. Nix not caching failures isn't a problem because Hercules CI does.

@jbboehr Your use case seems to be quite different. I don't quite understand why you need to permit evaluation failures to work with nixpkgs-unstable. Presumably you're building something that depends on nixpkgs-unstable, so you have an expression that depends on nixpkgs. If it fails to evaluate, that seems like something you'd want to know about and marking the build as failed seems to be the right way to achieve that.

jbboehr commented 4 years ago

@roberth It's been a while since I posted this so my memory of the details is a bit fuzzy, please excuse me if I remember incorrectly, but the problem is the whole build fails when evaluation for only nixpkgs-unstable fails. Ideally the evaluation failure would be contained to the job. This may have changed in the meantime or I may be mistaken.

Perhaps part of my problem is that I need to import a file from nixpkgs? https://github.com/jbboehr/php-psr/blob/5be30309c4096d5ca3216f87a6872f4c46a82562/nix/ci.nix#L9

I stopped testing against unstable a while ago, in part due to my poor tiny server having to recompile PHP for hours at a time.

roberth commented 4 years ago

@jbboehr The evaluator will try each attribute individually. I've checked that repo's history and the jobs that failed either produced plenty of attributes or the failure was caused by use of NIX_PATH which is still unsupported for now. I believe your request is already implemented to the extent possible, but if you have any questions feel free to open another issue.

I'll close this because it's a duplicate of #34.