search

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://UKGovernmentBEIS.github.io/inspect_ai/
MIT License
385 stars 41 forks source link

Add skip marker for failing test #32

Closed canyon289 closed 1 month ago

canyon289 commented 1 month ago

This PR contains:

What is the current behavior? (You can also link to an open issue here)

Main is failing because dataset is missing

What is the new behavior?

Green on main

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No

Other information:

Test should be properly fixed so its works as intended

aisi-inspect commented 1 month ago

This is believe it or not a much, much more vexing issue! That test actually passes fine when run from the shell and that file isn't actually missing. The problem we are seeing is that only when run in a GH action the built Python package does not included package data files (i.e. non Python files like datasets or the web assets for inspect view). We have not yet determined the cause of this but it's sure to be something terribly obscure :-(

canyon289 commented 1 month ago

Oh I feel you. I saw the same myself and have dealt with my fair share of Github CI issues. There no good way to figure out what's going on other than guess and check....

https://ravinkumar.com/DockerforDS.html#DockerforDS https://ravinkumar.com/AzurePipelines.html#AzurePipelines

My suggestion for now is add a skip, or if if you'd like a conditional that detects when test are running in CI. The test failure on main is a bit confusing, and it doesn't to justice for all the great tests youve included thus far

aisi-inspect commented 1 month ago

Thanks again for the nudge, I put in a skip on GH action decorator for the time being (that's on main now so I will close this PR).