quantified-uncertainty / squiggle

An estimation language
https://squiggle-language.com
MIT License
155 stars 23 forks source link

Try blocks can break IF statements #3391

Closed OAGr closed 1 month ago

OAGr commented 1 month ago

Description of suggestion or shortcoming:

I'm trying to write tests, in Squiggle, using sTest, to flag if an error was caught or not.

The problem is that when doing that, I'm getting a weird error.

See this. The badExample fails with a JS Exception: Error: Internal error: invalid capture id 1 error.

This seems like an interpreter thing. The LLMs like using try/catch in testing a lot, so this breaks a bunch of tests.

keyword = "____FAILED____"

fnThrewError(fn) = {
  foo = try(fn, {|| "____FAILED____"})
  foo
}

badExample(fn) = {
  a = fnThrewError({|| throw("ads")})
  a == keyword
}
badResult = badExample({|| throw("Error")}) // Fails. JS Exception: Error: Internal error: invalid capture id 1

goodResult = fnThrewError({|| throw("Error!")}) // Works. Outputs ____FAILED____, 

Depends on https://github.com/quantified-uncertainty/squiggle/issues/3391

OAGr commented 1 month ago

I wanted to add throw checking code to sTest, but couldn't, because of this.

toThrowAnyError: {
    ||
    if fnThrewError(
      actual
    ) then true else "Expected to throw an error, but no error was thrown"
  },
  toNotThrow: {
    ||
    if !fnThrewError(
      actual
    ) then true else "Expected not to throw an error, but an error was thrown"
  },

errorTests = describe(
  "Error Throwing Tests",
  [
    test(
      "throws any error",
      {|| expect({|| throw("SomeError")}).toThrowAnyError()}
    ),
    test("doesn't throw any error", {|| expect({|| 2 + 2}).toNotThrow()}),
    test(
      "fails when expecting an error but none is thrown",
      {|| expect({|| 2 + 2}).toThrowAnyError()}
    ),
    test(
      "fails when expecting no error but one is thrown",
      {|| expect({|| throw("UnexpectedError")}).toNotThrow()}
    ),
  ]
)
berekuk commented 1 month ago

Yeah, try(...) doesn't unwind the stack properly, minimal example where this is visible in the stacktrace:

Screenshot 2024-09-29 at 19 51 12