Rework Nim's exception handling

[Araq]

Nim's exception handling is currently tied to Nim's garbage collector and every raised exception triggers an allocation. For embedded systems this is not optimal, see http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p0709r1.pdf for more details.

So, here is what I propose: Clearly distinguish between "bugs" and "errors". Bugs are not mapped to exceptions.

The builtin "bugs" are mapped to the new system.FatalError enum, runtime errors are mapped to system.Error. FatalErrors are usually not recovered from, unless a "supervisor" is used. (See the section about them.)

type
  FatalError* = enum         ## Programming bugs and other error
                             ## conditions most programs cannot
                             ## realistically recover from.
                             ## However, a "supervisor" can be
                             ## registered that does allow to
                             ## recover from them.
    IndexError,              ## index out of bounds
    FieldError,              ## invalid object field accessed
    RangeError,              ## value out of its valid range
    ReraiseError,            ## nothing to re-raise

    ObjectAssignmentError,   ## object assignment would lose data (object slicing problem)
    ObjectConversionError,   ## object is not a subtype of the given type
    MoveError,               ## object cannot be accessed as it was moved
    DivByZeroError,          ## divison by zero

    OverflowError,           ## integer arithmetic under- or overflow
    AccessViolationError,    ## segfault
    NilAccessError,          ## attempt to deref a 'nil' value
    AssertionError,          ## assertion failed. Is used to indicate wrong API usage
    DeadThreadError,         ## the thread cannot be accessed as its dead.
    LibraryError,            ## a DLL could not be loaded

    OutOfMemError,           ## the system ran out of heap space
    StackOverflowError,
    FloatInvalidOpError,
    FloatDivByZeroError,
    FloatOverflowError,
    FloatUnderflowError,
    FloatInexactError

  Error* = enum ## NOTE: Not yet the real, exhaustive list of possible errors!
    NoError,
    SyntaxError,
    IOError, EOFError, OSError,

    ResourceExhaustedError,
    KeyError,
    ValueError,
    ProtocolError,
    TimeoutError

proc isBug*(e: FatalError): bool = e <= AssertionError

Every error that can be caught is of the type Error. The effect system tracks if a proc can potentially raise such an error. This means that an "empty" except can be used to conveniently check for any meaningful exception, comparable in convenience to an if statement/expression:

let x = try: parseJson(input) except: quit(getCurrentExceptionMsg())

The runtime calls system.panic on a detected bug, which roughly has this implementation:

var panicHook*: proc (e: FatalError; msg: cstring) {.nimcall.}

proc panic(e: FatalError; msg: cstring) {.noreturn.} =
  if panicHook != nil:
    panicHook(e, msg)
  else:
    if msg != nil: stdout.write msg
    quit(ord(e)+1)

Out of memory

Out of memory is a fatal error.

Rationale: "Stack unwinding with destructors invocations that free memory" is too much gambling, if it works, that's mostly by chance as in more complex programs it becomes intractable.

The systems that are designed to continue gracefully in case of an OOM situation should make use of the proposed "supervisor" feature. The systems that are proven to work with a fixed amount of heap space should allocate this space upfront (the allocator already has some support for this) and then an OOM situation is a fatal, unrecoverable error too.

Implementation of 'try'

There are different implementation strategies possible for these light weight exceptions:

1. C setjmp. Since this is pretty expensive, this will not be used.
2. C++'s try statement. Will be used for the C++ target unless the C++ dialect does not support exceptions. This is likely to offer the best performance.
3. Map Error to an additional return value and propagate it explicitly in the generated code via if (retError) goto error;
4. Map Error to a hidden pointer parameter so and propagate it explicitly in the generated code via if (*errorParam) goto error;. This is probably slower than using a return value, but this requires benchmarking.

The supervisor

Fatal errors cannot be caught by try. For servers in particular or for embedded devices that lack an OS a "supervisor" can be used to make a subsystem shut down gracefully without shutting down the complete process. The supervisor feature requires extensive stdlib support; every open call needs to register the file descriptor to a threadlocal list so that they can be closed even in the case of a fatal error which does not run the destructors if any implementation strategy but (2) is used. Ideally also the memory allocator supports "safe points", so every allocation that was done by the failed subsystem can be rolled back. For the usual segregated free-list allocators this seems feasible to provide.

The supervisor sets the panicHook to a proc that does the raise or longjmp operation. The supervisor runs on the same thread as the supervised subsystem.

Rationale: Easier to implement, stack corruptions are super rare even in low level Nim code, embedded systems might not support threads. Erlang's supervisors conflate "recover from this subsystem" with a concurrency mechanism. This seems to be unnecessary.

A supervisor will be part of the stdlib, but one also can write his own. A possible implementation looks like:

template shield(body: untyped) =
  let oldPanicHook = panicHook
  var target {.global.}: JmpBuf
  panicHook = proc (e: FatalError; msg: cstring) =
    log(e, msg)
    longjmp(target)
  let oldHeap = heapSnapshot()
  var failed = false
  if setjmp(target) == 0:
    try:
      body
    except:
      failed = true
  else:
    failed = true
  if failed:
    heapRollback(oldHeap)
    closeFileDescs()
  panicHook = oldPanicHook

# in a typical web server, that must have some resistance against
# programming bugs, this would be used like so:
shield:
  handleWebRequest()

How well the heap can be rolled back and external resources can be freed is a quality of implementation issue, even in the best case, created files would not be removed on failure. Perfect isolation is not even provided by Erlang's supervisors, in fact, every write to a disk is not rolled back! The supervisor feature is not a substitute for virtualization.

[cheatfate]

I think LibraryError must not be fatal error. Of course in most of the cases it is fatal error, but for projects which can embed DLLs into code at compile time it will be a problem. So to implement library which can enable such feature for any Nim project, you will need to implement supervisor.

[Araq]

@cheatfate This is nasty to solve since DLLs are loaded on program start, usually before the code runs that could override any "DLL loading hook" that we may want to introduce. This is a problem for another time. Currently the failure to load a DLL is a fatal error already and this proposal does not change that.

[Skrylar]

Doesn't using an enum for this brick user-defined errors?

[andreaferretti]

@Araq It is nice to have a more lightweight exception mechanism. I must say I have not read all your proposal, but two things stand out:

1. It is often useful to use custom exceptions. This is why exceptions are usually ref types. Custom exceptions can communicate more specific conditions (say HttpHeaderError vs ProtocolError) and can also have custom fields, which are fundamental to preserve some information that may be needed during recovery - for instance HttpHeaderError may have a field to specify which header was invalid in a response.

2. I don't agree with your classification of FatalError vs Error. Most errors that you list as fatals are actually very much recoverable:

var stat: float
try:
  stat = computeStatWithFastButNumericallyUnstableAlgorithm()
except FloatUnderflowError, FloatOverflowError:
  stat = computeStatWithSlowButNumericallyStableAlgorithm()

In your list I would consider at least the following recoverable: IndexError, FieldError, RangeError, ObjectConversionError, DivByZeroError, OverflowError, NilAccessError, LibraryError, FloatInvalidOpError, FloatDivByZeroError, FloatOverflowError, FloatUnderflowError, FloatInexactError.

[Araq]

1. Additional error information can be stored in thread local variables, this is how it's done under the hood anyway. There is very little benefit in hiding this fact. "Custom" exceptions are just sophistry and work against composable systems, from the paper:

Propagating arbitrarily typed exceptions breaks encapsulation by leaking implementation detail types from lower levels. (“Huh? What’s a bsafe::invalid_key? All I did was call PrintText!”)

Propagating arbitrarily typed exceptions is not composable, and is often lossy. It is already true that intermediate code should translate lower-level exception types to higher-level types as the exception moves across a semantic boundary where the error reporting type changes, so as to preserve correct semantic meaning at each level. In practice, however, programmers almost never do that consistently, and this has been observed in every language with typed propagated exceptions that I know of, including Java, C#, and Objective-C (see [Squires 2017] for entertaining discussion).

In fact, the "custom" exception DbError is worse than the more generic SyntaxError (ah, so the SQL query was illformed) vs ProtocolError (ah, so the client doesn't understand the bytes on the wire).

2. That cannot work in practice. If I write .raises: [] and the compiler tells me "no, you use array indexing which can throw IndexError" I have to throw away the compiler. I also don't want to catch this bug with try except, try except is for handling errors, not bugs. Protection against bugs is the business of the supervisor.

[Araq]

Doesn't using an enum for this brick user-defined errors?

That's a benefit in my opinion, Posix uses about 100 different error codes and does fine. In fact, every exception hierarchy also requires you to map your custom error to some existing error as you have to answer the question of which subtype of exception to inherit from.

[andreaferretti]

1. Using thread local variables is messy - like global variables you have the issue of choosing unique names, making sure you are not writing over something already used. This is a completely unstructured way of communicating errors. That the compiler does this under the hood is of little relevance - the compiler is there to help and translate a mantainable pattern into whatever is needed to make the thing work on actual hardware. The problem is when a human has to manually do this.

2. All the things I listed may genuinely be not bugs. I made an example with FloatOverflowError, but an example can be made for all of these cases.

Maybe I have a data structure like an array with holes. You can ask to get myDataStructure[i] for an integer i, but - unlike normal arrays - there is no simple rule to figure out whether index i is accessible, such as 0 <= i and i < len(myDataStrcuture). So if you are not sure you can do

let x = try:
  myDataStructure[i]
except IndexError:
  "no value"

Not to mention multithreading and the shared heap. For data stored on the thread-local heap, one can do

if 0 <= i and i < len(myArray):
  x = myArray[i]

In a multithreaded setting, this is a race condition, and the only safe way to do this may be (assume myArray is some kind of data structure stored on the shared heap but is not thread safe)

let x = try:
  myArray[i]
except IndexError, MoveError:
  "no value"

All the cases I have listed can be used for actual runtime errors that are not programming bugs.

Moreover, sometimes bugs do happen, and a programmer may want to use exceptions to make sure that a bug will not crash the whole program after running for a week. You propose the supervisor for this, but in your proposal the supervisor is global. One may want to be able to recover from errors at various levels of granularity

for file in files:
  try:
    let requests = readRequestsFrom(file)
    for request in requests:
      try:
        make(request)
      except:
        discard # go on with the next request
    except:
      discard # go on with the next file

[andreaferretti]

To make sure: I like your proposal, and if there is a way to make exception more lightweight and usable on embedded targets, I will be happy.

Nevertheless, I wouldn't like to make exceptions less useful than they are today

[andreaferretti]

answer the question of which subtype of exception to inherit from.

Exception, if nothing seems more specific

[andreaferretti]

Propagating arbitrarily typed exceptions is not composable, and is often lossy. It is already true that intermediate code should translate lower-level exception types to higher-level types as the exception moves across a semantic boundary where the error reporting type changes, so as to preserve correct semantic meaning at each level. In practice, however, programmers almost never do that consistently

No problem

# library.nim
type MyCustomError = ref object of IoError
...

# client.nim
# if I know about MyCustomError:
try:
  doStuff()
except MyCustomError:
  # let's handle this

# if I don't know what is going on under the hood:
try:
  doStuff()
except IoError:
  # let's handle this

[Araq]

All the things I listed may genuinely be not bugs.

True, but they are all pervasive, every floating point operation could trigger it. So then the fact that my proc uses floating point at all would be reported as .raises: [FloatInvalidOpError, FloatDivByZeroError, FloatOverflowError, FloatUnderflowError, FloatInexactError]. It's terrible and no wide-spread programming language does it this way.

let x = try:
  myDataStructure[i]
except IndexError:
  "no value"

is already invalid code. Exactly this example is mentioned in the existing spec.

You propose the supervisor for this, but in your proposal the supervisor is global.

It's not, it can be nested.

let x = try:
  myArray[i]
except IndexError, MoveError:
  "no value"

That sounds like a data structure where [] can raise an exception. This would probably raise KeyError then which you can catch.

Using thread local variables is messy - like global variables you have the issue of choosing unique names, making sure you are not writing over something already used.

Nim has a module system for that.

This is a completely unstructured way of communicating errors.

Yes, as that's used as a debugging help most of the time. APIs are free to clearly communicate errors in a "structured" way. And no, error messages in strings are not structured either and have serious i18n problems.

That the compiler does this under the hood is of little relevance - the compiler is there to help and translate a mantainable pattern into whatever is needed to make the thing work on actual hardware. The problem is when a human has to manually do this.

The pattern of attaching arbitrary stuff to an arbitrary exception subclass is not structured. Currently we attach a stack trace to a raised exception which is then gone in release builds. It's a debugging aid, nothing more.

In practice, however, programmers almost never do that consistently

No problem

You cannot argue with toy examples against what has been experienced in practice, in the real world with multiple different programming languages.

[ghost]

I love this RFC! One thing about fatal errors: stdlib would need some work to add procedures like getOrDefault (or other procedures with default value) for arrays, sequences and other data types.

[andreaferretti]

True, but they are all pervasive, every floating point operation could trigger it. So then the fact that my proc uses floating point at all would be reported as .raises: [FloatInvalidOpError, FloatDivByZeroError, FloatOverflowError, FloatUnderflowError, FloatInexactError]. It's terrible and no wide-spread programming language does it this way.

You are right, I wouldn't want the effect system to be littered by that. On the other hand, I would like to catch them! A division by zero, maybe hidden inside a library over whose computations I have no control, cannot be a fatal error!

Java distinguishes between checked and unchecked exceptions - maybe the same could makes sense here.

Exactly this example is mentioned in the existing spec.

Which spec? I do not see this example

Yes, as that's used as a debugging help most of the time. APIs are free to clearly communicate errors in a "structured" way. And no, error messages in strings are not structured either and have serious i18n problems.KeyError then which you can catch.

Well, by an actual exception can be a struct with fields which have useful information, I never said to use strings for that

The pattern of attaching arbitrary stuff to an arbitrary exception subclass is not structured

Well, how else would I programatically recover from the error if I do not know the context? Your proposal is to use thread local variables instead: why not find a pattern where the compiler can translate from something ergonomic like current exceptions to thread locals? For instance, with the restriction that only one exception of a type can be raised at a time.

You cannot argue with toy examples against what has been experienced in practice, in the real world with multiple different programming languages.

Fair enough, but the point is that one does not really need to handle the exact error raised in a library, a supertype will do.

[dom96]

As far as I'm concerned this diminishes exceptions in Nim greatly by removing their ability to be extended. Like I've said in the forum: I use exceptions very heavily and love them dearly. Please do not diminish them.

The rationale for this change seems to be embedded systems, but how many of Nim's users are actually targeting embedded systems? How many of those users are limited by Nim's exceptions? I have never heard anyone complain about Nim's exceptions so to me this seems like an almost theoretical problem with a solution that is very disruptive. Why don't you instead make it possible to raise stack-allocated objects (as well as heap-allocated objects)? Wouldn't that solve this problem too?

[Araq]

If we decide on a size limit for the largest possible exception type (say, 128 bytes), we can statically allocate the exception data. But then that's not a ref type anymore. All the other points my RFC addresses would remain:

• Can you write try removeFile except OSError? Or is that try removeFile except OSError, IOError? How can I write "do this when 'removeFile' fails" concisely? Note, that I'm not interested in handling the bugs (!) that removeFile might have.

• How can I ensure all errors (and not the bugs!) are handled exhaustively and in a meaningful way? An enum makes that easier than an open exception hierarchy.

• It enforces a clear methodology on how error handling needs to be done, map the errors to the existing error values, map bugs to AssertionError. No pointless inherited GoogleAppEngineError exceptions that are much more of an implementation detail than a clear ProtocolError.

[Varriount]

I've already mentioned this on IRC, but would like to add it here for visibility - chief situations in which extended information is useful are HTTP wrapper libraries and RPC libraries. HTTP protocols in particular don't always map their errors to HTTP codes. I've wrapped protocols that always return success, but put error information in the response body.

Furthermore, while extended error information is mostly used for debugging, it's also vital when analyzing crashes in production systems. While one can get around the lack of extended information by printing contextual information on the exception handler, this tends to make code bloat up with logging messages everywhere.

The C++ proposal provides an escape hatch for this - there is a "payload" part of the error structure that can point to another, larger structure containing more information.

[markprocess]

For reference, I'd like to point out that a class hierarchy for exceptions seems to confuse implementation inheritance and except: classification. For example, an imaginary FileError could have a name field that is inherited by FilePermissionError that adds detail about the actual permissions. Also, it would likely have overloaded methods. On the other hand, an except: FileError clause is operating on the classification of the exception, which might be entirely different from the implementation hierarchy.

Again, error handling is hard. Everyday programming (i.e. excluding security, concurrency, and versioning) would be trivial if it weren't for error handling.

Regarding the proposal - what can except: take? Is it limited to FatalError and Error?

[Araq]

I've wrapped protocols that always return success, but put error information in the response body.

Irrelevant. You need to write extra logic to map that to an error then. You don't need an "extensible" error in order to be able to do this.

While one can get around the lack of extended information by printing contextual information on the exception handler, this tends to make code bloat up with logging messages everywhere.

You don't have to "print" this information. You can return extended error information. Either in the return value, in an output parameter or in a thread local variable. Or you keep it in the "context" parameter that every library ends up having for other reasons (re-entrancy).

[markprocess]

proc isBug*(e: FatalError): bool = e <= AssertionError

This needs to be renamed isDefect

[andreaferretti]

Can you write try removeFile except OSError? Or is that try removeFile except OSError, IOError?

I don't know, but the effect system tracks this, so I can find out by either the documentation or an IDE

How can I ensure all errors (and not the bugs!) are handled exhaustively and in a meaningful way?

Surely not by compiling a list of all possible errors and hardcondig it into an enum. The world is varied, and the errors are as well. I cannot see how enforcing some typing on that (albeit dynamic) can be bad

It enforces a clear methodology on how error handling needs to be done, map the errors to the existing error values, map bugs to AssertionError

Unfortunately, many legitimate errors are also mapped to fatal in your proposal

[mikra01]

I also like this rfc. @dom96 with Nim I target both tiny and huge envs. I've planned to get some nim code running on the ATtiny....

[Araq]

I don't know, but the effect system tracks this, so I can find out by either the documentation or an IDE

https://nim-lang.org/docs/os.html#removeFile,string only lists OSError. existsOrCreateDir lists OSError, IOError. The effect system tracks implementation details and I'd better catch IOError too when calling removeFile, maybe.

[Araq]

Unfortunately, many legitimate errors are also mapped to fatal in your proposal

These cannot be caught in today's Nim either, these all come from checks that can be disabled.

Surely not by compiling a list of all possible errors and hardcondig it into an enum. The world is varied, and the errors are as well. I cannot see how enforcing some typing on that (albeit dynamic) can be bad

Sorry, but that's like saying "the world is dynamic, we need dynamic typing, static typing won't do".

[andreaferretti]

https://nim-lang.org/docs/os.html#removeFile,string only lists OSError. existsOrCreateDir lists OSError, IOError. The effect system tracks implementation details and I'd better catch IOError too when calling removeFile, maybe.

Sorry, I am a bit lost: if removeFile only lists OSError why do you insist in catching IOError?

These cannot be caught in today's Nim either

At least some of your fatal errors can

for x in 0 .. 10:
  try:
    echo 100 div (5 - x)
  except DivByZeroError:
    echo "can catch DivByZeroError"

let s = @[1, 2, 3, 4]

try:
  echo s[5]
except IndexError:
  echo "can catch IndexError"

[markprocess]

What can except: take? Is it limited to FatalError and Error?

Can I stil use try/except/raise to do an intentional stack-unwind (e.g. ForumError in nimforum?)

If all exceptions are suffixed with Error, why isn't it called try/error instead of try/except? Why isn't this called reworking errors instead of reworking exceptions?

Can you implement try/except alongside try/error? (error being this proposal)

[dom96]

Can you write try removeFile except OSError? Or is that try removeFile except OSError, IOError? How can I write "do this when 'removeFile' fails" concisely? Note, that I'm not interested in handling the bugs (!) that removeFile might have.

This is just a case of convention. removeFile should specify in its documentation that it will raise an IOError if an actual error occurs. It should wrap OSErrors into an IOError and then everything else can be considered a bug.

This is how I deal with errors in Nimble, choosenim and pretty much all my software. Nimble defines a NimbleError which is basically a "unrecoverable error", i.e. quit Nimble now and show the user an error message. The fact that this is an exception has many advantages:

• I can tell where this error was raised very easily.
• Users of my code can handle this error and show it in an appropriate fashion: CLI message or a GUI window.

How can I ensure all errors (and not the bugs!) are handled exhaustively and in a meaningful way? An enum makes that easier than an open exception hierarchy.

Do it in the same way you are with your enum? Give each exception type a field called isBug? This issue is IMO completely orthogonal to the problem of "ref exceptions".

[Varriount]

I've wrapped protocols that always return success, but put error information in the response body.

Irrelevant. You need to write extra logic to map that to an error then. You don't need an "extensible" error in order to be able to do this.

Mapping only works if the following conditions are met:

• There are values accurate enough to map to.
• The values to map are known ahead of time.

Given the above RFC, the values that one can map to are inherently limited - I have no way of declaring my own additions to the FatalError enum.

Take the following pseudo-code, which reflects some real-world code I have had to write:

import json

proc makeOrGetSubnet(client: Client, cidrBlock: string): Subnet =
  try:
    let
      request = makeRequest(client, {"action": "makeSubnet", "cidr": cidrBlock})
      data = request.json
  except ClientError:
    let exception = getCurrentException()
    if exception.message == "alreadyExists":
      return getSubnet(client, cidrBlock)
    else:
      raise
  # Turn `data` into `Subnet` ...

Below is how the code would need to be rewritten to use the new exception system:

type
  ClientResult[T] = object
  case isError: bool
  of true:
   errorData: JsonNode
 of false:
   res: T

proc makeOrGetSubnet(client: Client, cidrBlock: string): ClientResult[Subnet] =
  # Note - users will have to deal both with IOError through try, and ClientResult
  let request = makeRequest(client, {"action": "makeSubnet", "cidr": cidrBlock})
  if hasKey(request.json, "error"):
    if getErrorMessage(request.json) == "alreadyExists":
      result = getSubnet(client, cidrBlock) # Will also be a ClientResult
    else:
      result = ClientResult(isError: true, errorData: request.json)

Note that the protocol the above client uses has the following characteristics:

• Error codes are added over time (not evident from the code)
• Error data is sent as a JSON body from the request.

What has to be done under the proposed exception system is that any extra information about the error must be sent through a object variant return value (or a thread local variable, etc). The exception system can't be used, because translating ClientError into IOError causes too much loss of information - one can't tell whether an IOError occurred because of a bad connection, or a missing file. Furthermore, since the Error enum is static, one can't even add an error to the system, such as FileMissingError.

[andreaferretti]

If I understand correctly, this proposal conflates (at least) four concerns:

1. Exceptions are currently ref types, and there is no mechanism to raise stack or fixed size objects
2. Distinguishing which exceptions signal bugs and which ones runtime errors
3. Exceptions currently use setjmp, which is costly
4. Introducing a supervisor mechanism

Maybe it would be easier to treat these issues separately.

As one data point, I welcome 2 (possibly with an additional field, as @dom96 proposed) and 4. I also would like very much to find a solution to 1, but I think some discussion is needed to find a mechanism that is less limited than the one proposed here. Regarding 3, I am not in the position of making informed comments

[Araq]

Sorry, I am a bit lost: if removeFile only lists OSError why do you insist in catching IOError?

Ok, to put it differently, why can existsOrCreateDir raise IOError?

Or take this https://nim-lang.org/docs/asyncdispatch.html#poll,int example. Here is its declaration raises: [Exception, ValueError, OSError, FutureError, IndexError]. IndexError is a programming bug and doesn't have to be caught. It's unclear what a FutureError is. When would ValueError be raised? I doubt I would catch that. And then there is Exception which means Nim's exception tracking had to give up. Here is my guess what I would really need to write in order to catch its errors, but not its bugs:

try:
  poll()
except OSError, IOError:
  echo "error ", getCurrentExceptionMsg()

And that was an educated guess, and I would have guessed the same without Nim's exception tracking mechanism.

And here is what I would write if my RFC becomes a reality:

try: 
  poll()
except:
  echo "error ", getCurrentExceptionMsg()

And it didn't require the guesswork, errors are mapped to exceptions, bugs are not.

[Araq]

What can except: take? Is it limited to FatalError and Error?

It is in fact limited to Error.

Can I stil use try/except/raise to do an intentional stack-unwind (e.g. ForumError in nimforum?)

Yes.

If all exceptions are suffixed with Error, why isn't it called try/error instead of try/except? Why isn't this called reworking errors instead of reworking exceptions? Can you implement try/except alongside try/error? (error being this proposal)

You can catch Error with try and you can catch FatalError with shield.

[andreaferretti]

@Araq This is a better example - I agree that the current system does poorly on poll.

[Araq]

Mapping only works if the following conditions are met: There are values accurate enough to map to.

I think we can come up with a list of errors that is detailed enough.

The values to map are known ahead of time.

We can introduce an UnknownError for this.

[markprocess]

Can I stil use try/except/raise to do an intentional stack-unwind (e.g. ForumError in nimforum?)

Yes

Intentional stack-unwind is by definition not an Error, it is intentional control-flow. What actual value would be used in the raise? One that you listed above? Which one? I had the impression that exceptions were no longer available except for errors.

[markprocess]

UnspecifiedError is more accurate than UnknownError. At the point of raise, it is well-known and understood. It just doesn't provide any more specifics to the except: clause.

[Araq]

Intentional stack-unwind is by definition not an Error, it is intentional control-flow. What actual value would be used in the raise? One that you listed above? Which one?

Some value that makes sense. LoginError if there is a login failure, ProtocolError for an HTTP error, etc. Why should it be wrapped in a ForumError? What the heck is a ForumError? ;-)

[Araq]

Below is how the code would need to be rewritten to use the new exception system:

No, I think you misinterpret my proposal, much of your try except logic would be the same.

[markprocess]

Some value that makes sense. LoginError if there is a login failure, ProtocolError for an HTTP error, etc. Why should it be wrapped in a ForumError? What the heck is a ForumError? ;-)

https://forum.nim-lang.org/t/4044#25243 refactored the ForumError into an intentional CompletionException. The design of ForumError contains functional information, not diagnostic, that is passed back to the browser for presentation to the user.

There needs to be an Error that isn't an error, one that is for an intentional stack-unwind. The older frames on the stack will trap and check their data to see if the unwind was directed to them.

[andreaferretti]

It may be a completely silly idea - but what about a typed try that works with any enum?

The idea would be the following. Say I have a library that likes to define its own silly exception types. Great, let's define this as an enum:

type ForumError = enum
  LoginError, NetworkError, DbConnectionError, InvalidFormattingError

Then I can raise this enum:

proc foo(user: User) =
  if not authenticate(user):
    raise LoginError

In the place where I catch this, I can write

try:
  foo(user)
except LoginError:
 echo "failed login"

The compiler recognizes that I am trying to catch something of type ForumError, so it makes exactly the thing specified by Araq, but using our custom enum type instead of the default Error type.

Pros:

• no additional runtime cost wrt to the proposal of @Araq
• users can use custom errors
• no need to fit errors into hierarchys

Cons:

• cannot catch errors of different types in the same clause (say a LoginError together with a KeyError)

So authors of libraries are kind of forced to choose between:

• reusing the base Error enum
• providing their custom errors, in which case they have to translate all errors coming from lower levels into their own new enum

To me it seems that this mechanism would have the benefits described by Araq without losing extensibility, with the only price of using type inference to detect "different" types of except clauses.

With a little more support from the compiler, one could also raise and catch object variants: proposal is as above, but the compiler only propagates the enum in the discriminator and stores the actual object variant into a thread local variable

[Araq]

It may be a completely silly idea - but what about a typed try that works with any enum?

Ha, I was about to modify my proposal to allow just this. This "everything is not extensible" point is the least important part of the RFC.

providing their custom errors, in which case they have to translate all errors coming from lower levels into their own new enum

And that means the different "layers" in an application are enforced. It's great. (Unless it turns out to be a pita in practice.)

[andreaferretti]

This "everything is not extensible" point is the least important part of the RFC.

But the most controversial one! :-D

[markprocess]

It may be a completely silly idea - but what about a typed try that works with any enum?

This also permits try/except to handle exceptions and not just errors. (This might be at odds with the paper, but useful for intentional stack-unwind.)

[dom96]

@andreaferretti just to be clear, the proposal is to allow enums to be raised in addition to heap-allocated refs?

[andreaferretti]

@dom96 No, my proposal is to follow what Araq is saying, but recovering at least some of the flexibility we have today by allowing any custom enum instead of the hardcoded Error

Another extension would be the following:

• allow to raise any variant object
• the compiler actually propagates the discriminator only, which is of an enum type
• and stores the actual variant into a generated thread local variable

This would recover a lot of benefits:

• one can raise stack objects and discriminate them by just propagating an enum
• the context is not lost
• if one wants to store heap allocated data into the variant object, this is always possible, but this cost is not necessarily paid by everyone

[andreaferretti]

type
  ForumErrorKind = enum
    HttpError, NetworkError
  ForumError = object
    case kind: ForumError
    of HttpError:
      statusCode: int
    of NetworkError:
      discard

proc foo() =
  raise ForumError(kind: HttpError, statusCode: 404)

try:
  foo()
except HttpError:
  let e = getCurrentException() # infers ForumError
  echo e.statusCode

can be translated by the compiler into something like

var ForumErrorCurrentException {.thread.}

proc foo() =
  ForumErrorCurrentException = ForumError(kind: HttpError, statusCode: 404)
  raise HttpError

try:
  foo()
except HttpError:
  let e = ForumErrorCurrentException
  echo e.statusCode

This recovers at least enough functionality to be usable for me and does not require to allocate anything on the heap

This "extended" proposal may even be implementable with macros, even if possibly not in a very convenient syntax

[dom96]

Ahh, that example is much clearer. I'm happy with this, but let's go a step further and just implement what I proposed above: the ability to raise all stack-allocated objects. Even objects that aren't variants. The ForumError which you are using as an example is currently just one kind of error. I don't want to have to create a dummy variant just to appease the compiler.

---

As an aside, are there any valid use cases for an exception hierarchy that we will miss out on by implementing this proposal? If so we should at least discuss them.

[zah]

I see this discussion is already going in the right direction, but I'll provide my two cents.

The main distinction between recoverable errors and bugs is that the former represent situations that are planned for by the developer. A user may have entered incorrect data (potentially malicious one), a network connection may be interrupted, or an important data file may be missing. With such errors, the developer must decide what the policy of the software should be because they are expected to arise even in a perfectly implemented program. Bugs on the other hand are unexpected and there can't be any reasonable code that discriminates between the types of bugs that were encountered.

So, the recoverable errors will be domain-specific and each type of problem is likely to have a different work-around policy implemented in the code. For this reason, the error must be encoded in a precise way and not mapped to a general enum that doesn't allow you to implement these different "work-around policies".

Araq seems to motivated by the efficiency arguments suggested by the C++ paper, but even the paper has a slightly different solution that is still user-extensible. The error type described in the paper has two numeric fields: error_category and error_code. error_category is something that can take values such as "nim std error", "posix error", "windows error", etc. It can also take values such as "ID of user enum E". Through this mechanism and the high-level sugar suggested by @andreaferretti, you can have your cake and eat it too. error_code may also store a pointer in certain situations to regain more of the current exceptions capabilities.

Otherwise, I think we can find ways to keep the support for error types with fields as a high-level syntax sugar. It was already mentioned that an upper limit to the size of the error types may be one solution. You can also allocate memory on the stack just prior to raising the exception (in the C++ paper, it's mentioned that this is the implementation strategy of MSVC on Windows).

How many of you have read my additional proposals for high-level sugar: https://gist.github.com/zah/d2d729b39d95a1dfedf8183ca35043b3

Some of the ideas are already incorporated here, but if we are planning to break backward compatibility, we can also teach try some new tricks as suggested in the proposal.

[markprocess]

On the road from here to there, can we get a compiler switch that warns or fails on the following:

• Create an Exception that is not a leaf in the hierarchy (all internal nodes are abstract.)

  • In particular an UnspecifiedError would likely be useful because except: Exception is ambiguous (does it mean any exception subtype or does it mean a concrete instance of Exception used in a general way?) The former would then be excluded.

• Likewise except: can only refer to leaves.

• Define an exception type that includes fields (i.e. definitely can't be replaced with an enum.)

We could start to refactor nim in the wild and see where it goes.

[andreaferretti]

let's go a step further and just implement what I proposed above: the ability to raise all stack-allocated objects. Even objects that aren't variants

Fine, but the problem is what one uses at runtime to propagate what actual error happened.

In @Araq proposal, this is just a member of the Error enum.

I proposed to use any enum, but then the try .. except clause will be typed due to this enum. This means one cannot mix members of different enums and catch them in the same clause. If one follows this, the ability to extend it to variant objects is right behind the corner.

Raising any stack object would be fine, but the issue would be: how do you catch it? It could be done if the is just one except clause, in which case there is no problem.

But if there is more than one except clause, the problem is where to store the runtime data to distinguish the various cases. Without using either the heap or variant objects I do not see a good way.

[Araq]

As an aside, are there any valid use cases for an exception hierarchy that we will miss out on by implementing this proposal?

Well depending on how it's done, you lose much of the type propagation properties:

proc dontCareAboutExceptions =
  subsystemThatCanRaiseForumError()
  subsystemThatCanRaiseSystemError()
# --> Error: dontCareAboutExceptions can raise 'ForumError' but also 'SystemError'. 

Of course, this can also be seen as a benefit, it forces 'dontCareAboutExceptions' to either translate from ForumError to 'system.Error' or vice versa.

[Varriount]

@andreaferretti Wouldn't it be possible for the compiler to automatically create an object variant to hold the different values?