Give benchit users a way to see complete exception stack traces when something goes wrong

[jeremyfowers]

Right now, all exceptions are caught, and the stack trace is discarded. Save it to a file instead of discarding it.

[jeremyfowers]

@danielholanda I'm actually not sure how to test this. What's an example of a program that would throw an exception that would be caught during call_benchit() in analysis.py?

[jeremyfowers]

How to print the stack trace to a file: https://stackoverflow.com/questions/31636884/print-exception-with-stack-trace-to-file

[danielholanda]

program

@danielholanda I'm actually not sure how to test this. What's an example of a program that would throw an exception that would be caught during call_benchit() in analysis.py?

A script that contains multiple models but not all of them pass through all stages of analysis (e.g. one of the models does not conver to ONNX or FP16)

[jeremyfowers]

program

@danielholanda I'm actually not sure how to test this. What's an example of a program that would throw an exception that would be caught during call_benchit() in analysis.py?

A script that contains multiple models but not all of them pass through all stages of analysis (e.g. one of the models does not conver to ONNX or FP16)

@danielholanda that trigger a groqit exception, which we already do an ok job of catching and saving to a log file. But what about this code:

# This broad exception is ok since enumerating all exceptions is
    # not possible, as the tested software continuously evolves.
    except Exception as e:  # pylint: disable=broad-except
        util.stop_stdout_forward()
        model_info.status_message = f"Unknown benchit error: {e}"
        model_info.status_message_color = printing.Colors.WARNING

[jeremyfowers]

Ahhh ok this can be tested by having benchit() raise any exception that isn't a GroqFlow exception. For example if benchmarking fails.