[Feature]: Get CMOR log outputs printed in main `e3sm_to_cmip` log and stdout/stderr, instead of separate CMOR log

[tomvothecoder]

Debugging e3sm_to_cmip can be tedious if a CMOR error appears because e3sm_to_cmip does not capture and print the CMOR error directly in its own stderr/stdout or its log file.

Instead, a separate CMOR log is used to capture CMOR errors. This presents an issue of having to go through two logs and align timestamps to figure out when the CMOR error occurred in the stack trace.

The user will see CMOR errors that tell the user to check the CMOR log file, instead of just printing it out directly. Often times there are tons of CMOR logs for the same variables, so having to look up which CMOR log is the correct one also introduces friction in the debugging and development process.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/vo13/mambaforge/envs/e3sm_to_cmip_dev_115/lib/python3.11/site-packages/cmor/pywrapper.py", line 457, in axis
    return _cmor.axis(table_entry, units, l, coord_vals,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_cmor.CMORError: Problem with 'cmor.axis'. Please check the logfile (if defined).

Example of a CMOR log (e.g., cmor_logs/phalf.log_2023-09-21T09:51:15)

C Traceback:
In function: 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Logfile ./cmor_logs/phalf.log already exist.
! Renamed to: ./cmor_logs/phalf.log_2023-09-20T16:49:03
!
!!!!!!!!!!!!!!!!!!!!!!!!!

C Traceback:
! In function: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Could not find all axes for this variable. 
! Check that all your axes name matches the ones found in 
! the table Amon. 
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!

C Traceback:
! In function: cmor_treat_axis_values
! called from: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis atmosphere_sigma_coordinate (table: Amon) , detected value at:
! 0.000124 when valid_max is 0.000000
!
!!!!!!!!!!!!!!!!!!!!!!!!!

C Traceback:
! In function: cmor_treat_axis_values
! called from: cmor_axis
! 

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis atmosphere_sigma_coordinate (table: Amon) , detected value at:
! 0.000148 when valid_max is 0.000000
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Proposed Solution

Capture CMOR errors in stderr/stdout and record in single e3sm_to_cmip log file in order of the stack trace.

[tomvothecoder]

@TonyB9000 and @chengzhuzhang I'm sure this is a source of frustration for both of you as well. I captured it this as a GitHub issue that we should address in the future.

[TonyB9000]

@tomvothecoder @chengzhuzhang Thanks Tom! I always felt that the "--logdir LOGDIR" option was being ignored. Also (IMHO) error logs should be machine-readable (one-liners with helpful structure) amenable to subsequent collation and processing. These multi-colored, six-line outputs:

�[1;31;47m!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis atmosphere_sigma_coordinate (table: Amon) , detected value at:
! 0.000124 when valid_max is 0.000000
!
!!!!!!!!!!!!!!!!!!!!!!!!!�[0m

are very annoying. Not sure if that can be fixed, or relegated to a "very verbose" mode of output ("--splashy = True").