search

E3SM-Project / e3sm_to_cmip

Tools to CMORize E3SM output
https://e3sm-to-cmip.readthedocs.io/en/latest/
MIT License
7 stars 7 forks source link

[Bug]: Segmentation fault due to `cmor.close` being called before CMORizing is complete #233

Closed tomvothecoder closed 11 months ago

tomvothecoder commented 11 months ago

What happened?

There is a for loop that loops over files by year and CMORizes. In PR #213, I moved the call to cmor.close() after each iteration of this for loop. This causescmor to reset the internal mapping ID for the cmor.variable object back to 0. As a result, the second iteration attempts to write to the same file as the first iteration, rather than having a distinct file.

Example:

1. For year 1 -> cmor_axis_id = 0 -> good to go
2. For year 2 -> cmor_axis_id = 0 -> seg fault (clashing IDs and files)

What did you expect to happen? Are there are possible answers you came across?

  1. Remove cmor.close() from VarHandler_cmor_write() https://github.com/E3SM-Project/e3sm_to_cmip/blob/30d40c19c9e9c81eb95fb127a25cf2eaaa5ee572/e3sm_to_cmip/cmor_handlers/handler.py#L676-L679
  2. Add cmor.close() at the end of VarHandler.cmorize(), after the for loop https://github.com/E3SM-Project/e3sm_to_cmip/blob/30d40c19c9e9c81eb95fb127a25cf2eaaa5ee572/e3sm_to_cmip/cmor_handlers/handler.py#L260-L262

Example:

1. For year 1 -> cmor_axis_id = 0 -> good to go
2. For year 2 -> cmor_axis_id = 1 -> good to go

Minimal Complete Verifiable Example (MVCE)

from e3sm_to_cmip.__main__ import E3SMtoCMIP

cmortables_dir = "/lcrc/group/e3sm/diagnostics/cmip6-cmor-tables/Tables"
input_dir = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test_unified_1.9.2rc1/v2.LR.historical_0201/post/atm/180x360_aave/ts/monthly/2yr"
dest_cmip = "/lcrc/group/e3sm/ac.forsyth2/zppy_test_complete_run_output/test_unified_1.9.2rc1/v2.LR.historical_0201/post/atm/180x360_aave/cmip_ts/monthly"
output_path = "/home/ac.tvo/E3SM-Project/e3sm_to_cmip/qa/12-18-23-zppy-seg"

args = [
    "--output-path",
    output_path,
    "--var-list",
    "pr, tas, rsds, rlds, rsus",
    "--realm",
    "atm",
    "--input-path",
    input_dir,
    "--user-metadata",
    "/home/ac.tvo/E3SM-Project/e3sm_to_cmip/qa/12-18-23-zppy-seg/default_metadata.json",
    "--num-proc",
    "12",
    "--tables-path",
    cmortables_dir,
    "--serial",
]

run = E3SMtoCMIP(args)

run.run()

Relevant log output

No response

Anything else we need to know?

I did not catch this bug in PR #213 because the end-to-end testing script only tests for 1 year.

Environment

e3sm_to_cmip=1.11.0 and master branch