search

CABLE-LSM / benchcab

Tool for evaluation of CABLE land surface model
https://benchcab.readthedocs.io/en/latest/
Apache License 2.0
2 stars 3 forks source link

Use `multiprocessing.Pool.map()` for parallelism #182

Closed SeanBryan51 closed 11 months ago

SeanBryan51 commented 11 months ago

We currently do not exit with non-zero if an uncaught exception occurs within a multiprocessing child process.

multiprocessing.Pool.map() will kill all processes if any one process throws an exception and then re-raise that exception in the parent process.

Fixes #122

codecov[bot] commented 11 months ago

Codecov Report

Merging #182 (068e0bb) into main (107cc86) will increase coverage by 1.26%. Report is 1 commits behind head on main. The diff coverage is 50.00%.

@@            Coverage Diff             @@
##             main     #182      +/-   ##
==========================================
+ Coverage   87.10%   88.37%   +1.26%     
==========================================
  Files          25       25              
  Lines        1613     1591      -22     
==========================================
+ Hits         1405     1406       +1     
+ Misses        208      185      -23
Files Coverage Δ
benchcab/comparison.py 84.84% <50.00%> (+23.48%) :arrow_up:
benchcab/fluxsite.py 83.85% <50.00%> (+5.36%) :arrow_up:
SeanBryan51 commented 11 months ago

Integration tests

Test AU-Tum experiment with two branches

#!/bin/bash
bench_example_dir='bench_example_test_AU-Tum_experiment'
rm -rf $bench_example_dir
git clone git@github.com:CABLE-LSM/bench_example.git $bench_example_dir
cd $bench_example_dir
git reset --hard 6287539e96fc8ef36dc578201fbf9847314147fb
cat > config.yaml << EOL
project: tm70

experiment: AU-Tum

realisations:
  - path: trunk
  - path: branches/Users/sb8430/test-branch

modules: [
  intel-compiler/2021.1.1,
  netcdf/4.7.4,
  openmpi/4.1.0
]
EOL
benchcab run -v

Relevant output from PBS job script:

+ /home/189/sb8430/.local/bin/benchcab fluxsite-run-tasks --config=config.yaml -v
Running fluxsite tasks...
Running task AU-Tum_2002-2017_OzFlux_Met_R1_S3... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R1_S3/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S3_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S2... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R1_S2... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R1_S2/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S2_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R1_S0... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R1_S0/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S0_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S3... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S1... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S0... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R1_S1... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R1_S1/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S1_out.nc
Successfully ran fluxsite tasks

+ /home/189/sb8430/.local/bin/benchcab fluxsite-bitwise-cmp --config=config.yaml -v
Running comparison tasks...
Comparing files AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc and AU-Tum_2002-2017_OzFlux_Met_R1_S0_out.nc bitwise...
nccmp -df /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S0_out.nc
Success: files AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc AU-Tum_2002-2017_OzFlux_Met_R1_S0_out.nc are identical
Comparing files AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc and AU-Tum_2002-2017_OzFlux_Met_R1_S1_out.nc bitwise...
nccmp -df /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S1_out.nc
Success: files AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc AU-Tum_2002-2017_OzFlux_Met_R1_S1_out.nc are identical
Comparing files AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc and AU-Tum_2002-2017_OzFlux_Met_R1_S2_out.nc bitwise...
nccmp -df /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S2_out.nc
Success: files AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc AU-Tum_2002-2017_OzFlux_Met_R1_S2_out.nc are identical
Comparing files AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc and AU-Tum_2002-2017_OzFlux_Met_R1_S3_out.nc bitwise...
nccmp -df /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_AU-Tum_experiment/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S3_out.nc
Success: files AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc AU-Tum_2002-2017_OzFlux_Met_R1_S3_out.nc are identical
Successfully ran comparison tasks

======================================================================================
                  Resource Usage on 2023-10-16 12:40:00:
   Job Id:             98226767.gadi-pbs
   Project:            tm70
   Exit Status:        0
   Service Units:      1.02
   NCPUs Requested:    18                     NCPUs Used: 18              
                                           CPU Time Used: 00:11:27        
   Memory Requested:   30.0GB                Memory Used: 851.16MB        
   Walltime requested: 06:00:00            Walltime Used: 00:01:42        
   JobFS requested:    100.0MB                JobFS used: 0B              
======================================================================================

Test benchcab fails with bad setup

#!/bin/bash
bench_example_dir='bench_example_test_bad_setup'
rm -rf $bench_example_dir
git clone git@github.com:CABLE-LSM/bench_example.git $bench_example_dir
cd $bench_example_dir
git reset --hard 6287539e96fc8ef36dc578201fbf9847314147fb
cat > config.yaml << EOL
project: tm70

experiment: AU-Tum

realisations:
  - path: trunk

modules: [
  intel-compiler/2021.1.1,
  netcdf/4.7.4,
  openmpi/4.1.0
]
EOL
benchcab checkout -v && benchcab build -v && benchcab fluxsite-submit-job -v

Output from PBS job:

+ /home/189/sb8430/.local/bin/benchcab fluxsite-run-tasks --config=config.yaml -v
Running fluxsite tasks...
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S3... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_bad_setup/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3/out.txt
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S0... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_bad_setup/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0/out.txt
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S1... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_bad_setup/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1/out.txt
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S2... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_bad_setup/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2/out.txt
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/scratch/tm70/sb8430/conda/envs/benchcab-dev/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/scratch/tm70/sb8430/conda/envs/benchcab-dev/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/189/sb8430/.local/lib/python3.9/site-packages/benchcab/fluxsite.py", line 293, in run
    self.run_cable(verbose=verbose)
  File "/home/189/sb8430/.local/lib/python3.9/site-packages/benchcab/fluxsite.py", line 313, in run_cable
    with chdir(task_dir):
  File "/scratch/tm70/sb8430/conda/envs/benchcab-dev/lib/python3.9/contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "/home/189/sb8430/.local/lib/python3.9/site-packages/benchcab/utils/fs.py", line 13, in chdir
    os.chdir(newdir.expanduser())
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_bad_setup/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/189/sb8430/.local/bin/benchcab", line 8, in <module>
    sys.exit(main())
  File "/home/189/sb8430/.local/lib/python3.9/site-packages/benchcab/benchcab.py", line 324, in main
    app.main()
  File "/home/189/sb8430/.local/lib/python3.9/site-packages/benchcab/benchcab.py", line 309, in main
    self.fluxsite_run_tasks()
  File "/home/189/sb8430/.local/lib/python3.9/site-packages/benchcab/benchcab.py", line 235, in fluxsite_run_tasks
    run_tasks_in_parallel(tasks, n_processes=ncpus, verbose=self.args.verbose)
  File "/home/189/sb8430/.local/lib/python3.9/site-packages/benchcab/fluxsite.py", line 392, in run_tasks_in_parallel
    pool.map(run_task, tasks, chunksize=1)
  File "/scratch/tm70/sb8430/conda/envs/benchcab-dev/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/scratch/tm70/sb8430/conda/envs/benchcab-dev/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_bad_setup/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3'

======================================================================================
                  Resource Usage on 2023-10-16 17:11:41:
   Job Id:             98243615.gadi-pbs
   Project:            tm70
   Exit Status:        1
   Service Units:      0.06
   NCPUs Requested:    18                     NCPUs Used: 18              
                                           CPU Time Used: 00:00:03        
   Memory Requested:   30.0GB                Memory Used: 255.0MB         
   Walltime requested: 06:00:00            Walltime Used: 00:00:06        
   JobFS requested:    100.0MB                JobFS used: 0B              
======================================================================================
SeanBryan51 commented 11 months ago

@ccarouge the behaviour should still be the same but I've run a case that fails the comparison:

#!/bin/bash
bench_example_dir='bench_example_test_failed_bitwise_comparison'
rm -rf $bench_example_dir
git clone git@github.com:CABLE-LSM/bench_example.git $bench_example_dir
cd $bench_example_dir
git reset --hard 6287539e96fc8ef36dc578201fbf9847314147fb
cat > config.yaml << EOL
project: tm70

experiment: AU-Tum

realisations:
  - path: trunk
  - path: trunk
    name: trunk-patch
    patch:
      cable:
        cable_user:
          SSNOW_POTEV: P-M

modules: [
  intel-compiler/2021.1.1,
  netcdf/4.7.4,
  openmpi/4.1.0
]
EOL
benchcab run -v

Output from PBS job:

+ /home/189/sb8430/.local/bin/benchcab fluxsite-run-tasks --config=config.yaml -v
Running fluxsite tasks...
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S3... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S3/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R1_S3... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R1_S3/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S3_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R1_S2... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R1_S2/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S2_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S2... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S2/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R1_S1... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R1_S1/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S1_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S1... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S1/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R0_S0... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R0_S0/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc
Running task AU-Tum_2002-2017_OzFlux_Met_R1_S0... CABLE standard output saved in /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/tasks/AU-Tum_2002-2017_OzFlux_Met_R1_S0/out.txt
./cable cable.nml
Adding attributes to output file: /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S0_out.nc
Successfully ran fluxsite tasks

+ /home/189/sb8430/.local/bin/benchcab fluxsite-bitwise-cmp --config=config.yaml -v
Running comparison tasks...
Comparing files AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc and AU-Tum_2002-2017_OzFlux_Met_R1_S0_out.nc bitwise...
nccmp -df /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S0_out.nc
Failure: files AU-Tum_2002-2017_OzFlux_Met_R0_S0_out.nc AU-Tum_2002-2017_OzFlux_Met_R1_S0_out.nc differ. Results of diff have been written to /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/analysis/bitwise-comparisons/AU-Tum_2002-2017_OzFlux_Met_S0_R0_R1.txt
Comparing files AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc and AU-Tum_2002-2017_OzFlux_Met_R1_S2_out.nc bitwise...
nccmp -df /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S2_out.nc
Failure: files AU-Tum_2002-2017_OzFlux_Met_R0_S2_out.nc AU-Tum_2002-2017_OzFlux_Met_R1_S2_out.nc differ. Results of diff have been written to /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/analysis/bitwise-comparisons/AU-Tum_2002-2017_OzFlux_Met_S2_R0_R1.txt
Comparing files AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc and AU-Tum_2002-2017_OzFlux_Met_R1_S3_out.nc bitwise...
nccmp -df /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S3_out.nc
Failure: files AU-Tum_2002-2017_OzFlux_Met_R0_S3_out.nc AU-Tum_2002-2017_OzFlux_Met_R1_S3_out.nc differ. Results of diff have been written to /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/analysis/bitwise-comparisons/AU-Tum_2002-2017_OzFlux_Met_S3_R0_R1.txt
Comparing files AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc and AU-Tum_2002-2017_OzFlux_Met_R1_S1_out.nc bitwise...
nccmp -df /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/outputs/AU-Tum_2002-2017_OzFlux_Met_R1_S1_out.nc
Failure: files AU-Tum_2002-2017_OzFlux_Met_R0_S1_out.nc AU-Tum_2002-2017_OzFlux_Met_R1_S1_out.nc differ. Results of diff have been written to /scratch/tm70/sb8430/benchcab_integration_tests/bench_example_test_failed_bitwise_comparison/runs/fluxsite/analysis/bitwise-comparisons/AU-Tum_2002-2017_OzFlux_Met_S1_R0_R1.txt
Successfully ran comparison tasks

======================================================================================
                  Resource Usage on 2023-10-16 16:55:47:
   Job Id:             98242265.gadi-pbs
   Project:            tm70
   Exit Status:        0
   Service Units:      1.33
   NCPUs Requested:    18                     NCPUs Used: 18              
                                           CPU Time Used: 00:12:49        
   Memory Requested:   30.0GB                Memory Used: 4.35GB          
   Walltime requested: 06:00:00            Walltime Used: 00:02:13        
   JobFS requested:    100.0MB                JobFS used: 0B              
======================================================================================