Polished messaging to standard output

ccarouge commented 1 year ago

Currently, benchcab outputs stdout from other commands such as svn and ./build3.sh, but does not output anything when running ./cable or when generating particular directory structures. This can be improved by having benchcab print helpful messages to stdout so that the user is made aware of what benchcab is doing.

SeanBryan51 commented 1 year ago

Draft requirements

Draft requirements for what stdout should look like for various commands

When checking out CABLE via `svn`:

by default

$ benchcab checkout
Creating src directory: /absolute/path/to/src
Checking out repositories...
Successfully checked out <branch_name> at revision <revision>
Writing revision number info to rev_number-0.log

with verbose -v option:

print all output from the non-verbose case
print the svn command used to checkout repo

print all output from the svn command

$ benchcab checkout -v
Creating src directory: /absolute/path/to/src
Checking out repositories...
svn checkout https://trac.nci.org.au/svn/cable/<path_to_branch> src/<branch_name>
A    src/<branch_name>/offline
A    src/<branch_name>/offline/cable_mpiworker.F90
A    src/<branch_name>/offline/serial_cable
A    src/<branch_name>/offline/cable_LUC_EXPT.F90
... (this ellipsis won't be here, this is just to reduce the amount of text in the draft)
A    src/<branch_name>/util/cable_common.F90
A    src/<branch_name>/util/masks_cbl.F90
A    src/<branch_name>/util/cable_climate_type_mod.F90
U   src/<branch_name>
Checked out revision <revision>.
Successfully checked out <branch_name> at revision <revision>
Writing revision number in rev_number-0.log

When building CABLE

by default

$ benchcab build
Compiling CABLE <"with MPI" if MPI else "serially"> for realisation <branch_name>...
Successfully compiled CABLE for realisation <branch_name>

with verbose -v option:

print all output from the non-verbose case
print the command used to build

print all output from the build command


$ benchcab build -v
Compiling CABLE <"with MPI" if MPI else "serially"> for realisation <branch_name>...
./build3.sh

Building cable (serial)

Build setup complete. Compiling now ...

Building from source common across serial and MPI applications

Building drivers for either serial or MPI application

ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c grid_constants_cbl.F90 ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_maths_constants_mod.F90 ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c cable_phys_constants_mod.F90 ... (this ellipsis won't be here, this is just to reduce the amount of text in the draft) ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_feedback.F90 ifort -O2 -fp-model precise -I/apps/netcdf/4.7.4/include/Intel -c casa_sumcflux.F90

BUILD OK

Successfully compiled CABLE for realisation


### When creating directories
by default

$ benchcab fluxnet-setup-work-dir Setting up run directory tree for FLUXNET tests... Creating runs/site/logs directory: /absolute/path/to/runs/logs Creating runs/site/outputs directory: /absolute/path/to/runs/site/outputs Creating runs/site/tasks directory: /absolute/path/to/runs/site/tasks Creating task directories... Setting up tasks... Successfully setup FLUXNET tasks

with verbose `-v` option:
- print all output from the non-verbose case
- print paths to all directories created
- make all file manipulations explicit

$ benchcab fluxnet-setup-work-dir -v Setting up run directory tree for FLUXNET tests... Creating runs/site/logs directory: /absolute/path/to/runs/logs Creating runs/site/outputs directory: /absolute/path/to/runs/site/outputs Creating runs/site/tasks directory: /absolute/path/to/runs/site/tasks Creating task directories... Creating runs/site/tasks/: /absolute/path/to/runs/site/tasks/ Creating runs/site/tasks/: /absolute/path/to/runs/site/tasks/ ... (this ellipsis won't be here, this is just to reduce the amount of text in the draft) Creating runs/site/tasks/: /absolute/path/to/runs/site/tasks/ Setting up tasks... Setting up task: Cleaning task Copying namelist files from /absolute/path/to/namelists to /absolute/path/to/runs/site/tasks/ Copying CABLE executable from /absolute/path/to/ to /absolute/path/to/runs/site/tasks/ Adjusting namelist file: applying patch to CABLE namelist file /absolute/path/to/runs/site/tasks//cable.nml Setting up task: Cleaning task Copying namelist files from /absolute/path/to/namelists to /absolute/path/to/runs/site/tasks/ Copying CABLE executable from /absolute/path/to/ to /absolute/path/to/runs/site/tasks/ Adjusting namelist file: applying patch to CABLE namelist file /absolute/path/to/runs/site/tasks//cable.nml ... (this ellipsis won't be here, this is just to reduce the amount of text in the draft) Setting up task: Cleaning task Copying namelist files from /absolute/path/to/namelists to /absolute/path/to/runs/site/tasks/ Copying CABLE executable from /absolute/path/to/ to /absolute/path/to/runs/site/tasks/ Adjusting namelist file: applying patch to CABLE namelist file /absolute/path/to/runs/site/tasks//cable.nml Successfully setup FLUXNET tasks


### When running FLUXNET tasks (job submission)
by default

$ benchcab fluxnet-run-tasks Creating PBS job script to run FLUXNET tasks on compute nodes: benchmark_cable_qsub.sh PBS job submitted: .gadi-pbs The CABLE log file for each task is written to run/site/logs/_log.txt The CABLE standard output for each task is written to /absolute/path/to/tasks//out.txt The NetCDF output for each task is written to run/site/outputs/_out.nc

with verbose `-v` option:
- same output as non-verbose case

### When running FLUXNET tasks (no job submission)
by default
Note: keep <task_name> as is in printed output to indicate it's a generic pattern that changes for each task.

$ benchcab fluxnet-run-tasks --no-submit Running FLUXNET tasks... Successfully ran FLUXNET tasks

with verbose `-v` option:
- print all output from the non-verbose case
- print stdout running the CABLE executable for each task? (Need to be careful about parallel IO.)

$ benchcab fluxnet-run-tasks --no-submit -v Running FLUXNET tasks... Running task ... CABLE standard output saved in /absolute/path/to/tasks//out.txt cd /absolute/path/to/ ./cable > /absolute/path/to/tasks//out.txt Adding attributes to output file: /absolute/path/to/outputs/_out.nc cd /absolute/path/to/cwd Running task ... CABLE standard output saved in /absolute/path/to/tasks//out.txt cd /absolute/path/to/ ./cable > /absolute/path/to/tasks//out.txt Adding attributes to output file: /absolute/path/to/outputs/_out.nc cd /absolute/path/to/cwd ... (this ellipsis won't be here, this is just to reduce the amount of text in the draft) Running task ... CABLE standard output saved in /absolute/path/to/tasks//out.txt cd /absolute/path/to/ ./cable > /absolute/path/to/tasks//out.txt Adding attributes to output file: /absolute/path/to/outputs/_out.nc cd /absolute/path/to/cwd Successfully ran FLUXNET tasks

SeanBryan51 commented 1 year ago

@ccarouge I've added a rough draft for what I think we should output to stdout. Please feel free to add or change the output behaviour in the rough draft.

ccarouge commented 1 year ago

During compilation, I would specify "serially / with MPI". For fluxnet-setup-work-dir, I would start with a generic sentence, something like: "setting up run directory tree for flux sites tests". And maybe just list the logs, outputs and tasks paths in the default output. The "runs" and "site" paths aren't useful information.

For CABLE stdout, I would send it to a file in the tasks directory in all cases. So we save it but don't output to screen. In verbose version, we could output something like: "Running task X. CABLE standard output saved in /path/to/file".

Edit: Awesome to discuss the output here in the issue.

SeanBryan51 commented 1 year ago

Thanks for the feedback. I've made those changes to the draft

ccarouge commented 1 year ago

Added a few more details for the case $ benchcab fluxnet-run-tasks, non-verbose. So people know where all outputs are.

SeanBryan51 commented 1 year ago

I've just made an edit to keep <task_name> when describing the location of outputs for $ benchcab fluxnet-run-tasks

ccarouge commented 1 year ago

Edit: more details about what the PBS script is for under $ benchcab fluxnet-run-tasks

CABLE-LSM / benchcab