Additional Machine Learning mods

[cacraigucar]

The machine learning code no longer ran with the current PUMAS code. Made modifications to enable it to run again

Removed an unnecessary intent(inout) and changed it to intent(in). This intent reflects what was happening within the code and allows the variable to remain as module level data. This fixes the concern that I had with the module level data.

Additional cleanup was made

[sjsprecious]

Hi @cacraigucar , to switch to the ML code, do I just need to change the micro_mg_warm_rain namelist variable? Is there any other namelist variable that I need to set, especially when micro_mg_warm_rain = tau or micro_mg_warm_rain = emulated?

[cacraigucar]

Hi @cacraigucar , to switch to the ML code, do I just need to change the micro_mg_warm_rain namelist variable? Is there any other namelist variable that I need to set, especially when micro_mg_warm_rain = tau or micro_mg_warm_rain = emulated?

@sjsprecious - That is all you need to switch between the two modes. All of the other settings should be automatically set from using the appropriate name.

[sjsprecious]

Thanks @cacraigucar . One more question: do we still need to keep the code block for warm_rain = 'never' (https://github.com/ESCOMP/PUMAS/blob/pumas_cam-release_v1.30/micro_pumas_v1.F90#L4167 and https://github.com/ESCOMP/PUMAS/blob/pumas_cam-release_v1.30/micro_pumas_v1.F90#L4459)? I think the never option is already removed from this branch (https://github.com/PUMASDevelopment/CAM/blob/katetc/machlrn_camdev_pr2_cac/bld/namelist_files/namelist_definition.xml#L2637).

[cacraigucar]

@sjsprecious - good catch. I removed "never" from CAM, but I don't think I ever did it from PUMAS. I have pushed the changes back to this branch, but do not have time to test it. Please let me know if it doesn't work. I'm hopeful I can remove two if blocks without breaking anything, but without testing, it is not guaranteed.

[sjsprecious]

Hi @cacraigucar , I am testing your pumas_machlearn branch with this CAM branch (https://github.com/PUMASDevelopment/CAM/tree/katetc/machlrn_camdev_pr2_cac). When I set warm_rain = emulated, I got a runtime error as below:

ERROR: handle_errmsg: micro_pumas_cam_init: NetCDF: Malformed URL

Does this error look familiar to you? Do you know how to resolve it?

[cacraigucar]

@sjsprecious - That error message does not look familiar to me. You probably need to reach out to @Katetc. The last time I tested this setup was with my branches and I can say both emulated and tau worked at that point. @Katetc opened these new branches and pulled in my code and then made modifications. Perhaps she knows what might have been modified that could cause this error

[cacraigucar]

To clarify, this PUMAS branch is the same, it is the ESCOMP/CAM branch which is different and is where the micro_pumas_cam_init code is stored.

[sjsprecious]

Thanks @cacraigucar for your clarification. I just found that my CAM namelist file did not have the section of stochastic_emulated_nl, which was required by the emulated scheme (https://github.com/PUMASDevelopment/CAM/blob/katetc/machlrn_camdev_pr2_cac/src/physics/cam_dev/stochastic_emulated_cam.F90#L50-L51). I guess this is the reason why I got the error above. Do you or @Katetc know how to add this missing namelist section?

[cacraigucar]

Thanks @cacraigucar for your clarification. I just found that my CAM namelist file did not have the section of stochastic_emulated_nl, which was required by the emulated scheme (https://github.com/PUMASDevelopment/CAM/blob/katetc/machlrn_camdev_pr2_cac/src/physics/cam_dev/stochastic_emulated_cam.F90#L50-L51). I guess this is the reason why I got the error above. Do you or @Katetc know how to add this missing namelist section?

My user_nl_cam for emulated is:

micro_mg_warm_rain = 'emulated' stochastic_emulated_filename_quantile = '/glade/work/dgagne/cam_mp_run6_quantile_nn/quantile_neural_net_fortran.nc' stochastic_emulated_filename_input_scale = '/glade/work/dgagne/cam_mp_run6_quantile_nn/input_quantile_scaler.nc' stochastic_emulated_filename_output_scale = '/glade/work/dgagne/cam_mp_run6_quantile_nn/output_quantile_scaler.nc'

Sorry - I forgot about the input files that are needed

[sjsprecious]

Thanks @cacraigucar for your clarification. I just found that my CAM namelist file did not have the section of stochastic_emulated_nl, which was required by the emulated scheme (https://github.com/PUMASDevelopment/CAM/blob/katetc/machlrn_camdev_pr2_cac/src/physics/cam_dev/stochastic_emulated_cam.F90#L50-L51). I guess this is the reason why I got the error above. Do you or @Katetc know how to add this missing namelist section?

My user_nl_cam for emulated is:

micro_mg_warm_rain = 'emulated' stochastic_emulated_filename_quantile = '/glade/work/dgagne/cam_mp_run6_quantile_nn/quantile_neural_net_fortran.nc' stochastic_emulated_filename_input_scale = '/glade/work/dgagne/cam_mp_run6_quantile_nn/input_quantile_scaler.nc' stochastic_emulated_filename_output_scale = '/glade/work/dgagne/cam_mp_run6_quantile_nn/output_quantile_scaler.nc'

Sorry - I forgot about the input files that are needed

Thanks @cacraigucar for providing those information. Can you add these changes to Kate's CAM branch (https://github.com/PUMASDevelopment/CAM/tree/katetc/machlrn_camdev_pr2_cac) and set them by default? Or we want the user to set them up manually?

[cacraigucar]

Thanks @cacraigucar for your clarification. I just found that my CAM namelist file did not have the section of stochastic_emulated_nl, which was required by the emulated scheme (https://github.com/PUMASDevelopment/CAM/blob/katetc/machlrn_camdev_pr2_cac/src/physics/cam_dev/stochastic_emulated_cam.F90#L50-L51). I guess this is the reason why I got the error above. Do you or @Katetc know how to add this missing namelist section?

My user_nl_cam for emulated is: micro_mg_warm_rain = 'emulated' stochastic_emulated_filename_quantile = '/glade/work/dgagne/cam_mp_run6_quantile_nn/quantile_neural_net_fortran.nc' stochastic_emulated_filename_input_scale = '/glade/work/dgagne/cam_mp_run6_quantile_nn/input_quantile_scaler.nc' stochastic_emulated_filename_output_scale = '/glade/work/dgagne/cam_mp_run6_quantile_nn/output_quantile_scaler.nc' Sorry - I forgot about the input files that are needed

Thanks @cacraigucar for providing those information. Can you add these changes to Kate's CAM branch (https://github.com/PUMASDevelopment/CAM/tree/katetc/machlrn_camdev_pr2_cac) and set them by default? Or we want the user to set them up manually?

I do not know if these are the final versions of these files or not. Since they are pointing to a user directory on cheyenne/derecho this pathname will not be committed. Once the datafiles are finalized, we need to make sure they are ready for the svn inputdata repository (mandatory metadata is added if it is not there). This is part of the final review process, but Kate's PR is still in draft and not at that point yet. One thing we are waiting for is this PR to be merged and tagged.

[sjsprecious]

Thanks @cacraigucar for your clarification. I have manually tested the emulated namelist variables you have posted and my simulation finishes successfully. Thank you for your help and I will continue my GPU fix.

[Katetc]

Merging this PR today, and wanted to note that the new aux_pumas tests will test all four warm rain options with the following expectations (as per Andrew's notes): sb2001 works, kk works, tau works, emulated does not work yet.