Chemical model can not work properly after merging into the updates on July 1

[zhanglikate]

Description

The chemical model of FV3-Chem and CCPP-Chem is able to run properly in the version of " https://github.com/NOAA-GSL/ufs-weather-model/tree/gsl/develop-chem" at HERA"/scratch2/BMC/gsd-fv3-dev/lzhang/ufs-weather-model "

However, when we merged the FV3GFS updates with the version including the commits in "https://github.com/NOAA-EMC/fv3atm/commit/ad7bddc3fbe0b25914fb28439337603ca5d0f3bf" (the differences can be seen from " https://github.com/NOAA-EMC/fv3atm/compare/e4eb2bafabdc0528675f75563bcebcd909de6e2e...ad7bddc3fbe0b25914fb28439337603ca5d0f3bf "), the chemical results are not correct, it looks like there is no vertical mixing or transport/advection, even though we did not change anything in the chemical model part.

This is the merged version : " https://github.com/climbfuji/ufs-weather-model/tree/update_gsl_develop_chem_from_develop", at HERA: scratch2/BMC/gsd-fv3-dev/lzhang/ccpp-chem/ufs-weather-model "

[arunchawla-NOAA]

Kate is this an issue with the gsl fork ? If yes then I am not sure the code managers here can solve that problem

[zhanglikate]

@arunchawla-NOAA , Arun, this is not only an issue with the GSL. Both the GEFS-Aerosols (in operation now) and FV3GFS-CCPP-Chem (the same as GEFS-Aerosols, but use CCPP as the coupler instead of NUOPC) would be impacted. Because they are using the same chemical model and sharing the same parts in the FV3GFS. So they can not update to the current FV3GFS version since the commits in https://github.com/NOAA-EMC/fv3atm/commit/ad7bddc3fbe0b25914fb28439337603ca5d0f3bf , the chemical fields are not correct without advection and vertical transport and mixing since merging with this version. That is why we need the extension for HERA old library and modules for a while, because for these two models, we can not update the FV3GFS part. Hopefully, we can get it fixed soon.

[arunchawla-NOAA]

But GEFS-Aerosol for the next stage will go from the top of ufs-weather-model. I do not understand what the FV3GFS-CCPP-Chem model is ? Is this built on the GSL fork ? This PR went in in July 2021. That was a while ago. Why was this not tested till now? It is not practical to have parallel long term development on institutional forks and expect the auth repo to test for those capabilities

[zhanglikate]

Yes. GEFS-Aerosols would be replaced by UFS-Aerosols in the future as planned, but the chemical model of these two model are not the same, so they may not sharing the same part in FV3GFS. And UFS-Aerosols is still under developing, not ready to run real-time now. FV3GFS-CCPP-Chem is the CCPP version of GEFS-Aerosols, however it includes the aerosol feedback, but GEFS-Aerosols does not. It provide a frontier tool to quantify the aerosol impact on whether for future development in UFS-Aerosols. Currently, FV3GFS-CCPP-Chem has been running real-time in GSL for more than 1 years by including the aerosol feedback for research purpose. Also, it has been used by the aerosols data assimilation group for aerosol assimilation development (also running the real-time in GSL for a while ) and the S2S studies at GSL. We have reported this issue to Raffaele by email immediately last July when Dom finished the merging, it has not beed solved so far. We were also trying to figure out the issue ourself but we are not the experts in FV3GFS part and don't know what happen. Recently, the updates of HERA library and modules are a big challenges for these works using FV3GFS-CCPP-Chem if it can not be updated with recent FV3GFS. We have to get FV3GFS updated as soon as possible to keep the real-time running for these projects considering the HERA updates, especially, the UFS-Aerosols has not yet been ready to replace it in short-term. We can not get the help from Raffaele since now, so would like to see if we can get any help from you to fix this issue. We always kept on updating the FV3GFS part and its workflow as the development branch before this issue. Thanks very much.

[arunchawla-NOAA]

If fv3gfs-ccpp-chem is a critical development component then why is it not in the authoritative ufs repositories ? otherwise this problem will keep on occurring

[zhanglikate]

The FV3GFS was working well with GEFS-Aerosols before the commits, so by default I think it is in the authoritative ufs repositories. After the commits, some parts related to UFS-Aerosols may replace the previous GEFS-Aerosols part, since these two chemical components are not the same. But will need help to see what exactly happen.

Yes. I agree with you. I am not the CCPP code manager, but I know Dom was working on it until this issue occurring, he also created a process library, which can be shared by chemical components of UFS-Aerosols and GEFS-Aerosols. I think the plan was getting authoritative UFS repositories after some developments when the whole package is ready, including the regression test part.

[DeniseWorthen]

@zhanglikate Is there a bug that needs to be resolved in UFS regarding this issue? If not I think we should close it. Thanks.

[arunchawla-NOAA]

This is not an issue with the authoritative UFS-weather-model. I suggest we close this issue

[DeniseWorthen]

Thanks.