Modifying parameter files

[wwieder]

I'd suspect there's a bit of a cottage industry with tools to modify CTSM parameter files. Is this something that we should more formally support or provide tools for? This could be basic scripts in tools/contrib where we start by collecting example scripts that provide basic functionality? Maybe this already exists and I'm just unaware? More likely I'd suspect @olyson, @ekluzek, @billsacks, @slevisconsulting, @swensosc, @rosiealice, @djk2120 or others have lots of examples around? Should any of these be curated (even informally)?

The second part of this question is what's motivating the broader discussion. Specifically, I'd like to create a duplicate arctic C3 grasses for pft 12 & 13 grasses that are all based on the same parameterization, but that differ in some luna/phs/or stomatal conductance parameters to represent vegetation that is more resource acquisitive vs. conservative. What tools do we already have to to start thinking about this on the technical side?

[djk2120]

Katie and I have some tools (python) that are useful for modifying paramfiles, specifically for creating parameter perturbation ensembles.

Using xarray, etc., it's very easy to modify paramfiles, and I can provide some examples for changing just one parameter, if that's helpful.

On Thu, Oct 7, 2021 at 9:19 AM will wieder @.***> wrote:

I'd suspect there's a bit of a cottage industry with tools to modify CTSM parameter files. Is this something that we should more formally support or provide tools for? This could be basic scripts in tools/contrib where we start by collecting example scripts that provide basic functionality? Maybe this already exists and I'm just unaware? More likely I'd suspect @olyson https://github.com/olyson, @ekluzek https://github.com/ekluzek, @billsacks https://github.com/billsacks, @slevisconsulting https://github.com/slevisconsulting, @swensosc https://github.com/swensosc, @rosiealice https://github.com/rosiealice, @djk2120 https://github.com/djk2120 or others have lots of examples around? Should any of these be curated (even informally)?

The second part of this question is what's motivating the broader discussion. Specifically, I'd like to create a duplicate arctic C3 grasses for pft 12 & 13 grasses that are all based on the same parameterization, but that differ in some luna/phs/or stomatal conductance parameters to represent vegetation that is more resource acquisitive vs. conservative. What tools do we already have to to start thinking about this on the technical side?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/issues/1510, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADNHR5SUQKGA56UE4YKHDKTUFW3ARANCNFSM5FRRPCMQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[wwieder]

pointing to an example for how to do this would be super helpful @djk2120

[rgknox]

@wwieder we have some tools for modifying FATES parameter file kept here: https://github.com/NGEET/fates/tree/master/tools

[djk2120]

two examples here: https://github.com/djk2120/ppe_tools/tree/main/params

modify_params.ipynb is in line with what you've mentioned

ensemble.ipynb is more involved, and is specifically catered to generating ensembles

On Thu, Oct 7, 2021 at 9:40 AM will wieder @.***> wrote:

pointing to an example for how to do this would be super helpful @djk2120 https://github.com/djk2120

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/issues/1510#issuecomment-937913959, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADNHR5UXZ7NF5IYFK64FIHLUFW5NJANCNFSM5FRRPCMQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[wwieder]

Thanks for these examples @rgknox and @djk2120 .

@olyson also pointed me to these notes on the wiki

I think for my purposes, the tools from this script in FATES may be particularly helpful, as for now I just want to make all pfts different flavors of arctic C3 grasses. @rgknox, Am I correct in understanding that this script overwrites all pft-specific values with ones from a PFT that a user defines? If so, I wonder how hard it would be to adapt this to a CLM parameter file for big leaf runs?

[wwieder]

@djk2120 I noticed you had a comment in your script that xarray didn't write out .nc files correctly. This is a bummer, now that I have a script that overwrites parameters as I want them for simulations at Niwot. My case, however, seems to be working (modified from my orig post as I sheepishly realized I wasn't pointing to the correct parameter file...)

I wonder if others can make suggestions for a workaround that would let me use this .nc file, or the script that produced it, which can be found here. https://github.com/wwieder/NWT_CLM/blob/master/notebooks/modifyParameters.ipynb

Thanks in advance. Will

[djk2120]

Not sure of a solution, per se, or if the bug I experienced is still an issue. I can check. My workaround was to use the NetCDF4 package to write the files.

But I would double-check your problem, because that error looks like it can't find the file, which is different from what I was describing.

On Tue, Nov 30, 2021 at 3:42 PM will wieder @.***> wrote:

@djk2120 https://github.com/djk2120 I noticed you had a comment in your script that xarray didn't write out .nc files correctly. This is a bummer, now that I have a script that overwrites parameters as I want them for simulations at Niwot. But that can't be read by the model

Attempting to read PFT physiological data ..... (GETFIL): attempting to find local file ctsm51_params.c210803.NIWO_critDay13h_c3ArcticMods.nc (GETFIL): failed getting file from full path: /glade/work/wwieder/modified_inputs/NEON/ctsm51_params.c210803.NIWO_critDay13h_c3ArcticMods.nc

I wonder if others can make suggestions for a workaround that would let me use this .nc file, or the script that produced it, which can be found here.

https://github.com/wwieder/NWT_CLM/blob/master/notebooks/modifyParameters.ipynb

Thanks in advance. Will

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/issues/1510#issuecomment-983127987, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADNHR5WQ6TWMKNR7XYELKADUOVONVANCNFSM5FRRPCMQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[djk2120]

I.e. I can't stat that file, which means it was never written to that path. Whereas my files would write, but were somehow corrupted according to CLM.

On Tue, Nov 30, 2021 at 3:56 PM Daniel Kennedy @.***> wrote:

Not sure of a solution, per se, or if the bug I experienced is still an issue. I can check. My workaround was to use the NetCDF4 package to write the files.

But I would double-check your problem, because that error looks like it can't find the file, which is different from what I was describing.

On Tue, Nov 30, 2021 at 3:42 PM will wieder @.***> wrote:

@djk2120 https://github.com/djk2120 I noticed you had a comment in your script that xarray didn't write out .nc files correctly. This is a bummer, now that I have a script that overwrites parameters as I want them for simulations at Niwot. But that can't be read by the model

Attempting to read PFT physiological data ..... (GETFIL): attempting to find local file ctsm51_params.c210803.NIWO_critDay13h_c3ArcticMods.nc (GETFIL): failed getting file from full path: /glade/work/wwieder/modified_inputs/NEON/ctsm51_params.c210803.NIWO_critDay13h_c3ArcticMods.nc

I wonder if others can make suggestions for a workaround that would let me use this .nc file, or the script that produced it, which can be found here.

https://github.com/wwieder/NWT_CLM/blob/master/notebooks/modifyParameters.ipynb

Thanks in advance. Will

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/issues/1510#issuecomment-983127987, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADNHR5WQ6TWMKNR7XYELKADUOVONVANCNFSM5FRRPCMQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[ekluzek]

@wwieder at least one of your files in that directory was in NetCDF-4 format, which CESM can't use. So you need to convert them to a different format. One way to do that is to use nccopy to change the format to HDF-5 for example

So for example...

nccopy -k 5 ctsm51_params.c210803.NIWO_critDay13h_fastOnset_c3ArcticMods.nc **ctsm51_params.c210803.NIWO_critDay13h_fastOnset_c3ArcticMods_cdf5.nc

But, most likely in your script that you create the file you just need to change the format of the output file. You should be able to do that on the open statement.

[djk2120]

It looks like you're missing "fastOnset" in the path name in your user_nl_clm

On Tue, Nov 30, 2021 at 4:00 PM Daniel Kennedy @.***> wrote:

I.e. I can't stat that file, which means it was never written to that path. Whereas my files would write, but were somehow corrupted according to CLM.

On Tue, Nov 30, 2021 at 3:56 PM Daniel Kennedy @.***> wrote:

Not sure of a solution, per se, or if the bug I experienced is still an issue. I can check. My workaround was to use the NetCDF4 package to write the files.

But I would double-check your problem, because that error looks like it can't find the file, which is different from what I was describing.

On Tue, Nov 30, 2021 at 3:42 PM will wieder @.***> wrote:

@djk2120 https://github.com/djk2120 I noticed you had a comment in your script that xarray didn't write out .nc files correctly. This is a bummer, now that I have a script that overwrites parameters as I want them for simulations at Niwot. But that can't be read by the model

Attempting to read PFT physiological data ..... (GETFIL): attempting to find local file ctsm51_params.c210803.NIWO_critDay13h_c3ArcticMods.nc (GETFIL): failed getting file from full path: /glade/work/wwieder/modified_inputs/NEON/ctsm51_params.c210803.NIWO_critDay13h_c3ArcticMods.nc

I wonder if others can make suggestions for a workaround that would let me use this .nc file, or the script that produced it, which can be found here.

https://github.com/wwieder/NWT_CLM/blob/master/notebooks/modifyParameters.ipynb

Thanks in advance. Will

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ESCOMP/CTSM/issues/1510#issuecomment-983127987, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADNHR5WQ6TWMKNR7XYELKADUOVONVANCNFSM5FRRPCMQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[wwieder]

Thanks for your suggestions @djk2120 and @ekluzek. Sorry for some of the confusion. Almost as soon as I posted my question I realized I wasn't pointing to the right file in my user_nl_clm and edited my posting (but you don't get updates on that...).

As far as I can tell the NetCDF-4 format doesn't seem to be an issue, as the model is currently running with the correct file. Is that surprising, Erik? All of this is qualified by saying I haven't looked at any results to see if they are sensible... more on that tomorrow.

[rgknox]

I think for my purposes, the tools from this script in FATES may be particularly helpful, as for now I just want to make all pfts different flavors of arctic C3 grasses. @rgknox, Am I correct in understanding that this script overwrites all pft-specific values with ones from a PFT that a user defines? If so, I wonder how hard it would be to adapt this to a CLM parameter file for big leaf runs?

@wwieder, the script you mention only overwrites the pft of interest, it will not overwrite all pfts with the same value.

[billsacks]

As far as I can tell the NetCDF-4 format doesn't seem to be an issue

NetCDF4-formatted files work on cheyenne, we just avoid adding them to inputdata because they don't work on some CESM-supported systems.