Handle some special cases for wrf-python

[zmoon]

The "uvmet" variables and a few other tweaks

@quaz115 let me know if it doesn't work for you

[quaz115]

The "uvmet" variables

@quaz115 let me know if it doesn't work for you

@zmoon I am getting this issue as i tried it:

self.obj = mio.models._wrfchem_mm.open_mfdataset(self.files,**self.mod_kwargs)

File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/monetio/monetio/models/_wrfchem_mm.py", line 139, in open_mfdataset dset = xr.merge(var_wrf_list) File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 976, in merge merge_result = merge_core( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 701, in merge_core variables, out_indexes = merge_collected( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 290, in merge_collected merged_vars[name] = unique_variable( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 144, in unique_variable raise MergeError( xarray.core.merge.MergeError: conflicting values for variable 'wspd_wdir' on objects to be combined. You can skip this check by specifying compat='override'.

[quaz115]

The "uvmet" variables @quaz115 let me know if it doesn't work for you

@zmoon I am getting this issue as i tried it:

self.obj = mio.models._wrfchem_mm.open_mfdataset(self.files,**self.mod_kwargs)

File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/monetio/monetio/models/_wrfchem_mm.py", line 139, in open_mfdataset dset = xr.merge(var_wrf_list) File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 976, in merge merge_result = merge_core( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 701, in merge_core variables, out_indexes = merge_collected( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 290, in merge_collected merged_vars[name] = unique_variable( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 144, in unique_variable raise MergeError( xarray.core.merge.MergeError: conflicting values for variable 'wspd_wdir' on objects to be combined. You can skip this check by specifying compat='override'.

For YAML obs key added these:

obs:
  pasadena_ground: # obs label
    filename: '/scratch2/BMC/rcm1/qrasool/wrfout_postprocessing/LA_Ground_merge_NEW21Sept2023Update.csv' #'/wrk/users/charkins/MM_data/pasadena_ground_ict/20210725/recap-CIT-AMS1-NR-PM1_CITLL_20210725_RB.ict'
    obs_type: ground
    time_var: 'MasterTimewave_Start_UTC'
    ground_coordinate: {'latitude':34.136363,'longitude':-118.126817}
    #resample: '600S' #10 min so works on Hera as a test. Can comment this if submitting a job. 
    variables: #Opt 
      'uvmet10_wspd':
        unit_scale: 1 #Opt Scaling factor 
        unit_scale_method: '*' #'/' #'*' #Opt Multiply = '*' , Add = '+', subtract = '-', divide = '/'
        ylabel_plot: 'Wind speed (m s^-1)'
      'uvmet10_wdir':
        unit_scale: 1 #Opt Scaling factor 
        unit_scale_method: '*' #'/' #'*' #Opt Multiply = '*' , Add = '+', subtract = '-', divide = '/'
        ylabel_plot: 'Wind dir (degrees)'

[quaz115]

The "uvmet" variables @quaz115 let me know if it doesn't work for you

@zmoon I am getting this issue as i tried it:

self.obj = mio.models._wrfchem_mm.open_mfdataset(self.files,**self.mod_kwargs)

File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/monetio/monetio/models/_wrfchem_mm.py", line 139, in open_mfdataset dset = xr.merge(var_wrf_list) File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 976, in merge merge_result = merge_core( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 701, in merge_core variables, out_indexes = merge_collected( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 290, in merge_collected merged_vars[name] = unique_variable( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 144, in unique_variable raise MergeError( xarray.core.merge.MergeError: conflicting values for variable 'wspd_wdir' on objects to be combined. You can skip this check by specifying compat='override'.

For YAML obs key added these: obs: pasadena_ground: # obs label filename: '/scratch2/BMC/rcm1/qrasool/wrfout_postprocessing/LA_Ground_merge_NEW21Sept2023Update.csv' #'/wrk/users/charkins/MM_data/pasadena_ground_ict/20210725/recap-CIT-AMS1-NR-PM1_CITLL_20210725_RB.ict' obs_type: ground time_var: 'MasterTimewave_Start_UTC' ground_coordinate: {'latitude':34.136363,'longitude':-118.126817} #resample: '600S' #10 min so works on Hera as a test. Can comment this if submitting a job. variables: #Opt 'uvmet10_wspd': unit_scale: 1 #Opt Scaling factor unit_scalemethod: '' #'/' #'' #Opt Multiply = '' , Add = '+', subtract = '-', divide = '/' ylabel_plot: 'Wind speed (m s^-1)' 'uvmet10_wdir': unit_scale: 1 #Opt Scaling factor unit_scalemethod: '' #'/' #'' #Opt Multiply = '' , Add = '+', subtract = '-', divide = '/' ylabel_plot: 'Wind dir (degrees)'

@zmoon YAML path on Hera: /scratch2/BMC/rcm1/qrasool/wrfout_postprocessing/control_wrfchem_groundsunvex_QZR_mmdevelopaircraft_Copy_Rocky8_SSMATOMo3_LowvsHighNox_RefinedPlotting.yaml

[quaz115]

@zmoon still the same issue: m.open_model_files(time_interval=time_interval, control_dict=self.control_dict) File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/MELODIES-MONET/melodies_monet/driver.py", line 566, in open_model_files self.obj = mio.models._wrfchem_mm.open_mfdataset(self.files,**self.mod_kwargs) File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/monetio/monetio/models/_wrfchem_mm.py", line 139, in open_mfdataset dset = xr.merge(var_wrf_list) File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 976, in merge merge_result = merge_core( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 701, in merge_core variables, out_indexes = merge_collected( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 290, in merge_collected merged_vars[name] = unique_variable( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 144, in unique_variable raise MergeError( xarray.core.merge.MergeError: conflicting values for variable 'wspd_wdir' on objects to be combined. You can skip this check by specifying compat='override'.

[quaz115]

@zmoon I don't have permission to push my commit, but i made following updates to the wrf-chem reader py file and it is working now (for rh (%), wspd: uvmet_wspd or uvmet10_wspd (m/s), wdir: uvmet_wdir or uvmet10_wdir (degrees)), can you review this (@rschwant for reference):

    wrflist = []
    for files in fname:
        wrflist.append(Dataset(files))

    if not surf_only_nc:
        # Add some additional defaults needed for aircraft analysis
        # Turn this on also if need to convert aerosols
        var_list.append("pres")
        var_list.append("pressure") 
        var_list.append("height")
        var_list.append("tk")
        var_list.append("temp")  
        var_list.append("rh")  
        var_list.append("height_agl")
        var_list.append("PSFC")
        # need to calculate surface pressure and dp and optionally dz here.

        # Additional defaults for satellite analysis
        var_list.append("zstag")

    # Remove 'uvmet10_wspd' and 'uvmet10_wdir' if present
    # Adjust var_list for wind variables
    if any(v in {"uvmet10_wspd", "uvmet10_wdir"} for v in var_list):
        # Remove 'uvmet10_wspd' and 'uvmet10_wdir' from var_list
        var_list = [v for v in var_list if v not in {"uvmet10_wspd", "uvmet10_wdir"}]
        # Add 'uvmet10_wspd_wdir' to var_list
        var_list.append("uvmet10_wspd_wdir")

    var_wrf_list = []
    for var in var_list:
        if var == "pres":
            # Handle 'pres' variable
            var_wrf = getvar(
                wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False, units="Pa"
            )
            var_wrf = var_wrf.rename(var)
            var_wrf_list.append(var_wrf)
            continue
        elif var in {"height", "height_agl", "zstag", "temp", "rh"}:
            if var in {"height", "height_agl", "zstag"}:
                # For height variables, specify units
                var_wrf = getvar(
                    wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False, units="m"
                )
            else:
                # For 'temp' and 'rh', use default units
                var_wrf = getvar(
                    wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False
                )
            # Remove 'wspd_wdir' from coordinates if present
            if 'wspd_wdir' in var_wrf.coords:
                print(f"Removing 'wspd_wdir' from coordinates of variable '{var}'")
                var_wrf = var_wrf.reset_coords('wspd_wdir', drop=True)
            var_wrf = var_wrf.rename(var)
            var_wrf_list.append(var_wrf)
            continue
        elif var in {"uvmet10", "uvmet"}:
            # Existing code for wind variables
            pref = var.split("_")[0]
            var_wrf = getvar(
                wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False
            )
            var_wrf_list.extend(
                [
                    var_wrf.isel(u_v=0).rename(f"{pref}_u"),
                    var_wrf.isel(u_v=1).rename(f"{pref}_v"),
                ]
            )
            continue
        elif var in {"uvmet10_wspd_wdir", "uvmet_wspd_wdir"}:
            # Existing code for wind speed and direction
            pref = var.split("_")[0]
            var_wrf = getvar(
                wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False
            )
            var_wrf_list.extend(
                [
                    var_wrf.isel(wspd_wdir=0).rename(f"{pref}_wspd"),
                    var_wrf.isel(wspd_wdir=1).rename(f"{pref}_wdir").assign_attrs(units="deg"),
                ]
            )
            continue
        else:
            # Existing code for other variables
            var_wrf = getvar(
                wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False
            )
            # Remove 'wspd_wdir' from coordinates if present
            if 'wspd_wdir' in var_wrf.coords:
                print(f"Removing 'wspd_wdir' from coordinates of variable '{var}'")
                var_wrf = var_wrf.reset_coords('wspd_wdir', drop=True)
            var_wrf = var_wrf.rename(var)
            var_wrf_list.append(var_wrf)
        print(f"Fetched variable '{var}' with internal name '{var_wrf.name}'")

    # Remove 'wspd_wdir' from coordinates of all variables before merging
    for idx, var_data in enumerate(var_wrf_list):
        if 'wspd_wdir' in var_data.coords:
            print(f"Removing 'wspd_wdir' from coordinates of variable '{var_data.name}'")
            var_data = var_data.reset_coords('wspd_wdir', drop=True)
            var_wrf_list[idx] = var_data

    # Proceed with the merge
    dset = xr.merge(var_wrf_list)  

[quaz115]

@zmoon can you also add 'rh' as well (i added it for my case). I think, we should have these MET variables: T, P, RH, WSPD, WDIR for sure from getvar utility

[zmoon]

@quaz115 I'm pretty sure 'tk' (already in the aircraft list) is the same is 'temp' with default units. Would you say aircraft always needs RH? If not, would it work to just add it to the var list in your YAML section?

[quaz115]

@quaz115 I'm pretty sure 'tk' (already in the aircraft list) is the same is 'temp' with default units. Would you say aircraft always needs RH? If not, would it work to just add it to the var list in your YAML section?

@zmoon Sorry for the confusion, i meant in general (about these variables that should be there for MET obs vs model comparisons, for future as well, and PBLH additionally). Also Thanks for the simplified code fixes. Just re-ran with this branch and it works well. Please go ahead with the Merge

[rschwant]

Temperature update fixes the problem I was seeing!

[zmoon]

@quaz115 @rschwant any final thoughts?

[rschwant]

Looks good to me!

[quaz115]

@quaz115 @rschwant any final thoughts?

Looks good to be merged for me as well