Error when running exp1.run() in mesf_flow.ipynb

[Adrien63ro]

Hello everybody, Here is an error that occurred while running _meshflow.ipynb (see below). It seems the problem comes from different dimensions. Please find attached the files used in my case (domain shapefiles, modified landcover file, 3 remapped .nc files).

Thank you for your help

Adrien files_east.zip

Here is the error message :

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 exp1.run()

File ~/FHIMP/virtual_env/fhimp_venv/lib/python3.10/site-packages/meshflow/core.py:320, in MESHWorkflow.run(self)
    317 _ordered_dims = {self.main_id: self.main_seg}
    319 # 1 generate drainage database
--> 320 self.ddb = utility.prepare_mesh_ddb(riv=self.reordered_riv,
    321                                     cat=self.cat,
    322                                     landcover=self.landcover,
    323                                     cat_dim=self.main_id,
    324                                     gru_dim=self.gru_dim,
    325                                     gru_var=self.gru_var,
    326                                     hru_dim=self.hru_dim,
    327                                     hru_var=self.hru_var,
    328                                     gru_names=self.landcover_classes,
    329                                     include_vars=self.ddb_vars,
    330                                     attr_local=self.ddb_local_attrs,
    331                                     attr_global=self.ddb_global_attrs,
    332                                     min_values=self.ddb_min_values,
    333                                     fill_na=None,
    334                                     ordered_dims=_ordered_dims,
    335                                     ddb_units=self.ddb_units,
    336                                     ddb_to_units=self.ddb_to_units)
    338 # forcing unit registry
    339 _ureg = pint.UnitRegistry(force_ndarray_like=True)

File ~/FHIMP/virtual_env/fhimp_venv/lib/python3.10/site-packages/meshflow/utility/network.py:452, in prepare_mesh_ddb(riv, cat, landcover, cat_dim, gru_dim, gru_var, hru_dim, hru_var, gru_names, include_vars, attr_local, attr_global, min_values, fill_na, ordered_dims, ddb_units, ddb_to_units)
    436 # creating dictionary for final merging
    437 lc_names = {
    438     "coords": {
    439         gru_dim: {
   (...)
    450     },
    451 }
--> 452 lc_names_da = xr.Dataset.from_dict(lc_names)
    454 # making a list of all xarray.Dataset objects
    455 ds_list = [landcover_ds,
    456            lc_names_da,
    457            riv_ds,
    458            cat_ds]

File ~/FHIMP/virtual_env/fhimp_venv/lib/python3.10/site-packages/xarray/core/dataset.py:7438, in Dataset.from_dict(cls, d)
   7433 except KeyError as e:
   7434     raise ValueError(
   7435         "cannot convert dict without the key "
   7436         "'{dims_data}'".format(dims_data=str(e.args[0]))
   7437     )
-> 7438 obj = cls(variable_dict)
   7440 # what if coords aren't dims?
   7441 coords = set(d.get("coords", {})) - set(d.get("dims", {}))

File ~/FHIMP/virtual_env/fhimp_venv/lib/python3.10/site-packages/xarray/core/dataset.py:685, in Dataset.__init__(self, data_vars, coords, attrs)
    682 if isinstance(coords, Dataset):
    683     coords = coords._variables
--> 685 variables, coord_names, dims, indexes, _ = merge_data_and_coords(
    686     data_vars, coords
    687 )
    689 self._attrs = dict(attrs) if attrs is not None else None
    690 self._close = None

File ~/FHIMP/virtual_env/fhimp_venv/lib/python3.10/site-packages/xarray/core/dataset.py:416, in merge_data_and_coords(data_vars, coords)
    412     coords = create_coords_with_default_indexes(coords, data_vars)
    414 # exclude coords from alignment (all variables in a Coordinates object should
    415 # already be aligned together) and use coordinates' indexes to align data_vars
--> 416 return merge_core(
    417     [data_vars, coords],
    418     compat="broadcast_equals",
    419     join="outer",
    420     explicit_coords=tuple(coords),
    421     indexes=coords.xindexes,
    422     priority_arg=1,
    423     skip_align_args=[1],
    424 )

File ~/FHIMP/virtual_env/fhimp_venv/lib/python3.10/site-packages/xarray/core/merge.py:723, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value, skip_align_args)
    718 prioritized = _get_priority_vars_and_indexes(aligned, priority_arg, compat=compat)
    719 variables, out_indexes = merge_collected(
    720     collected, prioritized, compat=compat, combine_attrs=combine_attrs
    721 )
--> 723 dims = calculate_dimensions(variables)
    725 coord_names, noncoord_names = determine_coords(coerced)
    726 if explicit_coords is not None:

File ~/FHIMP/virtual_env/fhimp_venv/lib/python3.10/site-packages/xarray/core/variable.py:3257, in calculate_dimensions(variables)
   3255             last_used[dim] = k
   3256         elif dims[dim] != size:
-> 3257             raise ValueError(
   3258                 f"conflicting sizes for dimension {dim!r}: "
   3259                 f"length {size} on {k!r} and length {dims[dim]} on {last_used!r}"
   3260             )
   3261 return dims

ValueError: conflicting sizes for dimension 'gru': length 1 on 'landcover_names' and length 0 on {'gru': 'gru'}

[kasra-keshavarz]

Can you please upload the meshflow workflow you're using?

[kasra-keshavarz]

I also notice that your landcover fraction .csv file has an extra index column. Removing it could be the solution. Let me know.

[ShervanGharari]

@Adrien63ro I have tested Kasra's suggestion and it works. In your landsat file, you have a redundant column, the first column. please delete that column. the file you have is like:

,COMID,lat,lon,frac_1,frac_2,frac_3,frac_4,frac_5,frac_6,frac_7,frac_8,frac_9,frac_10,frac_11,frac_12,frac_13,frac_14,frac_15,frac_16,frac_17,frac_18,frac_19
1,72030083,48.7895340694417,-67.520446876425,0.15151220219793,1e-05,1e-05,1e-05,1e-05,0.465744662255012,1e-05,1e-05,1e-05,1e-05,1e-05,1e-05,1e-05,1e-05,0.282836374672032,1e-05,0.099906760875026,1e-05,1e-05
2,72030124,49.0428112709292,-54.7841663389144,0.32172392370621,0,0,0,0,0.238985142746144,0,0.278923476155061,0,0,0,0,0,0.093989125103776,0,0,0,0.0663783322888093,0

it should be like this:

COMID,lat,lon,frac_1,frac_2,frac_3,frac_4,frac_5,frac_6,frac_7,frac_8,frac_9,frac_10,frac_11,frac_12,frac_13,frac_14,frac_15,frac_16,frac_17,frac_18,frac_19
72030083,48.78953407,-67.52044688,0.151512202,1.00E-05,1.00E-05,1.00E-05,1.00E-05,0.465744662,1.00E-05,1.00E-05,1.00E-05,1.00E-05,1.00E-05,1.00E-05,1.00E-05,1.00E-05,0.282836375,1.00E-05,0.099906761,1.00E-05,1.00E-05
72030124,49.04281127,-54.78416634,0.321723924,0,0,0,0,0.238985143,0,0.278923476,0,0,0,0,0,0.093989125,0,0,0,0.066378332,0

you can just open your file with excel and remove the first column, or you can do this in python using pandas (suggested). Let us know if after this change you are successful. We close this issue after your confirmation.

[Adrien63ro]

Hi guys,

Yes indeed ! This was the problem. I removed the first column and mesh_flow.ipynb is now working. I just don’t know why I got this column. Anyway, this issue is solved.

Thanks again for your help that is very appreciated.

Have a good afternoon,

Adrien

De : Shervan Gharari @.> Envoyé : 13 février 2024 01:17 À : kasra-keshavarz/meshflow @.> Cc : Adrien Pierre @.>; Mention @.> Objet : Re: [kasra-keshavarz/meshflow] Error when running exp1.run() in mesf_flow.ipynb (Issue #15)

@Adrien63rohttps://github.com/Adrien63ro I have tested Kasra's suggestion and it works. In your landsat file, you have a redundant column, the first column. please delete that column. the file you have is like:

,COMID,lat,lon,frac_1,frac_2,frac_3,frac_4,frac_5,frac_6,frac_7,frac_8,frac_9,frac_10,frac_11,frac_12,frac_13,frac_14,frac_15,frac_16,frac_17,frac_18,frac_19

1,72030083,48.7895340694417,-67.520446876425,0.15151220219793,1e-05,1e-05,1e-05,1e-05,0.465744662255012,1e-05,1e-05,1e-05,1e-05,1e-05,1e-05,1e-05,1e-05,0.282836374672032,1e-05,0.099906760875026,1e-05,1e-05

2,72030124,49.0428112709292,-54.7841663389144,0.32172392370621,0,0,0,0,0.238985142746144,0,0.278923476155061,0,0,0,0,0,0.093989125103776,0,0,0,0.0663783322888093,0

it should be like this:

COMID,lat,lon,frac_1,frac_2,frac_3,frac_4,frac_5,frac_6,frac_7,frac_8,frac_9,frac_10,frac_11,frac_12,frac_13,frac_14,frac_15,frac_16,frac_17,frac_18,frac_19

72030083,48.78953407,-67.52044688,0.151512202,1.00E-05,1.00E-05,1.00E-05,1.00E-05,0.465744662,1.00E-05,1.00E-05,1.00E-05,1.00E-05,1.00E-05,1.00E-05,1.00E-05,1.00E-05,0.282836375,1.00E-05,0.099906761,1.00E-05,1.00E-05

72030124,49.04281127,-54.78416634,0.321723924,0,0,0,0,0.238985143,0,0.278923476,0,0,0,0,0,0.093989125,0,0,0,0.066378332,0

you can just open your file with excel and remove the first column, or you can do this in python using pandas (suggested). Let us know if after this change you are successful. We close this issue after your confirmation.

— Reply to this email directly, view it on GitHubhttps://github.com/kasra-keshavarz/meshflow/issues/15#issuecomment-1940489328, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ATM4WUREVVOEB7RU4NM3VULYTMAMLAVCNFSM6AAAAABDFF7XWWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBQGQ4DSMZSHA. You are receiving this because you were mentioned.Message ID: @.***>