Trouble to generate nc file

[mgaacGitHub]

Hi, When i select csv and matlab format for the outputs, all is ok, but are problems with Netcdf option. I get this message in the last line of proccessing of the example data.

TypeError: '(slice(None, None, None), slice(None, None, None))' is an invalid key

How can i solve this issue ?? Thanks Miguel

[mengqi-z]

Hi @mgaacGitHub, sorry that you ran into this issue. I fixed the bug and pushed the changes to the dev branch. Please clone the dev branch and run your model again. git clone -b dev --single-branch https://github.com/JGCRI/xanthos.git

Hope this fixes the issue!

[mgaacGitHub]

Dear Mengqi, Thanks a lot for your help. Only kept this problem "ValueError: could not convert string to float: 'USA'" when try to save outputs aggregated by GCAM region.

By the way, what is the grid resolution of the outputs and how many grids there are in N-S and W-E direction? because output is an vector of 67420 points that i can't understand how to distribute in 2-dimension to draw a map for each timestep. This is because NetCDF outputs dimension is (67420,31) that understand is (NumberPointsInLat x NumberPointsInLon, NumberYears)

[mengqi-z]

Now it should be fixed. Please clone the dev branch again and test the NetCDF outputs.

The grid resolution is 0.5 degree, so in theory there should be 259,200 grid cells (360 × 720). But in xanthos, we only uses 67420 grid cells that are identified as land grids. If you check out example\input\reference\coordinates.csv, you will see the id and corresponding lat and lon.

[mgaacGitHub]

Dear Mengqi, Now, when i delete xanthos directory in ~/mamba/envs/xanthos/lib/python3.7/site-packages/ (as i done afeter your first answer) and i clone Xanthos from dev branch and run Xanthos as befor, i get this msg

ModuleNotFoundError: No module named 'xarray'

AttributeError: module 'xanthos' has no attribute 'run_model'

[mengqi-z]

You will need to install Xanthos again so that the required python packages will be installed. Please use this command to clone and install Xanthos: python -m pip install git+https://github.com/JGCRI/xanthos.git@dev

[mgaacGitHub]

Dear Mengqi, I re-installed Xanthos and ran the model and get errors. i am using python 3.9.18

After, i remove xanthos/ directory in ~/mamba/envs/xanthos/lib/python3.9/site-packages/ and i downloaded it from git clone -b dev --single-branch https://github.com/JGCRI/xanthos.git I ran the model and got those messages:

/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/xanthos/hydropower/actual.py:104: FutureWarning: Calling int on a single element Series is deprecated and will raise a TypeError in the future. Use int(ser.iloc[0]) instead return int(self.loc_refs[(self.loc_refs["long"] == lon) & (self.loc_refs["lati"] == lat)]["ID"]) INFO: ---Hydropower Actual has finished successfully: 20.673751831054688 seconds ------ INFO: ---Start Diagnostics: Traceback (most recent call last): File "", line 1, in File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/xanthos/model.py", line 121, in run_model xth.execute() File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/xanthos/model.py", line 94, in execute results = config_runner.run() File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/xanthos/configurations.py", line 133, in run c.diagnostics() File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/xanthos/components.py", line 437, in diagnostics Diagnostics(self.s, self.Q, self.data) File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/xanthos/diagnostics/diagnostics.py", line 82, in init self.write_diagnostics('Basin', ref.basin_ids, ref.basin_names, q, qq, wbm, wbmc, UNH, 1) File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/xanthos/diagnostics/diagnostics.py", line 130, in write_diagnostics agg_df.loc[-1, 1:] = agg_df.sum(numeric_only=True) File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/pandas/core/indexing.py", line 845, in setitem indexer = self._get_setitem_indexer(key) File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/pandas/core/indexing.py", line 710, in _get_setitem_indexer return self._convert_tuple(key) File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/pandas/core/indexing.py", line 927, in _convert_tuple keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)] File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/pandas/core/indexing.py", line 927, in keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)] File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/pandas/core/indexing.py", line 1382, in _convert_to_indexer return labels._convert_slice_indexer(key, kind="loc") File "/home/miguel/mamba/envs/xanthos/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 4126, in _convert_slice_indexer raise TypeError( TypeError: Slicing a positional slice with .loc is not allowed, Use .loc with labels or .iloc with positions instead.

[mengqi-z]

Hi @mgaacGitHub, it seems that you didn't directly use the command python -m pip install git+https://github.com/JGCRI/xanthos.git@dev, which would setup Xanthos and install the dependencies required for Xanthos.

If you choose to clone it first, then you will need to install xanthos (and its dependencies) by navigating to the cloned xanthos folder, and try python setup.py install or pip install . This way we can avoid some package version issues. Could you try this and let me know if Xanthos runs successfully? Thanks!

[mgaacGitHub]

Dear mengqi

I installed xanthos without Conda. I applyed in a terminal: sudo python -m pip install git+https://github.com/JGCRI/xanthos.git@dev

After, in a new terminal, i run python, import and run xanthos, using the file example/pm_abcd_mrtm.ini

* First question: I change OutputFormat from 1 to 0 and the outputs i get are in format csv for both cases. What it could be happening ?

** Second question I get this msg: return labels._convert_slice_indexer(key, kind="loc") File "/home/miguel/.local/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 4126, in _convert_slice_indexer raise TypeError( TypeError: Slicing a positional slice with .loc is not allowed, Use .loc with labels or .iloc with positions instead.

And only generate this output files (only it's updated the last three)

-rw-rw-r-- 1 39659609 oct 24 18:18 pet_km3peryear_PruebaExample_1971_2001.csv -rw-rw-r-- 1 40707306 oct 24 18:18 aet_km3peryear_PruebaExample_1971_2001.csv -rw-rw-r-- 1 41393002 oct 24 18:18 q_km3peryear_PruebaExample_1971_2001.csv -rw-rw-r-- 1 38336405 oct 24 18:18 soilmoisture_km3peryear_PruebaExample_1971_2001.csv -rw-rw-r-- 1 33786089 oct 24 18:18 avgchflow_m3persec_PruebaExample_1971_2001.csv -rw-rw-r-- 1 18806 oct 24 18:18 GCAMRegion_runoff_km3peryear_PruebaExample_1971_2001.csv -rw-rw-r-- 1 20377 oct 27 12:58 Diagnostics_Runoff_Basin_Scale_km3peryr.csv -rw-rw-r-- 1 17434 oct 27 12:58 Diagnostics_Runoff_Country_Scale_km3peryr.csv -rw-rw-r-- 1 2869 oct 27 12:58 Diagnostics_Runoff_Region_Scale_km3peryr.csv -rw-rw-r-- 1 138783 nov 15 19:53 accessible_water_km3peryr_PruebaExample_1971_2001.csv -rw-rw-r-- 1 19367 nov 15 19:53 tech_hydro_pot_by_gcam_region_EJperyr_PruebaExample_1971_2001.csv -rw-rw-r-- 1 19162 nov 15 19:53 tech_expliot_hyd_pot_by_gcam_region_EJperyr_PruebaExample_1971_2001.csv -rw-rw-r-- 1 19260 nov 15 19:53 actual_hydro_by_gcam_region_EJperyr_PruebaExample_1971_2001.csv

* Third question I would like to get outputs in mat format, does it work that output format actually in Xanthos ?? or is something similar to netcdf output format.

Postdata: In the begining of the run, there is this message with respecto to python version 2.

/home/miguel/.local/lib/python3.10/site-packages/xanthos/data_reader/data_load.py:360: UserWarning: Reading .npy or .npz file required additional header parsing as it was created on Python 2. Save the file again to speed up loading and avoid this warning.

Best regards

[mgaacGitHub]

Dear mengqi, Now i cant generate outputs with csv format when install Xanthos with python -m pip install git+https://github.com/JGCRI/xanthos.git@dev

I get this message raise TypeError( TypeError: Slicing a positional slice with .loc is not allowed, Use .loc with labels or .iloc with positions instead.

Could you help me please. Best regards

[mengqi-z]

Hi @mgaacGitHub,

Could you please show me the full traceback error message? I cannot really tell what is happening without seeing the full picture. I tested Xanthos and didn't run into the issues you had. I suspect it is package version problem related to pandas. You can also try to create a new python virtual environment and install xanthos to that environment and see if it works.

[mgaacGitHub]

Hi, I installed Xanthos in mamba/Conda Use this libreries mamba create -n xanthos requests=2.31.0 joblib=1.0.1 matplotlib=3.4.3 numpy=1.24.4 scipy=1.6 pandas=1.1.5 configobj=5.0.8 python=3.7

When run Xanthos, i got this message and didn't generate outputs:

/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/hydropower/actual.py:104: FutureWarning: Calling int on a single element Series is deprecated and will raise a TypeError in the future. Use int(ser.iloc[0]) instead return int(self.loc_refs[(self.loc_refs["long"] == lon) & (self.loc_refs["lati"] == lat)]["ID"]) INFO: ---Hydropower Actual has finished successfully: 16.57504177093506 seconds ------ INFO: ---Start Diagnostics: Traceback (most recent call last): File "", line 1, in File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/model.py", line 121, in run_model xth.execute() File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/model.py", line 94, in execute results = config_runner.run() ^^^^^^^^^^^^^^^^^^^ File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/configurations.py", line 133, in run c.diagnostics() File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/components.py", line 437, in diagnostics Diagnostics(self.s, self.Q, self.data) File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/diagnostics/diagnostics.py", line 82, in init self.write_diagnostics('Basin', ref.basin_ids, ref.basin_names, q, qq, wbm, wbmc, UNH, 1) File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/diagnostics/diagnostics.py", line 130, in write_diagnostics agg_df.loc[-1, 1:] = agg_df.sum(numeric_only=True)

  File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 845, in __setitem__
    indexer = self._get_setitem_indexer(key)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 710, in _get_setitem_indexer
    return self._convert_tuple(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 927, in _convert_tuple
    keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 927, in <listcomp>
    keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 1382, in _convert_to_indexer
    return labels._convert_slice_indexer(key, kind="loc")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 4126, in _convert_slice_indexer
    raise TypeError(
TypeError: Slicing a positional slice with .loc is not allowed, Use .loc with labels or .iloc with positions instead.

[mgaacGitHub]

Dear Mengqi I have tried to install and run Xanthos in a virtuall session.

1. When i install xanthos with "pip install xanthos" and run Xanthos i get csv output but fail Netcdf format.
2. When i install xanthos with "python -m pip install git+https://github.com/JGCRI/xanthos.git@dev" and run Xanthos i get error to generate csv and Netcdf format
3. When a change xanthos/ directory by "git clone -b dev --single-branch https://github.com/JGCRI/xanthos.git" and run Xanthos i get error to generate csv and Netcdf format.

I am attaching two files, "Installation_Run_Xanthos.txt" with messages when i run Xanthos and "InstallationMsg.txt" with steps and messages during Xanthos installation. The number 1) ... 6) are for the same attempts.

Best regards Installation_Run_Xanthos.txt InstallationMsg.txt

[mgaacGitHub]

Dear Mengqi

I was finally able to install Xanthos from GitHub out of virtual session. Thanks a lot for your help.

I installed python3.7 https://www.linuxcapable.com/how-to-install-python-3-7-on-ubuntu-linux/

After i installed libraries in .local/ directory python3.7 -m pip install requests==2.31.0 joblib==1.0.1 matplotlib==3.4.3 numpy==1.19.4 scipy==1.6 pandas==1.1.5 configobj==5.0.8 python3.7 -m pip install python-dateutil pytz pillow --force-reinstall --upgrade

I put .local/ directory in .bashrc file export PATH=/home/miguel/.local/bin:$PATH

I installed Xanthos python3.7 -m pip install xanthos

Xanthos run ok, at least generates csv files.

Best regards, Miguel

[mengqi-z]

Hi @mgaacGitHub,

I am glad you solved the issue! I wanted to thank you for bringing up these bugs, so we can improve it in the future.

I checked the following errors you got, and it seems like you created a virtual environment with Python 3.7, but the actual python environment you are using for Xanthos uses python 3.11. This could mean the actual python environment you are using has the pandas version that might not work with Xanthos.

Another note is that the Xanthos on the dev branch is able to produce NetCDF. But it seems the version that works for you is the released version, which has the NetCDF issue.

Hi, I installed Xanthos in mamba/Conda Use this libreries mamba create -n xanthos requests=2.31.0 joblib=1.0.1 matplotlib=3.4.3 numpy=1.24.4 scipy=1.6 pandas=1.1.5 configobj=5.0.8 python=3.7

When run Xanthos, i got this message and didn't generate outputs:

/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/hydropower/actual.py:104: FutureWarning: Calling int on a single element Series is deprecated and will raise a TypeError in the future. Use int(ser.iloc[0]) instead return int(self.loc_refs[(self.loc_refs["long"] == lon) & (self.loc_refs["lati"] == lat)]["ID"]) INFO: ---Hydropower Actual has finished successfully: 16.57504177093506 seconds ------ INFO: ---Start Diagnostics: Traceback (most recent call last): File "", line 1, in File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/model.py", line 121, in run_model xth.execute() File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/model.py", line 94, in execute results = config_runner.run() ^^^^^^^^^^^^^^^^^^^ File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/configurations.py", line 133, in run c.diagnostics() File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/components.py", line 437, in diagnostics Diagnostics(self.s, self.Q, self.data) File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/diagnostics/diagnostics.py", line 82, in init self.write_diagnostics('Basin', ref.basin_ids, ref.basin_names, q, qq, wbm, wbmc, UNH, 1) File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/xanthos/diagnostics/diagnostics.py", line 130, in write_diagnostics agg_df.loc[-1, 1:] = agg_df.sum(numeric_only=True) ~~^^^^^^^^ File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 845, in setitem indexer = self._get_setitem_indexer(key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 710, in _get_setitem_indexer return self._convert_tuple(key) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 927, in _convert_tuple keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 927, in keyidx = [self._convert_to_indexer(k, axis=i) for i, k in enumerate(key)] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexing.py", line 1382, in _convert_to_indexer return labels._convert_slice_indexer(key, kind="loc") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/miguel/mamba/envs/xanthos/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 4126, in _convert_slice_indexer raise TypeError( TypeError: Slicing a positional slice with .loc is not allowed, Use .loc with labels or .iloc with positions instead.