search

UMEP-dev / UMEP

Urban Multi-scale Environmental Predictor
https://umep-docs.readthedocs.io/
59 stars 15 forks source link

UWG Analyzer takes the number of cells instead of the lists of id #407

Closed PetitMote closed 2 years ago

PetitMote commented 2 years ago

Describe the bug The UWG Analyzer will use the number of cells instead of the actual list of id. This means that it’s ok when your id start at 0 and are continuous. However, when your id don’t start at 0, and/or have gap in them, there will be 3 errors:

Also, my generated raster is completely bugged, and it might be because of the same issue.

To Reproduce Steps to reproduce the behavior:

  1. Generate a grid. Cut some lines and/or columns, so that the id don’t start at 0 and/or there are gaps.
  2. Go through the UWG model
  3. Try the UWG analyzer, adding result to the vector grid attribute table
  4. See error

Desktop (please complete the following information):

I can upload data if needed: the grid, uwg input and output, if necessary.

Thank you

PetitMote commented 2 years ago

In fact, here is the data

uwg_analyzer_cell_error.zip

PetitMote commented 2 years ago

So more information:

By the way, how are the rasters supposed to look ? They have only 2 values, 0 or n (n being 2.5, or 4.707… depending on the aggregation)

biglimp commented 2 years ago

What are the actual error messages? Please paste them here.

PetitMote commented 2 years ago

When using the menu based analyzer, I don’t get any error message (but it’s the same result). When using the processor, I get:

[…]
2022-04-28T11:49:04     CRITICAL    Couche grid_stfelix_multipoly : L'entité 80 pour la mise à jour de l'attribut n'a pas été trouvée.
2022-04-28T11:49:04     CRITICAL    Couche grid_stfelix_multipoly : L'entité 89 pour la mise à jour de l'attribut n'a pas été trouvée.
2022-04-28T11:49:04     CRITICAL    Couche grid_stfelix_multipoly : L'entité 90 pour la mise à jour de l'attribut n'a pas été trouvée.
2022-04-28T11:49:05     CRITICAL    Couche grid_stfelix_multipoly : L'entité 99 pour la mise à jour de l'attribut n'a pas été trouvée.
2022-04-28T11:49:05     CRITICAL    Couche grid_stfelix_multipoly : L'entité 100 pour la mise à jour de l'attribut n'a pas été trouvée.
2022-04-28T11:49:05     CRITICAL    Couche grid_stfelix_multipoly : L'entité 109 pour la mise à jour de l'attribut n'a pas été trouvée.
2022-04-28T11:49:06     CRITICAL    Couche grid_stfelix_multipoly : L'entité 110 pour la mise à jour de l'attribut n'a pas été trouvée.
2022-04-28T11:49:06     CRITICAL    Couche grid_stfelix_multipoly : L'entité 119 pour la mise à jour de l'attribut n'a pas été trouvée.

It’s only in the global journal, not in the processor journal. Translation:

2022-04-28T11:49:06 CRITICAL Layer grid_stfelix_multipoly: The entity 119 for the update of the attribute was not found

biglimp commented 2 years ago

I cannot read french. CAn you change your QGIS to english please.

biglimp commented 2 years ago

Hm, I ran your data with the menu-based UMEP and I cannot replicate your error. Below your see the settings and the result I got.

image

image

PetitMote commented 2 years ago

I cannot read french. CAn you change your QGIS to english please.

That’s why I’ve put a translation below:

2022-04-28T11:49:06 CRITICAL Layer grid_stfelix_multipoly: The entity 119 for the update of the attribute was not found

The error comes only with the "Add result to polygon grid" option. With the menu based analyzer, it’s silent: you can see after that a part of the grid will have a null value in the new attribute.

What’s strange is that raster creation wasn’t working before I changed the id. Now that it’s changed, I get this kind of result: image

Edit: I’m starting to think I’m gonna try on my linux at home

PetitMote commented 2 years ago

I tried at home on Archlinux and got the same raster, sadly.

PetitMote commented 2 years ago

Maybe it’s a dependency? Here is my pip freeze:

Edit: I’ve updated each and every library, still the same. I’ll try reinstalling qgis.

algopy==0.5.7
asteval==0.9.26
atmosp==0.2.9
cdsapi==0.5.1
certifi==2020.6.20
cftime==1.6.0
chardet==3.0.4
click==8.1.2
cloudpickle==2.0.0
colorama==0.4.4
coverage==5.3
cycler==0.10.0
dask==2022.4.0
decorator==4.4.2
dill==0.3.4
ExifRead==2.3.2
f90nml==1.3.1
fonttools==4.28.5
fsspec==2022.3.0
future==0.18.2
GDAL==3.4.2
geographiclib==1.50
h5py==3.6.0
httplib2==0.18.1
idna==2.10
Jinja2==2.11.2
kiwisolver==1.2.0
linecache2==1.0.0
lmfit==1.0.3
locket==0.2.1
lxml==4.6.2
MarkupSafe==1.1.1
matplotlib==3.5.1
mock==4.0.2
multiprocess==0.70.12.2
netCDF4==1.5.8
networkx==2.5
nose==1.3.7
nose2==0.9.2
numdifftools==0.9.40
numexpr==2.8.1
numpy==1.22.3
OWSLib==0.25.0
packaging==21.3
pandas==1.4.2
partd==1.2.0
patsy==0.5.2
Pillow==8.0.1
Pint==0.19.1
Platypus-Opt==1.0.4
plotly==4.12.0
ply==3.11
psycopg2-binary==2.8.6
pvlib==0.9.1
Pygments==2.7.2
pyodbc==4.0.30
PyOpenGL==3.1.5
pyparsing==2.4.7
pypiwin32==223
pyproj==3.2.1
PyQt5==5.15.4
PyQt5-sip==12.8.1
python-dateutil==2.8.1
pytz==2020.1
pywin32==228
PyYAML==5.3.1
requests==2.24.0
retrying==1.3.3
scipy==1.8.0
seaborn==0.11.2
Shapely==1.8.1.post1
simplejson==3.17.2
sip @ file:///D:/src/osgeo4w/src/python3-sip/sip-6.1.1/dist/sip-6.1.1-cp39-cp39-win_amd64.whl
six==1.15.0
statsmodels==0.13.2
supy==2022.4.7
supy-driver==2021a5
toml==0.10.1
toolz==0.11.2
tqdm==4.64.0
traceback2==1.4.0
uncertainties==3.1.6
unittest2==1.1.0
urllib3==1.25.11
uwg==5.8.9
wxPython @ file:///D:/src/osgeo4w/src/python3-wxpython/wxPython-4.1.1/dist/wxPython-4.1.1-cp39-cp39-win_amd64.whl
xarray==2022.3.0
xlrd==1.2.0
xlwt==1.3.0
biglimp commented 2 years ago

Tried now with adding result to the shapefile and it worked, hense I cannot replicate your error. I have noticed that on Windows the two columns (mean and max) does not show in the attribute table after the UWG Analyzer have been executed. Then try to refresh your table and the two new columns will show up.

I cannot see that any of your python libraries should cause any issues. I added my pip list below.

image

C:\Users\xlinfr\Documents\PythonScripts\processing_umep>pip list Package Version

alabaster 0.7.12 algopy 0.5.7 argon2-cffi 21.1.0 asteval 0.9.25 astroid 2.6.6 async-generator 1.10 atmosp 0.2.9 attrs 21.2.0 Babel 2.9.1 backcall 0.2.0 bleach 4.1.0 Cartopy 0.19.0.post1 cdsapi 0.5.1 certifi 2020.6.20 cffi 1.14.6 cftime 1.5.0 chardet 3.0.4 click 8.0.1 click-plugins 1.1.1 cligj 0.7.0 cloudpickle 1.6.0 colorama 0.4.4 coverage 5.3 cycler 0.10.0 dask 2021.6.2 debugpy 1.4.1 decorator 4.4.2 defusedxml 0.7.1 dill 0.3.4 docutils 0.17.1 entrypoints 0.3 et-xmlfile 1.1.0 ExifRead 2.3.2 f90nml 1.3.1 Fiona 1.8.20 fonttools 4.28.5 fsspec 2021.6.1 future 0.18.2 GDAL 3.4.2 geographiclib 1.50 httplib2 0.18.1 idna 2.10 imagesize 1.2.0 importlib-metadata 4.6.4 ipykernel 6.2.0 ipython 7.26.0 ipython-genutils 0.2.0 ipywidgets 7.5.1 isort 5.9.3 JayDeBeApi 1.2.3 jedi 0.18.0 Jinja2 2.11.2 JPype1 1.3.0 jsonschema 3.2.0 jupyter 1.0.0 jupyter-client 6.1.7 jupyter-console 6.2.0 jupyter-core 4.6.3 jupyterlab-pygments 0.1.2 kiwisolver 1.2.0 lazy-object-proxy 1.6.0 linecache2 1.0.0 llvmlite 0.34.0 lmfit 1.0.2 locket 0.2.1 lxml 4.6.2 MarkupSafe 1.1.1 matplotlib 3.5.1 matplotlib-inline 0.1.2 mccabe 0.6.1 mistune 0.8.4 mock 4.0.2 multiprocess 0.70.12.2 munch 2.5.0 nbclient 0.5.4 nbconvert 6.1.0 nbformat 5.1.3 nest-asyncio 1.5.1 netCDF4 1.5.7 networkx 2.5 nose 1.3.7 nose2 0.9.2 notebook 6.4.3 numba 0.51.2 numdifftools 0.9.40 numexpr 2.7.3 numpy 1.20.2 openpyxl 3.0.9 OWSLib 0.25.0 packaging 20.4 pandas 1.4.1 pandocfilters 1.5.0 parso 0.8.2 partd 1.2.0 patsy 0.5.1 pb-tool 3.1.0 pickleshare 0.7.5 Pillow 8.0.1 Pint 0.17 pip 21.2.4 Platypus-Opt 1.0.4 plotly 4.12.0 ply 3.11 prometheus-client 0.11.0 prompt-toolkit 3.0.19 psycopg2-binary 2.8.6 pvlib 0.8.1 pycparser 2.20 Pygments 2.7.2 pylint 2.9.6 pyodbc 4.0.30 PyOpenGL 3.1.5 pyparsing 2.4.7 pypiwin32 223 pyproj 3.2.1 PyQt5 5.15.4 PyQt5-sip 12.8.1 pyrsistent 0.18.0 pyshp 2.1.3 python-dateutil 2.8.1 pytz 2020.1 pywin32 228 pywinpty 1.1.4 PyYAML 5.3.1 pyzmq 22.2.1 qtconsole 4.7.7 QtPy 1.9.0 requests 2.24.0 retrying 1.3.3 scipy 1.6.2 seaborn 0.11.1 Send2Trash 1.8.0 setuptools 56.0.0 Shapely 1.8.0 simplejson 3.17.2 sip 6.1.1 six 1.15.0 snowballstemmer 2.1.0 Sphinx 4.3.1 sphinx-rtd-theme 1.0.0 sphinxcontrib-applehelp 1.0.2 sphinxcontrib-devhelp 1.0.2 sphinxcontrib-htmlhelp 2.0.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-qthelp 1.0.3 sphinxcontrib-serializinghtml 1.1.5 statsmodels 0.12.2 supy 2022.2.7 supy-driver 2021a2 terminado 0.12.1 testpath 0.5.0 toml 0.10.1 toolz 0.11.1 tornado 6.1 tqdm 4.61.1 traceback2 1.4.0 traitlets 5.0.5 uncertainties 3.1.5 unittest2 1.1.0 urllib3 1.25.11 uwg 5.8.9 wbgt 0.2 wcwidth 0.2.5 webencodings 0.5.1 widgetsnbextension 3.5.1 wrapt 1.12.1 wxPython 4.1.1 xarray 0.18.2 xlrd 1.2.0 xlwt 1.3.0 zipp 3.5.0

PetitMote commented 2 years ago

Alright, I tried again with the shapefile I provided, in a new project, and indeed it worked. I tried putting it in a new gpkg, and it worked too, except for just the last line having no value.

I tried in my old project, with the shapefile, and fucked up again.

I guess it was my project that was corrupted, maybe when I updated qgis.

So thank you very much! It was on my side, I wasted a lot of time, and took some of yours on thise, I’m sorry 😅

PetitMote commented 2 years ago

I might have spoken a little too faste. Average is working great, however max will give me a raster full of the same value. Is it the same for you?

biglimp commented 2 years ago

I also noticed the max column just now. It should not be there just ignore it. I will try to remove it.

PetitMote commented 2 years ago

I also noticed the max column just now. It should not be there just ignore it. I will try to remove it.

I’m sorry, I’m not sure I understand. Do you mean the max aggregation should be removed, or that the column in my grid is wrong? (in the last case, indeed, I removed it and relaunched the analysis).

biglimp commented 2 years ago

I mena that when I should add mean column in the attribute table, max is also added.

PetitMote commented 2 years ago

I mena that when I should add mean column in the attribute table, max is also added.

It’s probably because there was already a max column, no?

When I’m trying to simulate the max, I get a raster full of only 1 number, don’t you?

biglimp commented 2 years ago

There was no wrong in the columns created in the attribute table, my mistake.

Regarding why you get similar values in many grids when calculating max or median is that your output is very similar. This is not because of the analyzer as far as I can see but the output from the UWG model. Maybe you try to model an area with small variations and hence you get same max values for all grids OR the model does not perform so well. If you do a text comparison between two of your grids you will notice that the temperature and humidity output is very similar. As the model itself is taken from https://github.com/ladybug-tools/uwg I recommend to submit an issue there if you think your result looks strange.

PetitMote commented 2 years ago

Yes, results are very similar, probably because I misconfigured the Land Cover Fraction pre-processor. However, when using the mean, we can see a difference, from 3.6 to 3.7. But when I’m using the max, the output raster is only full of 8.1, or another value depending on the time period.

biglimp commented 2 years ago

Exactly, that comes from that your that same maximum in all your grids at some point in the data series.

biglimp commented 2 years ago

@PetitMote , can I close this issue or is it still active?

PetitMote commented 2 years ago

Indeed!