search

UMEP-dev / UMEP

Urban Multi-scale Environmental Predictor
https://umep-docs.readthedocs.io/
59 stars 15 forks source link

LQf Python error #304

Open S9LOU opened 3 years ago

S9LOU commented 3 years ago

The bug We ran the LQF processor with local data("China Shanghai") following its annual (https://umep-docs.readthedocs.io/en/latest/OtherManuals/LQF_Manual.html#lqf-manual) . However, it failed.

  1. The first bug is
    addQuotes(x): return "'" + x + "'" ： type error . Then we change the source python file to addQuotes(x): return "'" + str(x) + "'".
  2. After changing RegionalParameters.py, we got the other error message as below. Exception: The LQF database contains no weekday building cycles for: {NULL}

However, the input database contains "China_Shanghai" in both WeeklyBuildingCylcles and World table. Additionally, we have tried it on another computer with QGIS Version 3.10.13, but also got the same error. The log and the data have been attached below. We have used the LQf processor on QGIS2 version based on those data in 2018 and it succeeded. Is there anything wrong with the processor or our data?

Desktop:

Screenshots image When press the button Run model, an error occurs below as logs.

Logs

  1. first one 
TypeError: can only concatenate str (not "QVariant") to str 
Traceback (most recent call last):
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker
    self.model.run()
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run
    self.setupAndRun(self.config['startDates'], self.config['endDates'])
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 276, in setupAndRun
    regPar.injectAttributedOutputLayer(os.path.join(self.downscaledPath, al['file']), al['EPSG'], outFeatIds)
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 197, in injectAttributedOutputLayer
    self.extractPropertiesForCountries(con, countries)
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 210, in extractPropertiesForCountries
    attrs = "SELECT * FROM attributes WHERE id IN " + "(" + ','.join(map(addQuotes, countries)) + ") ORDER BY as_of_year ASC"
  File "C:/Users/ax/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 15, in addQuotes
    def addQuotes(x): return "'" + x + "'"
TypeError: can only concatenate str (not "QVariant") to str
  1. second one 
2021-07-02T11:15:55     WARNING    Traceback (most recent call last):
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker
              self.model.run()
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run
              self.setupAndRun(self.config['startDates'], self.config['endDates'])
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 276, in setupAndRun
              regPar.injectAttributedOutputLayer(os.path.join(self.downscaledPath, al['file']), al['EPSG'], outFeatIds)
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 197, in injectAttributedOutputLayer
              self.extractPropertiesForCountries(con, countries)
              File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 218, in extractPropertiesForCountries
              raise Exception('The LQF database contains no weekday building cycles for: ' + str(diffs))
             Exception: The LQF database contains no weekday building cycles for: {NULL}

Data Since GitHub didn't support uploading our filetype, the dataset online link is: https://cloud.tsinghua.edu.cn/d/0bf54aea37da46589226/
The zip file is : input.zip

It includes the LQFDatabase_V1-1.sqlite and Shanghai Pop shapefile,

biglimp commented 3 years ago

Were you able to go through the London tutorial without any errors?

https://umep-docs.readthedocs.io/projects/tutorial/en/latest/Tutorials/LQF.html#lqf

S9LOU commented 3 years ago

Hi, thank you for your kind reply. We could go through the London tutorial without any errors, indicating the LQf tool worked successfully. However, when we double checked our data and python code error, we found some interesting problems with the data type.

  1. Firstly, the tool couldn’t deal with the null data in our shapefiles. “TypeError: can only concatenate str (not "QVariant") to str” in above issue, is probably due to that the “admin” field of area_allocations shape file contains null value. However, we have checked the “admin” field of our data has the same datatype(Qstring) as area_allocations shape file extracted from the sample data. Therefore, we are not sure whether the null value is QVariant type in the python processing or whether the null value leads to the python error. It’s impossible for us to manually correct null type since the “admin” field is generated from the preprocess session automatically. 

- Besides its admin, area_allocations shapefile’s pop field contains null, leading to a similar error. However, it also shows double type in the QGIS same with the sample data. The error log is attached below.

An error has occurred while executing Python code:

TypeError: unsupported operand type(s) for : 'float' and 'QVariant' Traceback (most recent call last): File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\array_ops.py", line 143, in na_arithmetic_op result = expressions.evaluate(op, left, right) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\computation\expressions.py", line 233, in evaluate return _evaluate(op, op_str, a, b) # type: ignore File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\computation\expressions.py", line 119, in _evaluate_numexpr result = _evaluate_standard(op, op_str, a, b) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\computation\expressions.py", line 68, in _evaluate_standard return op(a, b) TypeError: unsupported operand type(s) for : 'float' and 'QVariant'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker self.model.run() File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run self.setupAndRun(self.config['startDates'], self.config['endDates']) File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 382, in setupAndRun energyUse = regPar.getEnergyUse(areas.index, d_lookup.year) File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\RegionalParameters.py", line 396, in getEnergyUse return data['kwh_year'] data['resPop']/data['population'] # Disaggregate energy use by building borne population File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\common.py", line 65, in new_method return method(self, other) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops__init__.py", line 343, in wrapper result = arithmetic_op(lvalues, rvalues, op) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\array_ops.py", line 190, in arithmetic_op res_values = na_arithmetic_op(lvalues, rvalues, op) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\array_ops.py", line 150, in na_arithmetic_op result = masked_arith_op(left, right, op) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\ops\array_ops.py", line 92, in masked_arith_op result[mask] = op(xrav[mask], yrav[mask]) TypeError: unsupported operand type(s) for : 'float' and 'QVariant'

-  After we deleted those grids containing the null value of pop or admin, it worked. However, it’s impossible for us to delete all the null value in a larger area. Moreover, LQf with QGIS2 version could process exactly the same grid without any error.  
- **If the LQf still can deal with the null value now? If yes, why it failed this time? If not, is there any suggested way for us to correct our data rather than simply deleted grids.**

2. Secondly, the tool requires the date of daily temperature csv file must be monotonic increasing or decreasing. Our daily temperature (2014-11-30 to 2015-11-30) have its time zone field starting from 335 to 365, and following 1 to 334.  
That’s weird since when we used to put the same daily temperature csv file into the LQf with QGIS2 version it worked. **Is it still allowed to process such daily temperature file?** 

- The temperature data is attached below.  

[dailyTemperature_2015_Shanghai.zip](https://github.com/UMEP-dev/UMEP/files/6768743/dailyTemperature_2015_Shanghai.zip)

- The error log is attached below.

`An error has occurred while executing Python code:

ValueError: index must be monotonic increasing or decreasing Traceback (most recent call last): File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\LQF.py", line 581, in startWorker self.model.run() File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 212, in run self.setupAndRun(self.config['startDates'], self.config['endDates']) File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\LUCY.py", line 386, in setupAndRun temperatures.append(dailyT.getTemp(d_lookup_temperature, timeStepDuration)[0]) # Mean daily temperature File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\DailyTemperature.py", line 59, in getTemp return self.temperature.getValueForDateTime(requestDateTime, timeBinDuration) File "C:/Users/administrant/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\UMEP\LucyQF\PythonLUCY\DataManagement\GenericAnnualSampler.py", line 384, in getValueForDateTime minDiff = dsts[dstDates].index.asof(localLookupTimeBin.date()) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 4356, in asof loc = self.get_loc(label, method="pad") File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2905, in get_loc indexer = self.get_indexer([key], method=method, tolerance=tolerance) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 2994, in get_indexer indexer = self._get_fill_indexer(target, method, limit, tolerance) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 3034, in _get_fill_indexer indexer = self._get_fill_indexer_searchsorted(target, method, limit) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 3057, in _get_fill_indexer_searchsorted indexer[nonexact] = self._searchsorted_monotonic(target[nonexact], side) File "C:\Users\administrant\AppData\Roaming\Python\Python37\site-packages\pandas\core\indexes\base.py", line 5050, in _searchsorted_monotonic raise ValueError("index must be monotonic increasing or decreasing") ValueError: index must be monotonic increasing or decreasing `

suegrimmond commented 3 years ago

Does you python suggested fix work? I.e. did you try it?