search

Jannik-Schilling / generate_swmm_inp

a QGIS plugin for the import and export of SWMM input files (conversion of geodata, e.g. shapefiles or geopackage into SWMM models and vice versa)
GNU General Public License v2.0
35 stars 10 forks source link

crash when generating default data #81

Closed schorsch2 closed 5 months ago

schorsch2 commented 6 months ago

I am trying to generate the default data and have selected a CRS, empty data and a folder. This ends in an exception error. `## Report Details

Python Stack Trace

Windows fatal exception: access violation

Current thread 0x00003478 (most recent call first):
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\openpyxl\styles\builtins.py", line 1346 in <module>
    ('Normal', NamedStyle.from_tree(fromstring(normal))),
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888 in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\openpyxl\styles\styleable.py", line 14 in <module>
    from .builtins import styles
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888 in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\openpyxl\cell\cell.py", line 28 in <module>
    from openpyxl.styles.styleable import StyleableObject
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888 in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\openpyxl\cell\__init__.py", line 3 in <module>
    from .cell import Cell, WriteOnlyCell, MergedCell
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888 in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\openpyxl\worksheet\worksheet.py", line 25 in <module>
    from openpyxl.cell import Cell, MergedCell
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888 in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\openpyxl\workbook\workbook.py", line 7 in <module>
    from openpyxl.worksheet.worksheet import Worksheet
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888 in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\openpyxl\workbook\__init__.py", line 4 in <module>
    from .workbook import Workbook
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888 in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\openpyxl\__init__.py", line 6 in <module>
    from openpyxl.workbook import Workbook
  File "C:\OSGeo4W/apps/qgis/./python\qgis\utils.py", line 888 in _import
    mod = _builtin_import(name, globals, locals, fromlist, level)
  File "C:\OSGeo4W\apps\Python39\lib\site-packages\pandas\io\excel\_openpyxl.py", line 56 in __init__
    from openpyxl.workbook import Workbook
  File "C:\Users/wei.IBE/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\generate_swmm_inp\g_s_read_write_data.py", line 376 in dict_to_excel
    with pd.ExcelWriter(fname) as writer:
  File "C:\Users/wei.IBE/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\generate_swmm_inp\generate_swmm_import_inp_file.py", line 586 in processAlgorithm
    dict_to_excel(
  File "C:\OSGeo4W/apps/qgis/./python/plugins\processing\gui\AlgorithmExecutor.py", line 70 in execute
    results, ok = alg.run(parameters, context, feedback, {}, False)
  File "C:\OSGeo4W/apps/qgis/./python/plugins\processing\core\Processing.py", line 176 in runAlgorithm
    ret, results = execute(alg, parameters, context, feedback, catch_exceptions=False)
  File "C:\OSGeo4W/apps/qgis/./python/plugins\processing\tools\general.py", line 114 in run
    return Processing.runAlgorithm(algOrName, parameters, onFinish=post_process, feedback=feedback, context=context)
  File "C:\Users/wei.IBE/AppData/Roaming/QGIS/QGIS3\profiles\default/python/plugins\generate_swmm_inp\generate_default_data.py", line 132 in processAlgorithm
    subalg_outputs = processing.run('GenSwmmInp:ImportInpFile', alg_params, context=context, feedback=feedback, is_child_algorithm=True)

Thread 0x00001778 (most recent call first):
  File "C:\OSGeo4W/apps/qgis/./python/plugins\processing\ProcessingPlugin.py", line 432 in executeAlgorithm
    dlg.exec_()
  File "C:\OSGeo4W/apps/qgis/./python/plugins\processing\gui\ProcessingToolbox.py", line 232 in executeAlgorithm
    self.executeWithGui.emit(alg.id(), self, self.in_place_mode, False)

Stack Trace


RtlpWaitOnCriticalSection :
RtlpEnterCriticalSectionContended :
RtlEnterCriticalSection :
xmlGetGlobalState :
_xmlDefaultSAXHandler :
xmlDefaultSAXHandlerInit :
xmlInitParserCtxt :
xmlNewParserCtxt :
PyInit_etree :
PyInit_etree :
PyInit_etree :
PyInit_etree :
PyInit_etree :
PyInit_etree :
PyInit_etree :
PyObject_Call :
Py_read :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyEval_GetBuiltins :
PyEval_GetBuiltins :
PyObject_GetBuffer :
PyVectorcall_Call :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyBytes_Repr :
PyEval_SignalReceived :
Py_gitversion :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyEval_GetBuiltins :
PyEval_GetBuiltins :
PyObject_GetBuffer :
PyVectorcall_Call :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyBytes_Repr :
PyEval_SignalReceived :
Py_gitversion :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyEval_GetBuiltins :
PyEval_GetBuiltins :
PyObject_GetBuffer :
PyVectorcall_Call :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyBytes_Repr :
PyEval_SignalReceived :
Py_gitversion :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyEval_GetBuiltins :
PyEval_GetBuiltins :
PyObject_GetBuffer :
PyVectorcall_Call :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyBytes_Repr :
PyEval_SignalReceived :
Py_gitversion :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyEval_GetBuiltins :
PyEval_GetBuiltins :
PyObject_GetBuffer :
PyVectorcall_Call :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyBytes_Repr :
PyEval_SignalReceived :
Py_gitversion :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyEval_GetBuiltins :
PyEval_GetBuiltins :
PyObject_GetBuffer :
PyVectorcall_Call :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyBytes_Repr :
PyEval_SignalReceived :
Py_gitversion :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyEval_GetBuiltins :
PyEval_GetBuiltins :
PyObject_GetBuffer :
PyVectorcall_Call :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyBytes_Repr :
PyEval_SignalReceived :
Py_gitversion :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalCodeWithName :
PyEval_EvalCodeEx :
PyEval_EvalCode :
PyEval_GetBuiltins :
PyEval_GetBuiltins :
PyObject_GetBuffer :
PyVectorcall_Call :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyObject_Call :
PyObject_Call :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalFrameDefault :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyObject_CallFunction :
PyDict_DelItem :
PyObject_CallMethodIdObjArgs :
PyArena_New :
PyImport_ImportModuleLevelObject :
PyList_Size :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyBytes_Repr :
PyEval_SignalReceived :
Py_gitversion :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyType_GenericNew :
PyType_GenericNew :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyFloat_FromDouble :
PyVectorcall_Call :
PyObject_Call :
pdal::PointContainer::freeTemp :
pdal::Option::getName :
QgsProcessingAlgorithm::runPrepared :
QgsProcessingAlgorithm::run :
pdal::Option::getName :
PyArg_ParseTuple_SizeT :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyObject_GC_Del :
PyFunction_Vectorcall :
PyEval_EvalFrameDefault :
PyFunction_Vectorcall :
PyFloat_FromDouble :
PyVectorcall_Call :
PyObject_Call :
pdal::PointContainer::freeTemp :
pdal::Option::getName :
QgsProcessingAlgorithm::runPrepared :
QgsProcessingAlgRunnerTask::run :
pdal::Option::getName :
QgsTask::start :
QThreadPoolPrivate::reset :
QThread::start :
BaseThreadInitThunk :
RtlUserThreadStart :

QGIS Info QGIS Version: 3.34.1-Prizren QGIS code revision: 133927424d Compiled against Qt: 5.15.3 Running against Qt: 5.15.3 Compiled against GDAL: 3.8.1 Running against GDAL: 3.8.1

System Info CPU Type: x86_64 Kernel Type: winnt Kernel Version: 10.0.22621 `

Jannik-Schilling commented 6 months ago

hello schorsch2, unfortunately, I cannot reproduce the error. Can you tell me which version of the plugin you are using and which version of openpyxl you are using? It seems like there is a problem with the export of excel-files.

Jannik

schorsch2 commented 6 months ago

Plugin: 0.32 python3-openpyxl: 3.0.9-1 I have tried it again in an otherwise empty project. It worked once and then unfortunately not again.

schorsch2 commented 6 months ago

grafik

Jannik-Schilling commented 6 months ago

yup, writing the tables to excel files seems to be the problem. Maybe updating openpyxl will help. Version 3.0.9 is from 2021; current version is 3.1.2; there have been several bug-fixes in this package.

schorsch2 commented 6 months ago

I have now updated openpyxl to 3.1.2 - this still causes a crash.

Jannik-Schilling commented 6 months ago

(...) I cannot reproduce the error. (...)

Now I can (on Windows). As far as I can see it happens only with empty tables.

schorsch2 commented 6 months ago

I can also generate the example tables on a freshly installed computer. This does not work on my workstation either.

Jannik-Schilling commented 6 months ago

I´ll do some tests. So far I could not find the problem.

schorsch2 commented 6 months ago

I have installed with pip install odfpy and uninstalled openpyxl. Now the creation of the default tables runs smoothly.

schorsch2 commented 6 months ago

see https://github.com/qgis/QGIS/issues/40297

MMR100 commented 5 months ago

Not working for us. I installed odfpy 1.4.1, uninstalled openyxl 3.1.2 and QGIS (3.28.10) is still crashing.

Jannik-Schilling commented 5 months ago

ok.

I am generally not happy with the dependence from packages like openpyxl / odfpy etc. As far as I can test it, the current approach works fine on Linux, but crashes (with different error reports) on Windows.

I tried to replace the code which is writing the xlsx / odf files with the native tool "export to spreadsheet" in qgis. However, it does not write column headers for empty tables so far. I'm currently trying to fill the tables with empty strings in order to prevent the crash... still need some time for testing.

If any of you has a another idea... please write your suggestions. The function in the currently published plugin version starts here: https://github.com/Jannik-Schilling/generate_swmm_inp/blob/0336fc2280ad83db6a26b85bb92898228ad93cf6/g_s_read_write_data.py#L345

A totally different approach would be replacing the xlsx / xls / odf tables with geopackage files. This would be much easier to write but users might prefer editing timeseries with a different software than QGIS.

MMR100 commented 5 months ago

This is a great idea to suggest gpkg but indeed many users might prefer Excel. What about csv files? FYI I used your plugin last year with QGIS 3.16 and the creation of default files was working. (But I cannot test it again as our organization is controlling our computers)

Jannik-Schilling commented 5 months ago

recent commits seem to work to in branch dev_0_33: 705350f6d06747ba9e2d9b2227b87e285bb399e0, a0218295c63a3f6b3a96e95c49f7b3b8f31224ac, ec573cce8d53ca4db3339e2aa523b0ac6d5e9bad, and 2f698a9ab5dcb5068f4cc54d660975eebcfa7a06

There is still a problem with the timeseries and hydrographs sections (date and time formats).

schorsch2 commented 5 months ago

Yes, I have installed dev_0_33 and the data is generated without crashing. Thank you very much!

Jannik-Schilling commented 5 months ago

There are still some bugs. The next version will likely be published as an experimental version.

Jannik-Schilling commented 5 months ago

e11217a8446e0943c2e11df3d15d7c3cee9ae89e

Jannik-Schilling commented 5 months ago

version 0.33 is available now as 'experimental plugin' on the QGIS plugin server. Tested with Windows and Linux and had no crashes so far. If you're experiencing errors please open another issue.

@schorsch2 : I´d recommend to use the latest version, because the second tool (2_GenerateSwmmInpFile) had some troubles with date and time formats before.

MMR100 commented 5 months ago

It's working well. Thank you!