attribute forms for layers with umlauts in name are not generated

[w0pr]

Version tag: v2023.10.28

Was it always like that and only comes up because of improved logging?

WARNING: Failed to write form for layer Trägerabschnitt Querschnitte: 'ascii' codec can't encode character '\xe4' in position 56: ordinal not in range(128)

qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049676386Z --- Logging error ---
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049873715Z Traceback (most recent call last):
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049891266Z   File "/srv/qwc_service/./config_generator/dnd_form_generator.py", line 31, in generate_form
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049895565Z     with open(outputfile, "wb") as fh:
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049898803Z UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 56: ordinal not in range(128)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049901996Z
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049904923Z During handling of the above exception, another exception occurred:
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049907983Z
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049911046Z Traceback (most recent call last):
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049913962Z   File "/usr/lib/python3.10/logging/__init__.py", line 1103, in emit
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049917374Z     stream.write(msg + self.terminator)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049920294Z UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 88: ordinal not in range(128)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.049935965Z Call stack:
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050051718Z   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2213, in __call__
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050240881Z     return self.wsgi_app(environ, start_response)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050248200Z   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2190, in wsgi_app
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050251639Z     response = self.full_dispatch_request()
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050259064Z   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1484, in full_dispatch_request
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050262392Z     rv = self.dispatch_request()
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050265556Z   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1469, in dispatch_request
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050268862Z     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050271799Z   File "/srv/qwc_service/./server.py", line 55, in generate_configs
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050274756Z     generator.write_configs()
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050277748Z   File "/srv/qwc_service/./config_generator/config_generator.py", line 354, in write_configs
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050281195Z     self.write_service_config(service_config['name'])
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050284257Z   File "/srv/qwc_service/./config_generator/config_generator.py", line 387, in write_service_config
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050287440Z     config = config_handler.config()
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050290539Z   File "/srv/qwc_service/./config_generator/map_viewer_config.py", line 115, in config
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050293643Z     resources['qwc2_themes'] = self.qwc2_themes(assets_dir)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050296551Z   File "/srv/qwc_service/./config_generator/map_viewer_config.py", line 240, in qwc2_themes
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050299531Z     theme_item = self.theme_item(item, themes_config, assets_dir, autogenExternalLayers, bgLayerCrs)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050302759Z   File "/srv/qwc_service/./config_generator/map_viewer_config.py", line 492, in theme_item
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050305874Z     item['editConfig'] = self.edit_config(name, cfg_item, assets_dir)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050308803Z   File "/srv/qwc_service/./config_generator/map_viewer_config.py", line 854, in edit_config
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050311769Z     forms = self.themes_reader.collect_ui_forms(map_name, assets_dir, layer_name)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050314825Z   File "/srv/qwc_service/./config_generator/theme_reader.py", line 140, in collect_ui_forms
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050317831Z     return self.theme_metadata[service_name]['project'].collect_ui_forms(assets_dir, edit_dataset, metadata)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050320805Z   File "/srv/qwc_service/./config_generator/qgs_reader.py", line 764, in collect_ui_forms
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050324009Z     uipath = gen.generate_form(maplayer, projectname, layername, self.root)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050328577Z   File "/srv/qwc_service/./config_generator/dnd_form_generator.py", line 35, in generate_form
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050339373Z     self.logger.warning("Failed to write form for layer %s: %s" % (layername, str(e)))
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050342787Z   File "/srv/qwc_service/./config_generator/config_generator.py", line 85, in warning
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050345909Z     self.logger.warning(msg)
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050348775Z Unable to print the message and arguments - possible formatting error.
qwc-docker-qwc-config-service-1           | 2023-10-31T15:40:34.050351949Z Use the traceback above to help find the error.

I also had to remove an umlaut in a setting "DxfExport" of config.json (current release of config.json also had this removed).

[manisandro]

Are your config files UTF-8 encoded?

[w0pr]

Are your config files UTF-8 encoded?

Strangely, not all. Just checked and re-edited with qwc_admin Config Editor. Seems to have happened 2 days ago in case of config.json.

qgis@srv-ada-gis01:~/qwc-docker/volumes/config-in/default$ file -i confi*
config.json:                     application/json; charset=us-ascii
config.json-20230519-015457.bak: application/json; charset=utf-8
config.json-20230521-085038.bak: application/json; charset=utf-8
config.json-20230521-112357.bak: application/json; charset=utf-8
config.json-20230522-044505.bak: application/json; charset=utf-8
config.json-20230524-032620.bak: application/json; charset=utf-8
config.json-20230622-142231.bak: application/json; charset=utf-8
config.json-20230719-050241.bak: application/json; charset=utf-8
config.json-20231031-135813.bak: application/json; charset=utf-8
**config.json-20231031-151430.bak: application/json; charset=utf-8**
config.json-20231101-144718.bak: application/json; charset=us-ascii

qgis@srv-ada-gis01:~/qwc-docker/volumes/config-in/default$ file -i tenantConfig*
tenantConfig.json:                     text/plain; charset=us-ascii
tenantConfig.json-20230519-014013.bak: text/plain; charset=us-ascii
tenantConfig.json-20230519-014815.bak: text/plain; charset=us-ascii
tenantConfig.json-20230519-015216.bak: text/plain; charset=us-ascii
tenantConfig.json-20230519-032406.bak: text/plain; charset=us-ascii
tenantConfig.json-20230520-070525.bak: text/plain; charset=us-ascii
tenantConfig.json-20230521-083246.bak: text/plain; charset=us-ascii
tenantConfig.json-20230522-044417.bak: text/plain; charset=us-ascii

qgis@srv-ada-gis01:~/qwc-docker/volumes/config/default$ file -i *
adminGuiConfig.json:    application/json; charset=utf-8
dataConfig.json:        application/json; charset=utf-8
dbAuthConfig.json:      application/json; charset=us-ascii
documentConfig.json:    application/json; charset=us-ascii
elevationConfig.json:   application/json; charset=us-ascii
featureInfoConfig.json: text/plain; charset=utf-8
index.html:             text/html; charset=us-ascii
legendConfig.json:      text/plain; charset=utf-8
mapinfoConfig.json:     application/json; charset=us-ascii
mapViewerConfig.json:   text/plain; charset=utf-8
ogcConfig.json:         text/plain; charset=utf-8
permalinkConfig.json:   application/json; charset=us-ascii
permissions.json:       application/json; charset=utf-8
printConfig.json:       application/json; charset=us-ascii
searchConfig.json:      application/json; charset=us-ascii

[manisandro]

I guess if you change them back to UTF-8 you don't get the errors anymore?

[w0pr]

I guess if you change them back to UTF-8 you don't get the errors anymore?

I will try and report back; does that also go for all under config(supposedly those are wtitten by config generator), not just those under config-in?

[manisandro]

Config-in is sufficient, a successful config-generator run will regenerate all files below config.

[w0pr]

Config-in is sufficient, a successful config-generator run will regenerate all files below config.

But error happens while parsing widget elements from .qgs project file, no? https://github.com/qwc-services/qwc-config-generator/blob/96644309206b4d46adc883833a14db1a5249f22b/config_generator/dnd_form_generator.py#L35

[...]
WARNING: Warning: field bemerkung has empty widget type
INFO: Wrote SWBSA_STROM_2021_Rita_QWC2_Maste.ui
INFO: Wrote SWBSA_STROM_2021_Rita_QWC2_ew_bauwerk.ui
INFO: Wrote SWBSA_STROM_2021_Rita_QWC2_Knoten_sch.ui
WARNING: Failed to write form for layer Hinweislinie für Querschnitte: 'ascii' codec can't encode character '\xfc' in position 68: ordinal not in range(128)
INFO: Wrote SWBSA_STROM_2021_Rita_QWC2_Snap Point.ui
INFO: Wrote SWBSA_STROM_2021_Rita_QWC2_ew_traeger.ui
WARNING: Failed to write form for layer Trägerabschnitt: 'ascii' codec can't encode character '\xe4' in position 56: ordinal not in range(128)
WARNING: Warning: field point_number has empty widget type
[...]

[manisandro]

As far as this error is concerned, it might indeed be necessary to force utf-8 output when writing the ui file. Does changing line 35 from

    with open(outputfile, "wb") as fh:

to

    with open(outputfile, "wb", encoding='utf-8') as fh:

help?

As far as the issue in config.json is concerned, does it help if the file is saved as utf-8?

I take your system encoding is not utf-8?

[manisandro]

This should be fixed with https://github.com/qwc-services/qwc-config-generator/commit/6120b252319c128f461e0b071881b33efb4ddf99

[w0pr]

This should be fixed with 6120b25

Used image: v2024.01.12

With the latest config generator image, there is still this UnicodeEncodeError when opening a an attribute form file for writing with an umlaut ('ä') in the name (from the layername); also another UnicodeEncodeError is triggered when logging the message with an umlaut:

qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054046162Z --- Logging error ---
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054207791Z Traceback (most recent call last):
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054230900Z   File "/srv/qwc_service/./config_generator/dnd_form_generator.py", line 32, in generate_form
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054235070Z     with open(outputfile, "wb") as fh:
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054238327Z UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 56: ordinal not in range(128)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054241448Z
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054244269Z During handling of the above exception, another exception occurred:
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054247316Z
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054250453Z Traceback (most recent call last):
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054253458Z   File "/usr/lib/python3.10/logging/__init__.py", line 1103, in emit
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054256868Z     stream.write(msg + self.terminator)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054259761Z UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 88: ordinal not in range(128)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054274109Z Call stack:
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054600818Z   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1478, in __call__
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054609916Z     return self.wsgi_app(environ, start_response)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054613286Z   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1455, in wsgi_app
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054616559Z     response = self.full_dispatch_request()
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054619574Z   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 867, in full_dispatch_request
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054622590Z     rv = self.dispatch_request()
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054625484Z   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 852, in dispatch_request
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054645081Z     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054648004Z   File "/srv/qwc_service/./server.py", line 55, in generate_configs
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054650939Z     generator.write_configs()
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054653851Z   File "/srv/qwc_service/./config_generator/config_generator.py", line 360, in write_configs
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054656891Z     self.write_service_config(service_config['name'])
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054659721Z   File "/srv/qwc_service/./config_generator/config_generator.py", line 393, in write_service_config
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054662723Z     config = config_handler.config()
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054665579Z   File "/srv/qwc_service/./config_generator/map_viewer_config.py", line 115, in config
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054668640Z     resources['qwc2_themes'] = self.qwc2_themes(assets_dir)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054671480Z   File "/srv/qwc_service/./config_generator/map_viewer_config.py", line 240, in qwc2_themes
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054674406Z     theme_item = self.theme_item(item, themes_config, assets_dir, autogenExternalLayers, bgLayerCrs)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054677410Z   File "/srv/qwc_service/./config_generator/map_viewer_config.py", line 492, in theme_item
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054680409Z     item['editConfig'] = self.edit_config(name, cfg_item, assets_dir)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054683242Z   File "/srv/qwc_service/./config_generator/map_viewer_config.py", line 861, in edit_config
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054686157Z     forms = self.themes_reader.collect_ui_forms(map_name, assets_dir, layer_name)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054689051Z   File "/srv/qwc_service/./config_generator/theme_reader.py", line 146, in collect_ui_forms
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054731085Z     return self.theme_metadata[service_name]['project'].collect_ui_forms(assets_dir, edit_dataset, metadata)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054809247Z   File "/srv/qwc_service/./config_generator/qgs_reader.py", line 776, in collect_ui_forms
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054820955Z     uipath = gen.generate_form(maplayer, projectname, layername, self.root)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054845985Z   File "/srv/qwc_service/./config_generator/dnd_form_generator.py", line 36, in generate_form
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054859066Z     self.logger.warning("Failed to write form for layer %s: %s" % (layername, str(e)))
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054862189Z   File "/srv/qwc_service/./config_generator/config_generator.py", line 86, in warning
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054865221Z     self.logger.warning(msg)
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054867995Z Unable to print the message and arguments - possible formatting error.
qwc-docker-qwc-config-service-1           | 2024-01-13T11:35:00.054871013Z Use the traceback above to help find the error.

[manisandro]

What do you get for locale.getencoding() in the container? I.e.

$ docker-compose exec qwc-config-service sh
# python3
>>> import locale
>>> print(locale.getencoding())

Do you get UTF-8? Does it help if you set PYTHONUTF8=1 in the environment for the qwc-config-service container?

[w0pr]

What do you get for locale.getencoding() in the container? I.e.

$ docker-compose exec qwc-config-service sh
# python3
>>> import locale
>>> print(locale.getencoding())

With docker image tag latest-2023-lts:

qgis@srv-ada-gis01:~/qwc-docker$ docker exec -it qwc-docker-qwc-config-service-1 bash
root@eeadbd53201b:/#  python3
Python 3.8.10 (default, Nov 26 2021, 20:14:08)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> print(locale.getencoding())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'locale' has no attribute 'getencoding'

Same error with docker image tag latest:

qgis@srv-ada-gis01:~/qwc-docker$ docker-compose exec qwc-config-service sh
# python3
Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> print(locale.getencoding())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'locale' has no attribute 'getencoding'

Probably because python3 version is still less than 3.11:

But "getlocale()" yields "('en_US', 'UTF-8')":

>>> print(locale.getlocale(category=locale.LC_CTYPE))
('en_US', 'UTF-8')

Do you get UTF-8? Does it help if you set PYTHONUTF8=1 in the environment for the qwc-config-service container?

See above; I get UTF-8, but even after setting PYTHONUTF8=1 via "export PYTHONUTF8=1", the exception(s) still remain.

[manisandro]

Looks as if you aren't getting the latest image, as you should get python 3.11.6:

$ docker-compose exec qwc-config-service sh
/ # python3
Python 3.11.6 (main, Oct  4 2023, 06:22:18) [GCC 12.2.1 20220924] on linux
Type "help", "copyright", "credits" or "license" for more information.

$ docker ps | grep config
f16bebca6930   sourcepole/qwc-config-generator:v2024.01.15-noqgis ...

[w0pr]

Hmpf, I just re-pulled :latest! Now retrying with v2024.01.15...

Nope. Seems that only the -noqgis image gets the newer python3.

[manisandro]

You will need to investigate this on your end, it looks like the image isn't getting pulled correctly.

[w0pr]

You will need to investigate this on your end, it looks like the image isn't getting pulled correctly.

Seems like v2024.01.15-noqgis has the newer python3 version, v2024.01.15 does not.

Anyways, both report UTF-8, but only -noqgis is working without above exceptions.

[manisandro]

Ah this may indeed be the case since one is alpine based the other ubuntu-lts.

[w0pr]

Ah this may indeed be the case since one is alpine based the other ubuntu-lts.

Are there plans to streamline this to have a common base? Not that there are subtle diffs that are leading to thos exceptions...

[manisandro]

Uhm not really easy. We use the alpine images wherever possible to reduce the size of the images, however alpine only provides a minimal package set, so for instance no qgis. For these cases we need to use base image with a richer package set, so i.e. ubuntu.

In any case I've tried also with the v2024.01.15 (without -noqgis) and it works as expected with a layername containing Umlaute and accents.

[w0pr]

Uhm not really easy. We use the alpine images wherever possible to reduce the size of the images, however alpine only provides a minimal package set, so for instance no qgis. For these cases we need to use base image with a richer package set, so i.e. ubuntu.

Ok, makes sense.

In any case I've tried also with the v2024.01.15 (without -noqgis) and it works as expected with a layername containing Umlaute and accents.

I re-tried it with v2024.01.15, even set PYTHONUTF8=1 and still same errors. I'm going insane, but I'll keep digging...

[manisandro]

I was finally able to reproduce this: this happened with an ubuntu based image with no overridden locale, in which case the qwc-uwsgi-base startup script failed to properly set a UTF-8 locale. This is fixed in [1], and in qwc-config-generator:v2024.01.16 which uses the fixed qwc-uwsgi-base:ubuntu.

[1] https://github.com/qwc-services/qwc-uwsgi-base/commit/65ac667808e6b4c10889ae483847e7bcafb81e6d

[w0pr]

I was finally able to reproduce this: this happened with an ubuntu based image with no overridden locale, in which case the qwc-uwsgi-base startup script failed to properly set a UTF-8 locale. This is fixed in [1], and in qwc-config-generator:v2024.01.16 which uses the fixed qwc-uwsgi-base:ubuntu.

[1] qwc-services/qwc-uwsgi-base@65ac667

Thanks a lot!