[Feature] Improve schema error reporting and language translation

[fmigneault]

Context

Since colander is used to validate input schemas of requests body and generated responses, it is possible to better handle the returned errors as translated text. Given that Accept-Language header is now also supported (#74, #82), we should report invalid inputs messages accordingly in the REST API.

More precisely, when a colander.Invalid (exception) schema is generated, the responses contain messages as TranslationString (pyramid.i18n), which can be used to return locale-aware descriptions.

See also: https://cornice.readthedocs.io/en/latest/i18n.html

TODO

• [x] remove all try/except everywhere around the wps-restapi portion of the code for colander.Invalid and move them to the tween as a single HTTPBadRequest catch. Using a global catch of colander.Invalid, we can run the translation string operations with the full error stack since they are accumulated from the first raised to the top.
  • Resolution: won't do. Explicit re-raise of colander.Invalid into a specific OWSException/HTTPException now provides more details, such as the context where the error happened and specific OGC error codes to use for it. Easier to edit inplace.
• [ ] Use asdict or similar variants for error reporting. For the moment, items like oneOf aggregates error strings which makes it extremely hard to read. All their errors are dumped in the response description which is "cleaned", but all the special characters and string concatenation generates a lot of garbage. The description should be replaced by a generic (summary) string, and another field would have the full detail as dict/list.
• [ ] add weaver domain with corresponding message strings for the app. colander messages are pre-packaged with it. Not only errors should be translated, but as much items as possible such as successful deployment or job status.

reference code (minimal - not complete)

def translator(key="<msgid>", domain="colander"):
    return <translation>

try: 
    # ...
except: 
    invalid = colander.Invalid()
    invalid.add(<more-error>)
    invalid.add(<more-error>)
    raise invalid 

# in tween
# returns a dict of all steps that caused an error with mapping
# also .messages() available for a list of strings
invalid.asdict(translate=translator)
# ... 

[fmigneault]

Need this (or similar) because pyramid checks _LOCALE_ in URL parameter, cookies and settings. Need to add Accept-Language manually instead (don't want the specific to pyramid _LOCALE_). Also allow for <weaver-url>?lang=<locale> variant.

#: Mapping of language codes send by browsers to supported dialects.
BROWSER_LANGUAGES = {
    'en': 'en_CA',
    'en-ca': 'en_CA',
    'en-us': 'en_CA',
    'en-gb': 'en_CA',
    'fr': 'fr_CA',
    'fr-ca': 'fr_CA',
    'fr-fr': 'fr_CA'
}

def locale_negotiator(request):
    """Locale negotiator for pyramid views.

This version differs from Pyramid's :py:func:`default_locale_negotiator
<pyramid.i18n.default_locale_negotiator>` in that it has a fallback to the
preferred languages from browser.
"""
    locale = default_locale_negotiator(request)  # default 'en'
    if locale is None and request.accept_language:
        locale = request.accept_language.best_match(BROWSER_LANGUAGES.keys()).lower()
        # default to en_GB as it does not have silly units etc...
        locale = BROWSER_LANGUAGES.get(locale, 'en_GB')
        request._LOCALE_ = locale

    return locale

[fmigneault]

from pyramid.i18n import TranslationString, make_localizer
# note: colander (and many more) use  "_(<str>)" factory for TranslationString with predefined domain

msgid = 'Required' 
colander_locale = <path-to-colander-pkg>/locale
ts = TranslationString(msgid , domain='colander')
L = make_localizer('fr', [colander_locale, <others> ])
L.translate(ts, domain='colander')    # will return 'Requis' (corresponding msgstr)

from pyramid.i18n import TranslationStringFactory
_ = TranslationStringFactory('weaver')  # our .mo files domain
ts = _('deploy', default='Invalid ${thing}', mapping={'thing': <thing>})   # default optional

See pyramid.config.Configurator.add_translation_dirs allowing 'some.package:locale' spec instead of literal path (which is deep somewhere in the installed env)

see pyramid.config.Configurator.set_locale_negotiator with https://github.com/crim-ca/weaver/issues/112#issuecomment-622662309 to resolve appropriate header to local/translation selection

see extracting-messages-from-code-and-templates, initializing-a-message-catalog-file and compiling-a-message-catalog-file to respectively make language file template (.pot), initialize and compile a specific language definition (.po) to machine locale (.mo)

Application will need to execute compilation in setup.py using msgfmt (subprocess?) so that files are updated each time the package is installed (machine, docker, etc.).