Machine translation dependencies aren't optional anymore

Bubu commented 2 years ago

Describe the issue

I've update https://weblate.bubu1.eu to version 4.13.1 and then started getting errors about missing boto3 package: Error importing module weblate.machinery.aws in WEBLATE_MACHINERY: "No module named 'boto3'"

I have never configured/activated the aws machine translation backend in the settings.py before and not via the new UI way either.

The settings.py still contains this:

# List of machine translations
MT_SERVICES = (
    #     "weblate.machinery.apertium.ApertiumAPYTranslation",
    #     "weblate.machinery.baidu.BaiduTranslation",
    #     "weblate.machinery.deepl.DeepLTranslation",
    #     "weblate.machinery.glosbe.GlosbeTranslation",
    #     "weblate.machinery.google.GoogleTranslation",
    #     "weblate.machinery.googlev3.GoogleV3Translation",
    #     "weblate.machinery.libretranslate.LibreTranslateTranslation",
    #     "weblate.machinery.microsoft.MicrosoftCognitiveTranslation",
    #     "weblate.machinery.microsoftterminology.MicrosoftTerminologyService",
    #     "weblate.machinery.modernmt.ModernMTTranslation",
    #     "weblate.machinery.mymemory.MyMemoryTranslation",
    #     "weblate.machinery.netease.NeteaseSightTranslation",
    #     "weblate.machinery.tmserver.AmagamaTranslation",
    #     "weblate.machinery.tmserver.TMServerTranslation",
    #     "weblate.machinery.yandex.YandexTranslation",
    #     "weblate.machinery.saptranslationhub.SAPTranslationHub",
    #     "weblate.machinery.youdao.YoudaoTranslation",
    "weblate.machinery.weblatetm.WeblateTranslation",
    "weblate.memory.machine.WeblateMemory",
)

Is there a place I missed deactivating these backends?

I already tried

[X] I've read and searched the documentation.
[X] I've searched for similar issues in this repository.

Steps to reproduce the behavior

Easiest to reproduce on /manage/performance/ but also got the error on /projects/searxng/searxng/de/ and /js/translate/weblate-translation-memory/201752/
Internal server error is produced and send out via email

I installed the missing packages (boto3, google-cloud-translate and zeep) which made everything work again.

Expected behavior

A warning is produced in weblate check or in the UI but the unavailable translation backends are simply deactivated.

Screenshots

No response

Exception traceback

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 36, in load_class
    mod = import_module(module)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
    <source code not available>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    <source code not available>
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    <source code not available>
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
    <source code not available>
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
    <source code not available>
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    <source code not available>
  File "/usr/lib/python3.10/site-packages/weblate/machinery/aws.py", line 20, in <module>
    import boto3

During handling of the above exception (No module named 'boto3'), another exception occurred:
  File "/usr/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.10/site-packages/django/views/decorators/http.py", line 43, in inner
    return func(request, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/weblate/machinery/views.py", line 351, in translate
    return handle_machinery(request, service, unit)
  File "/usr/lib/python3.10/site-packages/weblate/machinery/views.py", line 304, in handle_machinery
    if service not in MACHINERY:
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 104, in __contains__
    return self.data.__contains__(item)
  File "/usr/lib/python3.10/site-packages/django/utils/functional.py", line 49, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 77, in data
    return self.load_data()
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 69, in load_data
    obj = load_class(path, self.name)
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 38, in load_class
    raise ImproperlyConfigured(

Exception Type: ImproperlyConfigured at /js/translate/weblate/201752/
Exception Value: Error importing module weblate.machinery.aws in WEBLATE_MACHINERY: "No module named 'boto3'"

Later for missing google:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 36, in load_class
    mod = import_module(module)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
    <source code not available>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    <source code not available>
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    <source code not available>
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
    <source code not available>
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
    <source code not available>
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    <source code not available>
  File "/usr/lib/python3.10/site-packages/weblate/machinery/googlev3.py", line 24, in <module>
    from google.cloud.translate_v3 import TranslationServiceClient

During handling of the above exception (No module named 'google'), another exception occurred:
  File "/usr/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python3.10/site-packages/django/views/decorators/cache.py", line 62, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/lib/python3.10/site-packages/weblate/trans/views/basic.py", line 345, in show_translation
    "autoform": optional_form(
  File "/usr/lib/python3.10/site-packages/weblate/utils/views.py", line 59, in optional_form
    return form(**kwargs)
  File "/usr/lib/python3.10/site-packages/weblate/trans/forms.py", line 944, in __init__
    engines = sorted(
  File "/usr/lib/python3.10/site-packages/weblate/trans/forms.py", line 948, in <genexpr>
    if engine in MACHINERY
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 104, in __contains__
    return self.data.__contains__(item)
  File "/usr/lib/python3.10/site-packages/django/utils/functional.py", line 49, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 77, in data
    return self.load_data()
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 69, in load_data
    obj = load_class(path, self.name)
  File "/usr/lib/python3.10/site-packages/weblate/utils/classloader.py", line 38, in load_class
    raise ImproperlyConfigured(

Exception Type: ImproperlyConfigured at /projects/searxng/searxng/sr/
Exception Value: Error importing module weblate.machinery.googlev3 in WEBLATE_MACHINERY: "No module named 'google'"
Request information:
USER: Marcus


### How do you run Weblate?

PyPI module

### Weblate versions

 * Weblate: 4.13.1
 * Django: 4.0.5
 * siphashc: 2.1
 * translate-toolkit: 3.6.2
 * lxml: 4.9.0
 * Pillow: 9.1.1
 * bleach: 5.0.0
 * python-dateutil: 2.8.2
 * social-auth-core: 4.3.0
 * social-auth-app-django: 5.0.0
 * django-crispy-forms: 1.14.0
 * oauthlib: 3.2.0
 * django-compressor: 4.0
 * djangorestframework: 3.13.1
 * django-filter: 22.1
 * django-appconf: 1.0.5
 * user-agents: 2.2.0
 * filelock: 3.7.1
 * jellyfish: 0.9.0
 * openpyxl: 3.0.10
 * celery: 5.2.6
 * kombu: 5.2.3
 * translation-finder: 2.13
 * weblate-language-data: 2022.3
 * html2text: 2020.1.16
 * pycairo: 1.21.0
 * pygobject: 3.42.1
 * diff-match-patch: 20200713
 * requests: 2.27.1
 * django-redis: 5.2.0
 * hiredis: 2.0.0
 * sentry_sdk: 1.6.0
 * Cython: 0.29.30
 * misaka: 2.1.1
 * GitPython: 3.1.27
 * borgbackup: 1.2.1
 * pyparsing: 3.0.9
 * pyahocorasick: 1.4.2
 * python-redis-lock: 3.7.0
 * charset-normalizer: 2.1.0
 * Python: 3.10.5
 * Git: 2.37.0
 * psycopg2: 2.9.3
 * ruamel.yaml: 0.17.21
 * boto3: 1.24.17
 * zeep: 4.1.0
 * Redis server: 6.2.6
 * PostgreSQL server: 14.3
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis+socket:///var/run/redis/redis.sock?virtual_host=1, redis+socket:///var/run/redis/redis.sock?virtual_host=1, regular
 * Platform: Linux 5.17.5-arch1-1 (x86_64)

### Weblate deploy checks

```shell
System check identified some issues:

WARNINGS:
?: (weblate.W025.ass) Failure in loading handler for ass file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.fluent) Failure in loading handler for fluent file format: No module named 'fluent'
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.ini) Failure in loading handler for ini file format: Missing iniparse library.
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.islu) Failure in loading handler for islu file format: Missing iniparse library.
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.laravel) Failure in loading handler for laravel file format: No module named 'phply'
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.php) Failure in loading handler for php file format: No module named 'phply'
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.resourcedictionary) Failure in loading handler for resourcedictionary file format: No module named 'translate.storage.resourcedictionary'
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.srt) Failure in loading handler for srt file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.ssa) Failure in loading handler for ssa file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W025.sub) Failure in loading handler for sub file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#optional-deps
?: (weblate.W033.Gerrit) Failure in loading VCS module for Gerrit: git: 'review' is not a git command. See 'git --help'.
 (1)
    HINT: https://docs.weblate.org/en/weblate-4.13.1/vcs.html
?: (weblate.W033.Mercurial) Failure in loading VCS module for Mercurial: [Errno 2] No such file or directory: 'hg'
    HINT: https://docs.weblate.org/en/weblate-4.13.1/vcs.html
?: (weblate.W033.Subversion) Failure in loading VCS module for Subversion: Can't locate SVN/Core.pm in @INC (you may need to install the SVN::Core module) (@INC contains: /usr/share/perl5/vendor_perl /usr/lib/perl5/5.36/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/5.36/vendor_perl /usr/lib/perl5/5.36/core_perl /usr/share/perl5/core_perl) at /usr/share/perl5/vendor_perl/Git/SVN/Utils.pm line 6.
BEGIN failed--compilation aborted at /usr/share/perl5/vendor_perl/Git/SVN/Utils.pm line 6.
Compilation failed in require at /usr/share/perl5/vendor_perl/Git/SVN.pm line 32.
BEGIN failed--compilation aborted at /usr/share/perl5/vendor_perl/Git/SVN.pm line 32.
Compilation failed in require at /usr/lib/git-core/git-svn line 23.
BEGIN failed--compilation aborted at /usr/lib/git-core/git-svn line 23.
 (2)
    HINT: https://docs.weblate.org/en/weblate-4.13.1/vcs.html

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
    HINT: https://docs.weblate.org/en/weblate-4.13.1/admin/install.html#collecting-errors

System check identified 14 issues (1 silenced).

Additional context

No response

github-actions[bot] commented 2 years ago

Thank you for your report; the issue you have reported has just been fixed.

In case you see a problem with the fix, please comment on this issue.
In case you see a similar problem, please open a separate issue.
If you are happy with the outcome, don’t hesitate to support Weblate by making a donation.

tipa commented 1 year ago

I am seeing a very similar problem with my freshly installed Weblate 5.1 instance:

?: (weblate.W039.AWSTranslation) Error importing module 'weblate.machinery.aws' in WEBLATE_MACHINERY: No module named 'boto3'
        HINT: https://docs.weblate.org/en/weblate-5.1/admin/machine.html
?: (weblate.W039.GoogleV3Translation) Error importing module 'weblate.machinery.googlev3' in WEBLATE_MACHINERY: No module named 'google'
        HINT: https://docs.weblate.org/en/weblate-5.1/admin/machine.html
?: (weblate.W039.MicrosoftTerminologyService) Error importing module 'weblate.machinery.microsoftterminology' in WEBLATE_MACHINERY: No module named 'zeep'
        HINT: https://docs.weblate.org/en/weblate-5.1/admin/machine.html

After installing the three packages Iboto3, google-cloud-translate and zeep via pip resolved the warnings and the components became available in the "Automatic suggestions" tag.

I did not expect to see warnings after installing Weblate from scratch. I would have expected to either see the plugin to be deactivated (but visible) with a hint to the missing dependency or the dependency to be installed automatically. With he exception of zeep, these are not listed under the optional dependencies list

It can also be confusing that the warning states that the google module is missing, but one has to install the google-cloud-translate module

I can also open a new issue for this if that is preferred

nijel commented 1 year ago

I've improved the docs in 2f855daf6266dfed27be8e54c7b1cd6711552a85, but yes a separate issue would be better.

WeblateOrg / weblate