Bench update-translations is broken

muniter commented 3 years ago

Description of the issue

I'm trying to update some translations after using the get-untranslated command, I edit the new file, when It's time to update the translation it crashes. I reported a month ago the error on get-untranslated https://github.com/frappe/frappe/issues/11488 now It works getting the untranslated strings, but the next part of the process is broken.

Seems to be related to the same issue of mismatch of types on opening and writing. It would be really nice for this to be fixed seems for over a year It has been impossible to do translations as explained in the documentation

It would be really important for a complete process to be tested before submitting a fix that leads to another road block, meaning exporting untranslated strings, translating, updating and seeing if they show up in the UI (Which I haven't been able to do)

Context information (for bug reports)

Output of bench version

erpnext 12.12.1
frappe 12.1.0

Steps to reproduce the issue

Go to `/frappe-bench/sites'
Run bench get-untranslated es ./my-trans
Run bench update-translations es ./my-trans.csv ./my-completed-trans.csv
Error will be shown

Observed result

Expected result

Stacktrace / full error message

WARN: bench is installed in editable mode!

This is not the recommended mode of installation for production. Instead, install the package from PyPI with: `pip install frappe-benc
h`

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 99, in <module>
    main()
  File "/workspace/development/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/workspace/development/frappe-bench/env/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 26, in _func
    ret = f(frappe._dict(ctx.obj), *args, **kwargs)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/commands/translate.py", line 68, in update_translations
    frappe.translate.update_translations(lang, untranslated_file, translated_file)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/translate.py", line 653, in update_translations
    write_translations_file(app, lang, full_dict)
  File "/workspace/development/frappe-bench/apps/frappe/frappe/translate.py", line 688, in write_translations_file
    app_messages, full_dict or get_full_dict(lang))
  File "/workspace/development/frappe-bench/apps/frappe/frappe/translate.py", line 581, in write_csv_file
    w.writerow([p.encode('utf-8') if p else '', m.encode('utf-8'), t.encode('utf-8')])
TypeError: a bytes-like object is required, not 'str'

Additional information

Running under docker, with provided devcontainers set up.

stephenBDT commented 3 years ago

I can confirm this! It is broken since a long time. Even more broken in 13-beta! see this issue I dug into the code and fixed the encoding stuff, but than it always OVERRIDES all my previous translations in myapp/translations/de.csv and does not realize that I already translated some of the strings.

This is a long broken

muniter commented 3 years ago

@stephenBDT I'm running a thread on the forum about the issues of translating, If you want to check it out. https://discuss.erpnext.com/t/translation-not-working-for-custom-apps/64522

icnbing commented 3 years ago

def write_csv_file(path, app_messages, lang_dict): """Write translation CSV file.

:param path: File path, usually `[app]/translations`.
:param app_messages: Translatable strings for this app.
:param lang_dict: Full translated dict.
"""
app_messages.sort(key = lambda x: x[1])
from csv import writer
# with open(path, 'wb') as msgfile: 
with open(path, 'w', encoding='utf-8') as msgfile:  
    w = writer(msgfile, lineterminator='\n')
    for p, m in app_messages:
        t = lang_dict.get(m, '')
        # strip whitespaces
        t = re.sub('{\s?([0-9]+)\s?}', "{\g<1>}", t)
        # w.writerow([p.encode('utf-8') if p else '', m.encode('utf-8'), t.encode('utf-8')])
        w.writerow([p if p else '',m, t])

frappe / frappe