Issues with charset, too many requests, auto requests, etc.

[yash4ik]

1. Using Google: `PS>md-translate -s Google -S en -T ru index.html.md Using Russia server backend. [20:02:19] Processing: index.html.md [20:02:19] Got 3 lines to process [20:02:20] Processed 2 lines [20:02:20] 'charmap' codec can't encode characters in position 0-7: character maps to Traceback (most recent call last):

   File "c:\users\ya\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, │ │ └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\ya\AppDa... │ └ <code object at 0x000001AB22363660, file "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-... └ <function _run_code at 0x000001AB22196C10>

   File "c:\users\ya\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code exec(code, run_globals) │ └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\Ya\AppDa... └ <code object at 0x000001AB22363660, file "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-...

   File "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-translate.exe__main__.py", line 7, in

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\app.py", line 22, in run App().process() └ <class 'md_translate.app.App'>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\app.py", line 16, in process processing_file.translate() │ └ <function FileTranslator.translate at 0x000001AB24C5D1F0> └ <md_translate.file_translator.FileTranslator object at 0x000001AB24C01D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\file_translator.py", line 39, in translate self._write_translated_data_to_file() │ └ <function FileTranslator._write_translated_data_to_file at 0x000001AB24C5D160> └ <md_translate.file_translator.FileTranslator object at 0x000001AB24C01D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\file_translator.py", line 48, in _write_translated_data_to_file self.__translating_file.writelines(self.file_contents_with_translation) │ │ └ ['\n', 'title: Documentation\n', '\n', 'Название: Документация\n', '\n'] │ └ <md_translate.file_translator.FileTranslator object at 0x000001AB24C01D30> └ <md_translate.file_translator.FileTranslator object at 0x000001AB24C01D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] │ │ │ │ │ └ <EncodingMap object at 0x000001AB21B51350> │ │ │ │ └ 'strict' │ │ │ └ <encodings.cp1252.IncrementalEncoder object at 0x000001AB24C8D9A0> │ │ └ 'Название: Документация\r\n' │ └ └ <module 'codecs' from 'c:\users\ya\appdata\local\programs\python\python38\lib\codecs.py'>

UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-7: character maps to `

2. Using Yandex: `PS>md-translate -s Yandex -S en -T ru index.html.md Using Russia server backend. [20:04:32] Processing: index.html.md [20:04:32] Got 3 lines to process [20:04:36] 403 Client Error: Forbidden for url: https://translate.yandex.net/api/v1/tr.json/translate?id=7a400b5a.c5d5d906.410dc5ea.47875647d22747-0-0&lang=en-ru&srv=tr-text&reason=auto&format=text Traceback (most recent call last):

   File "c:\users\ya\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, │ │ └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\Ya\AppDa... │ └ <code object at 0x0000020E0E973660, file "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-... └ <function _run_code at 0x0000020E0E6A6C10>

   File "c:\users\ya\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code exec(code, run_globals) │ └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\Ya\AppDa... └ <code object at 0x0000020E0E973660, file "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-...

   File "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-translate.exe__main__.py", line 7, in

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\app.py", line 22, in run App().process() └ <class 'md_translate.app.App'>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\app.py", line 16, in process processing_file.translate() │ └ <function FileTranslator.translate at 0x0000020E1117D040> └ <md_translate.file_translator.FileTranslator object at 0x0000020E11121D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\file_translator.py", line 37, in translate self.file_contents_with_translation.append(line.fixed) │ │ │ │ └ <property object at 0x0000020E0EA93770> │ │ │ └ Line: "title: Documentation │ │ │ " │ │ └ <method 'append' of 'list' objects> │ └ ['\n', 'title: Documentation\n', '\n'] └ <md_translate.file_translator.FileTranslator object at 0x0000020E11121D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\line_processor.py", line 43, in fixed if self._line.endswith('\n') and not self.translated.endswith('\n'): │ │ │ │ └ <property object at 0x0000020E0EA40220> │ │ │ └ Line: "title: Documentation │ │ │ " │ │ └ <method 'endswith' of 'str' objects> │ └ 'title: Documentation\n' └ Line: "title: Documentation "

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\line_processor.py", line 38, in translated self._translate() │ └ <function Line._translate at 0x0000020E11158CA0> └ Line: "title: Documentation "

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\line_processor.py", line 63, in _translate self._translated_line = self._translator( │ │ │ └ <bound method Yandex.yandex_api of <translators.apis.Yandex object at 0x0000020E111451F0>> │ │ └ Line: "title: Documentation │ │ " │ └ '' └ Line: "title: Documentation "

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\translators\apis.py", line 1002, in yandex_api r.raise_for_status() │ └ <function Response.raise_for_status at 0x0000020E10DC23A0> └ <Response [403]>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\requests\models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) │ │ └ <Response [403]> │ └ '403 Client Error: Forbidden for url: https://translate.yandex.net/api/v1/tr.json/translate?id=7a400b5a.c5d5d906.410dc5ea.478... └ <class 'requests.exceptions.HTTPError'>

requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://translate.yandex.net/api/v1/tr.json/translate?id=7a400b5a.c5d5d906.410dc5ea.47875647d22747-0-0&lang=en-ru&srv=tr-text&reason=auto&format=text`

3. Using Bing: `PS>md-translate -s Bing -S en -T ru index.html.md Using Russia server backend. [20:05:04] Processing: index.html.md [20:05:04] Got 3 lines to process [20:05:08] Processed 2 lines [20:05:08] 'charmap' codec can't encode characters in position 0-7: character maps to Traceback (most recent call last):

   File "c:\users\ya\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, │ │ └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\Ya\AppDa... │ └ <code object at 0x0000019273293660, file "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-... └ <function _run_code at 0x0000019272FC6C10>

   File "c:\users\ya\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code exec(code, run_globals) │ └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\Ya\AppDa... └ <code object at 0x0000019273293660, file "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-...

   File "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-translate.exe__main__.py", line 7, in

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\app.py", line 22, in run App().process() └ <class 'md_translate.app.App'>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\app.py", line 16, in process processing_file.translate() │ └ <function FileTranslator.translate at 0x0000019275ABD1F0> └ <md_translate.file_translator.FileTranslator object at 0x0000019275A61D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\file_translator.py", line 39, in translate self._write_translated_data_to_file() │ └ <function FileTranslator._write_translated_data_to_file at 0x0000019275ABD160> └ <md_translate.file_translator.FileTranslator object at 0x0000019275A61D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\file_translator.py", line 48, in _write_translated_data_to_file self.__translating_file.writelines(self.file_contents_with_translation) │ │ └ ['\n', 'title: Documentation\n', '\n', 'название: Документация\n', '\n'] │ └ <md_translate.file_translator.FileTranslator object at 0x0000019275A61D30> └ <md_translate.file_translator.FileTranslator object at 0x0000019275A61D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] │ │ │ │ │ └ <EncodingMap object at 0x00000192727DC2D0> │ │ │ │ └ 'strict' │ │ │ └ <encodings.cp1252.IncrementalEncoder object at 0x0000019275AED9A0> │ │ └ 'название: Документация\r\n' │ └ └ <module 'codecs' from 'c:\users\ya\appdata\local\programs\python\python38\lib\codecs.py'>

UnicodeEncodeError: 'charmap' codec can't encode characters in position 0-7: character maps to `

4. Using Deepl: `PS>md-translate -s Deepl -S en -T ru index.html.md Using Russia server backend. [20:07:08] Processing: index.html.md [20:07:08] Got 3 lines to process [20:07:13] 429 Client Error: Too many Requests for url: https://www2.deepl.com/jsonrpc Traceback (most recent call last):

   File "c:\users\ya\appdata\local\programs\python\python38\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, │ │ └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\Ya\AppDa... │ └ <code object at 0x000001707BD54660, file "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-... └ <function _run_code at 0x000001707BB86C10>

   File "c:\users\ya\appdata\local\programs\python\python38\lib\runpy.py", line 87, in _run_code exec(code, run_globals) │ └ {'name': 'main', 'doc': None, 'package': '', 'loader': <zipimporter object "C:\Users\Ya\AppDa... └ <code object at 0x000001707BD54660, file "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-...

   File "C:\Users\Ya\AppData\Local\Programs\Python\Python38\Scripts\md-translate.exe__main__.py", line 7, in

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\app.py", line 22, in run App().process() └ <class 'md_translate.app.App'>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\app.py", line 16, in process processing_file.translate() │ └ <function FileTranslator.translate at 0x000001707E66D1F0> └ <md_translate.file_translator.FileTranslator object at 0x000001707E611D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\file_translator.py", line 37, in translate self.file_contents_with_translation.append(line.fixed) │ │ │ │ └ <property object at 0x000001707BF73770> │ │ │ └ Line: "title: Documentation │ │ │ " │ │ └ <method 'append' of 'list' objects> │ └ ['\n', 'title: Documentation\n', '\n'] └ <md_translate.file_translator.FileTranslator object at 0x000001707E611D30>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\line_processor.py", line 43, in fixed if self._line.endswith('\n') and not self.translated.endswith('\n'): │ │ │ │ └ <property object at 0x000001707BF20220> │ │ │ └ Line: "title: Documentation │ │ │ " │ │ └ <method 'endswith' of 'str' objects> │ └ 'title: Documentation\n' └ Line: "title: Documentation "

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\line_processor.py", line 38, in translated self._translate() │ └ <function Line._translate at 0x000001707E648CA0> └ Line: "title: Documentation "

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\md_translate\line_processor.py", line 63, in _translate self._translated_line = self._translator( │ │ │ └ <bound method Deepl.deepl_api of <translators.apis.Deepl object at 0x000001707E6355B0>> │ │ └ Line: "title: Documentation │ │ " │ └ '' └ Line: "title: Documentation "

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\translators\apis.py", line 933, in deepl_api r.raise_for_status() │ └ <function Response.raise_for_status at 0x000001707E2B23A0> └ <Response [429]>

File "c:\users\ya\appdata\local\programs\python\python38\lib\site-packages\requests\models.py", line 943, in raise_for_status raise HTTPError(http_error_msg, response=self) │ │ └ <Response [429]> │ └ '429 Client Error: Too many Requests for url: https://www2.deepl.com/jsonrpc' └ <class 'requests.exceptions.HTTPError'>

requests.exceptions.HTTPError: 429 Client Error: Too many Requests for url: https://www2.deepl.com/jsonrpc`

[ilyachch]

@yash4ik Thank you opening this issue. Could you provider the source file, please? (or at least part of it) Also, what is your OS and Python verison?

[yash4ik]

@ilyachch, thanks for the quick answer! Python 3.8.6 (same problem caught with the latest Python), Windows 10 (20H2) (latest). You mean that index.html.md file, which I'm trying to translate? It has just these 4 lines: ` title: Documentation

`

[ilyachch]

Could you attach the file here, please? I have to chech it's encoding to solve the issue

[yash4ik]

Sure. index.html.md

[ilyachch]

@yash4ik I've checked you file. And it was processed successfully Output was:

md-translate -s Bing -S en -T ru /home/.../Downloads/index.html.md
Using ... server backend.
[17:18:39] Processing: index.html.md
[17:18:39] Got 3 lines to process
[17:18:41] UseTimeSeconds(fn: bing_api): 0.9
[17:18:41] Processed 2 lines
[17:18:41] Processed: /home/.../Downloads/index.html.md

Could you try WSL or some Linux system to try? I've got Ubuntu and all works fine. I'm going to add windows system in CI to check, how it work in Windows

BTW, I've got 2 options causing this error: 1) something goes wrong with encoding in windows 2) something is wrong in symbols in file (I mean there could be some symbols from another language, that look like common symbols from english), but i've checked hex codes of document and everything is fine.

So I think it's something wrong with decoding in windows, but I don't know where to start as I don't have any Windows system based

[yash4ik]

Thanks for this idea, it works with WSL2 and Google/Bing. But there're still some authorization troubles with Yandex and Deepl. Yandex error: 403 Client Error: Forbidden for url: https://translate.yandex.net/api/v1/tr.json/translate?id=c6c2fa0c.f1482a06.7851757f.47875647d22747-0-0&lang=en-ru&srv=tr-text&reason=auto&format=text <...> requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://translate.yandex.net/api/v1/tr.json/translate?id=c6c2fa0c.f1482a06.7851757f.47875647d22747-0-0&lang=en-ru&srv=tr-text&reason=auto&format=text

Deepl error: 429 Client Error: Too many Requests for url: https://www2.deepl.com/jsonrpc <...> requests.exceptions.HTTPError: 429 Client Error: Too many Requests for url: https://www2.deepl.com/jsonrpc

Still, thanks a lot!

[ilyachch]

@yash4ik I'm using side library fot communication with translation services, so, I'm not responsible for this issue. But, as I've tried all this services, bing works great and I'm using it for all my work.

I'm thinking of making own clients for translation services, but I'm busy with work and have not much time.