search

ffreemt / deepl-tr-async

deepl translate via pyppeteer
MIT License
11 stars 0 forks source link

"Session closed." when trying to make multiple translations via deepl.com #1

Closed freigus closed 4 years ago

freigus commented 4 years ago

Sorry, it might be a very noob question. When I'm trying to use res = loop.run_until_complete(deepl_tr_async(sent, to_lang='ru', from_lang='en')) in a loop it stops working after about 20 seconds with an error Protocol Error (Runtime.callFunctionOn): Session closed. Most likely the page has been closed. Code that I'm trying to use:

import asyncio
from deepl_tr_async import deepl_tr_async

def tl_this(sent):
    res = loop.run_until_complete(deepl_tr_async(sent, to_lang='ru', from_lang='en'))
    return res

while True:
    en_txt = input("Input EN text: ")
    loop = asyncio.get_event_loop()
    result = tl_this(en_txt)
    print("Russian result: {}".format(result))

Log for this code is:

[I 200411 00:26:43 deepl_tr_async:99]  dotenv.find_dotenv(Path().cwd() / ".env"): C:\Users\username\source\repos\Wolf_TL_txt\Wolf_TL_txt\.env
[W 200411 00:26:43 deepl_tr_async:108]  env.bool("HEADFUL") exc: Environment variable "HEADFUL" not set
[W 200411 00:26:43 deepl_tr_async:116]  env.bool("DEBUG") [None] exc: Environment variable "DEBUG" not set, DEBUG setting to False
[W 200411 00:26:43 deepl_tr_async:122]  env.str("PROXY") exc: Environment variable "PROXY" not set
[I 200411 00:26:43 deepl_tr_async:124]  HEADFUL: False
[I 200411 00:26:43 deepl_tr_async:125]  DEBUG: False
[I 200411 00:26:43 deepl_tr_async:126]  PROXY: 
Input EN text: C:\Users\username\source\repos\Wolf_TL_txt\Wolf_TL_txt\env_name\lib\site-packages\deepl_tr_async\deepl_tr_async.py:352: RuntimeWarning: coroutine 'sleep' was never awaited
  asyncio.sleep(0)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
C:\Users\username\source\repos\Wolf_TL_txt\Wolf_TL_txt\env_name\lib\site-packages\deepl_tr_async\deepl_tr_async.py:353: RuntimeWarning: coroutine 'sleep' was never awaited
  asyncio.sleep(0)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
C:\Users\username\source\repos\Wolf_TL_txt\Wolf_TL_txt\env_name\lib\site-packages\deepl_tr_async\deepl_tr_async.py:363: RuntimeWarning: coroutine 'sleep' was never awaited
  asyncio.sleep(0.2)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
Russian result: Почему мы здесь? Просто чтобы страдать?
Input EN text: [E 200411 00:27:09 deepl_tr_async:241]  browser.newPage exc: Protocol error Target.createTarget: Target closed., failed attempt: 1
C:\Users\username\source\repos\Wolf_TL_txt\Wolf_TL_txt\env_name\lib\site-packages\deepl_tr_async\deepl_tr_async.py:242: RuntimeWarning: coroutine 'sleep' was never awaited
  asyncio.sleep(0)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
[E 200411 00:27:09 deepl_tr_async:241]  browser.newPage exc: Connection is closed, failed attempt: 2
Russian result: None[E 200411 00:27:09 deepl_tr_async:241]  browser.newPage exc: Connection is closed, failed attempt: 3

Input EN text:

Am I doing something wrong? Should I somehow restart chromium page every time it closes? I'm using Windows 10 1909. Python 3.7

ffreemt commented 4 years ago

I tried your code in Win10 Py3.6 without any problem. I am not sure what the problem is. The lib should run in Py3.7 since it builds OK (https://github.com/ffreemt/deepl-tr-async/actions/runs/79603579).

Your code seems to be quite alright for an interactive session (maybe loop = asyncio.get_event_loop() should be moved out of the while loop). When doing a batch of sents, it will be much more efficient to make use of the async nature of deepl_tr_async.

freigus commented 4 years ago

I updated to 0.0.2 and it's still crushes after 20 seconds. I tried to run code mentioned above under different conditions:

And every time it crashed after 20 seconds. Now - more verbosity (a lot of logs from ubuntu) (I hope it can make something understandable):

2nd launch (env) psw123456@ubuntu:~/wolf_mtl_test$ /home/psw123456/wolf_mtl_test/env/bin/python /home/psw123456/wolf_mtl_test/deepl_tr_async_test.py [I 200416 09:32:10 deepl_tr_async:99] dotenv.find_dotenv(Path().cwd() / ".env"): /home/psw123456/wolf_mtl_test/.env [W 200416 09:32:10 deepl_tr_async:108] env.bool("HEADFUL") exc: Environment variable "HEADFUL" not set [W 200416 09:32:10 deepl_tr_async:116] env.bool("DEBUG") [None] exc: Environment variable "DEBUG" not set, DEBUG setting to False [W 200416 09:32:10 deepl_tr_async:122] env.str("PROXY") exc: Environment variable "PROXY" not set [I 200416 09:32:10 deepl_tr_async:124] HEADFUL: False [I 200416 09:32:10 deepl_tr_async:125] DEBUG: False [I 200416 09:32:10 deepl_tr_async:126] PROXY: [0416/093211.064919:ERROR:gpu_process_transport_factory.cc(1007)] Lost UI shared context. DevTools listening on ws://127.0.0.1:59073/devtools/browser/16b0c0fc-6477-4c8f-8ddf-916cf22f37ef Input EN text: hello [0416/093221.370327:INFO:CONSOLE(18)] "DeepL Translator", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (18) [0416/093221.573359:INFO:CONSOLE(1)] "Welcome to DeepL.com!", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (1) Russian result: Alternatives: приветствие приветствуйте приветствие на сайте Input EN text: [0416/093223.080791:WARNING:spdy_session.cc(3075)] Received HEADERS for invalid stream 19 [0416/093223.421568:WARNING:spdy_session.cc(3075)] Received HEADERS for invalid stream 17 ok [0416/093226.888242:INFO:CONSOLE(18)] "DeepL Translator", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (18) [0416/093227.019773:INFO:CONSOLE(1)] "Welcome to DeepL.com!", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (1) Russian result: Alternatives: хорошо ok Input EN text: [0416/093227.801829:WARNING:spdy_session.cc(3075)] Received HEADERS for invalid stream 21 hi [0416/093230.439212:INFO:CONSOLE(18)] "DeepL Translator", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (18) [0416/093230.560365:INFO:CONSOLE(1)] "Welcome to DeepL.com!", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (1) Traceback (most recent call last): File "/home/psw123456/wolf_mtl_test/deepl_tr_async_test.py", line 11, in result = tl_this(en_txt) File "/home/psw123456/wolf_mtl_test/deepl_tr_async_test.py", line 5, in tl_this res = loop.run_until_complete(deepl_tr_async(sent, to_lang='ru', from_lang='en')) File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/deepl_tr_async/deepl_tr_async.py", line 347, in deepl_tr_async content = await page.content() File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/page.py", line 780, in content return await frame.content() File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/frame_manager.py", line 379, in content '''.strip()) File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/frame_manager.py", line 295, in evaluate pageFunction, *args, force_expr=force_expr) File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 55, in evaluate pageFunction, *args, force_expr=force_expr) File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 109, in evaluateHandle _rewriteError(e) File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 239, in _rewriteError raise error File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/execution_context.py", line 106, in evaluateHandle 'userGesture': True, File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/connection.py", line 218, in send f'Protocol Error ({method}): Session closed. Most likely the ' pyppeteer.errors.NetworkError: Protocol Error (Runtime.callFunctionOn): Session closed. Most likely the page has been closed.
A bit different (and most common) error (env) psw123456@ubuntu:~/wolf_mtl_test$ /home/psw123456/wolf_mtl_test/env/bin/python /home/psw123456/wolf_mtl_test/deepl_tr_async_test.py [I 200416 09:34:25 deepl_tr_async:99] dotenv.find_dotenv(Path().cwd() / ".env"): /home/psw123456/wolf_mtl_test/.env [W 200416 09:34:25 deepl_tr_async:108] env.bool("HEADFUL") exc: Environment variable "HEADFUL" not set [W 200416 09:34:25 deepl_tr_async:116] env.bool("DEBUG") [None] exc: Environment variable "DEBUG" not set, DEBUG setting to False [W 200416 09:34:25 deepl_tr_async:122] env.str("PROXY") exc: Environment variable "PROXY" not set [I 200416 09:34:25 deepl_tr_async:124] HEADFUL: False [I 200416 09:34:25 deepl_tr_async:125] DEBUG: False [I 200416 09:34:25 deepl_tr_async:126] PROXY: [0416/093426.091110:ERROR:gpu_process_transport_factory.cc(1007)] Lost UI shared context. DevTools listening on ws://127.0.0.1:34487/devtools/browser/e7702813-f425-4099-9462-f54952c2543f Input EN text: hello [0416/093441.978274:INFO:CONSOLE(18)] "DeepL Translator", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (18) [0416/093442.155610:INFO:CONSOLE(1)] "Welcome to DeepL.com!", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (1) Russian result: Alternatives: приветствие приветствуйте приветствие на сайте Input EN text: [0416/093444.169035:WARNING:spdy_session.cc(3075)] Received HEADERS for invalid stream 17 [0416/093444.249327:WARNING:spdy_session.cc(3075)] Received HEADERS for invalid stream 5 hi [E 200416 09:34:46 deepl_tr_async:241] browser.newPage exc: Protocol error Target.createTarget: Target closed., failed attempt: 1 [E 200416 09:34:46 deepl_tr_async:241] browser.newPage exc: Connection is closed, failed attempt: 2 [E 200416 09:34:46 deepl_tr_async:241] browser.newPage exc: Connection is closed, failed attempt: 3 Russian result: None Input EN text: what [E 200416 09:34:57 deepl_tr_async:241] browser.newPage exc: Connection is closed, failed attempt: 1 [E 200416 09:34:57 deepl_tr_async:241] browser.newPage exc: Connection is closed, failed attempt: 2 [E 200416 09:34:57 deepl_tr_async:241] browser.newPage exc: Connection is closed, failed attempt: 3 Russian result: None Input EN text:

Next few tries had the same error. After this I went and closed 20-25 chrome processes in system monitor.

After closing chromes I had a new error. (env) psw123456@ubuntu:~/wolf_mtl_test$ /home/psw123456/wolf_mtl_test/env/bin/python /home/psw123456/wolf_mtl_test/deepl_tr_async_test.py [I 200416 09:46:26 deepl_tr_async:99] dotenv.find_dotenv(Path().cwd() / ".env"): /home/psw123456/wolf_mtl_test/.env [W 200416 09:46:26 deepl_tr_async:108] env.bool("HEADFUL") exc: Environment variable "HEADFUL" not set [W 200416 09:46:26 deepl_tr_async:116] env.bool("DEBUG") [None] exc: Environment variable "DEBUG" not set, DEBUG setting to False [W 200416 09:46:26 deepl_tr_async:122] env.str("PROXY") exc: Environment variable "PROXY" not set [I 200416 09:46:26 deepl_tr_async:124] HEADFUL: False [I 200416 09:46:26 deepl_tr_async:125] DEBUG: False [I 200416 09:46:26 deepl_tr_async:126] PROXY: [0416/094627.011313:ERROR:gpu_process_transport_factory.cc(1007)] Lost UI shared context. DevTools listening on ws://127.0.0.1:44621/devtools/browser/084f93c5-2db4-4868-a772-0d897a7b80c8 Input EN text: hello Traceback (most recent call last): File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/deepl_tr_async/deepl_tr_async.py", line 263, in deepl_tr_async await page.goto(url_, {"timeout": 90 * 1000}) File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/page.py", line 854, in goto result = await self._navigate(url, referrer) File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/page.py", line 869, in _navigate 'Page.navigate', {'url': url, 'referrer': referrer}) pyppeteer.errors.NetworkError: Protocol error Page.navigate: Target closed. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/psw123456/wolf_mtl_test/deepl_tr_async_test.py", line 11, in result = tl_this(en_txt) File "/home/psw123456/wolf_mtl_test/deepl_tr_async_test.py", line 5, in tl_this res = loop.run_until_complete(deepl_tr_async(sent, to_lang='ru', from_lang='en')) File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete return future.result() File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/deepl_tr_async/deepl_tr_async.py", line 268, in deepl_tr_async page = await browser.newPage() File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/browser.py", line 192, in newPage return await self._defaultContext.newPage() File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/browser.py", line 329, in newPage return await self._browser._createPageInContext(self._id) File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/browser.py", line 200, in _createPageInContext 'Target.createTarget', options)).get('targetId') File "/home/psw123456/wolf_mtl_test/env/lib/python3.6/site-packages/pyppeteer/connection.py", line 85, in send raise ConnectionError('Connection is closed') ConnectionError: Connection is closed (env) psw123456@ubuntu:~/wolf_mtl_test$ [0416/094655.933131:INFO:CONSOLE(18)] "DeepL Translator", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (18) [0416/094656.094711:INFO:CONSOLE(1)] "Welcome to DeepL.com!", source: https://www.deepl.com/js/translator_late.min.js?v=1518847586 (1)

After this I started to close chromes after every try and I always encountered browser.newPage exc: Connection is closed, failed attempt: 2

After switching to 4G connection nothing changed. (yes, i'm totally out of ideas)

ffreemt commented 4 years ago

A new version (0.0.3) is available, pip install deepl-tr-async -U to upgrade. The new version uses pyppeteer2. You may need to remove puppeteer (pip remove puppeteer) and install puppeteer2 (pip install puppeteer2).

If you really want to saty with deepl-tr-async 0.0.2, try this patch: https://github.com/miyakogi/pyppeteer/pull/160/files

freigus commented 4 years ago

Updated to 0.0.3 and puppeteer2 and it works! Thank you!