search

nsapa / fanfictionnet_ff_proxy

fanfictionnet_ff_proxy: an experimental "proxy" for fanfiction.net piloted by FanFicFare
CeCILL Free Software License Agreement v2.1
23 stars 3 forks source link

Breaks with Chromium 115, Python 3.11, fanficfare 3.19.0 and the proxy from master #18

Open mcepl opened 1 year ago

mcepl commented 1 year ago 
sedmikraska~/r/fanfictionnet_ff_proxy (master)$ python3.11 chrome_content.py --verbose --write-log  --chrome-path /usr/bin/chromium-browser --log-filename /tmp/fanfiction_proxy_log.txt
2023-08-16 21:40:57.598 CEST INFO main fanfictionnet_ff_proxy version 0.5.8 by Nicolas SAPA <nico@byme.at>
2023-08-16 21:40:57.599 CEST INFO main This Alpha software is licensed under CECILL-2.1
2023-08-16 21:40:57.600 CEST INFO main Running on Linux-5.14.21-150500.55.19-default-x86_64-with-glibc2.31
2023-08-16 21:40:57.686 CEST DEBUG main ChromeVersionFinder returned 115 for /usr/bin/chromium-browser
2023-08-16 21:40:57.687 CEST DEBUG ProxiedBrowser(init) Forcing binary path to /usr/bin/chromium-browser
2023-08-16 21:40:57.689 CEST DEBUG undetected_chromedriver.patcher getting release number from /LATEST_RELEASE_114
2023-08-16 21:40:57.765 CEST DEBUG undetected_chromedriver.patcher downloading from https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip
2023-08-16 21:41:11.303 CEST DEBUG undetected_chromedriver.patcher unzipping /tmp/tmpm52c3eoz
2023-08-16 21:41:11.466 CEST INFO undetected_chromedriver.patcher patching driver executable /home/matej/.local/share/undetected_chromedriver/undetected_chromedriver
2023-08-16 21:41:11.483 CEST DEBUG undetected_chromedriver.patcher found block:
b'{window.cdc_adoQpoasnfa76pfcZLmcfl_Array = window.Array;window.cdc_adoQpoasnfa76pfcZLmcfl_Object = window.Object;window.cdc_adoQpoasnfa76pfcZLmcfl_Promise = window.Promise;window.cdc_adoQpoasnfa76pfcZLmcfl_Proxy = window.Proxy;window.cdc_adoQpoasnfa76pfcZLmcfl_Symbol = window.Symbol;}'
replacing with:
b'{console.log("undetected chromedriver 1337!")}                                                                                                                                                                                                                                               '
2023-08-16 21:41:11.486 CEST DEBUG undetected_chromedriver.patcher patching took us 0.02 seconds
2023-08-16 21:41:11.494 CEST DEBUG selenium.webdriver.common.service Started executable: `/home/matej/.local/share/undetected_chromedriver/undetected_chromedriver` in a child process with pid: 22990
2023-08-16 21:41:11.996 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "pageLoadStrategy": "normal", "goog:chromeOptions": {"extensions": [], "binary": "/usr/bin/chromium-browser", "args": ["--no-first-run --no-service-autorun --password-store=basic", "--remote-debugging-host=127.0.0.1", "--remote-debugging-port=37477", "--user-data-dir=/tmp/tmp9dndvg1o", "--lang=cs-CZ", "--no-default-browser-check", "--no-first-run", "--no-sandbox", "--test-type", "--window-size=1920,1080", "--start-maximized", "--no-sandbox", "--log-level=0"], "debuggerAddress": "127.0.0.1:37477"}}}}
2023-08-16 21:41:11.996 CEST DEBUG urllib3.connectionpool Starting new HTTP connection (1): localhost:35903
2023-08-16 21:41:14.969 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session HTTP/1.1" 200 0
2023-08-16 21:41:14.969 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":{"capabilities":{"acceptInsecureCerts":false,"browserName":"chrome","browserVersion":"115.0.5790.170","chrome":{"chromedriverVersion":"114.0.5735.90 (386bc09e8f4f2e025eddae123f36f6263096ae49-refs/branch-heads/5735@{#1052})"},"goog:chromeOptions":{"debuggerAddress":"127.0.0.1:37477"},"pageLoadStrategy":"normal","platformName":"","proxy":{},"setWindowRect":false,"strictFileInteractability":false,"timeouts":{"implicit":0,"pageLoad":300000,"script":30000},"unhandledPromptBehavior":"dismiss and notify","webauthn:extension:credBlob":true,"webauthn:extension:largeBlob":true,"webauthn:extension:minPinLength":true,"webauthn:extension:prf":true,"webauthn:virtualAuthenticators":true},"sessionId":"8c8ed1f3a720b4b0cf6ee41895d58afb"}} | headers=HTTPHeaderDict({'Content-Length': '739', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:14.969 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:14.969 CEST INFO ProxiedBrowser(init) chromedriver version 114.0.5735.90 (386bc09e8f4f2e025eddae123f36f6263096ae49-refs/branch-heads/5735@{#1052}) running as pid 22990 driving Chrome version 115.0.5790.170 running as pid 22989 with undetected_chromedriver 3.5.0
2023-08-16 21:41:14.970 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "chrome://version"}
2023-08-16 21:41:15.247 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:15.247 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:15.248 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:15.248 CEST INFO ProxiedBrowser(init) Trying to load existing cookie...
2023-08-16 21:41:15.248 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:20.992 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:20.993 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:20.993 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:20.993 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": ".fanfiction.net", "expiry": 1721776020, "httpOnly": false, "name": "__gpi", "path": "/", "sameSite": "Lax", "secure": false, "value": "UID=00000c7c8a1569c3:T=1688080020:RT=1689611326:S=ALNI_MYE67Qi8lv-_tS0dHsGY39tRETfIQ"}}
2023-08-16 21:41:20.999 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:20.999 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:21.000 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:21.000 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:34.339 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:34.339 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:34.340 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:34.340 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": ".fanfiction.net", "expiry": 1721776020, "httpOnly": false, "name": "__gads", "path": "/", "sameSite": "Lax", "secure": false, "value": "ID=d275f01438cd380f-22da69f236e2003a:T=1688080020:RT=1689611326:S=ALNI_MZ-nzyOgy6_TL0DIiivWxJAMSoe0w"}}
2023-08-16 21:41:34.360 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:34.361 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:34.361 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:34.362 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:38.367 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:38.367 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:38.367 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:38.368 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": ".fanfiction.net", "expiry": 1689613032, "httpOnly": true, "name": "__cf_bm", "path": "/", "sameSite": "Strict", "secure": true, "value": "fwarHa7Giu_dvfcvIQX9J2oyboHFQl_wC0zoryJ99ho-1689611232-0-AePdLYYDOJP2r5BVFlGcuqBx5LLK/d1dUKMTdeRVuliVLeXuupE503A6bfZ/BPJtMas53mlrjYzTousy1REl3fxQHCbbXiagJ2WLju6utxWX"}}
2023-08-16 21:41:38.374 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:38.374 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:38.374 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:38.374 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:44.329 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:44.330 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:44.330 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:44.330 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": ".fanfiction.net", "expiry": 1721146351, "httpOnly": true, "name": "cf_clearance", "path": "/", "sameSite": "Strict", "secure": true, "value": "Vnd68anJN93GI.murJ5JvnmdYQevT1r69dNr5vl.ZcU-1689610349-0-160"}}
2023-08-16 21:41:44.336 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:44.336 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:44.336 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:44.336 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:50.826 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:50.826 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:50.827 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:50.827 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": "www.fanfiction.net", "expiry": 1689613949, "httpOnly": false, "name": "cf_chl_2", "path": "/", "sameSite": "Lax", "secure": false, "value": "30a437ac0c75762"}}
2023-08-16 21:41:50.833 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:50.833 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:50.834 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:50.834 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/url {"url": "http://www.fanfiction.net"}
2023-08-16 21:41:57.225 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/url HTTP/1.1" 200 0
2023-08-16 21:41:57.225 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:57.225 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:57.225 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://localhost:35903/session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie {"cookie": {"domain": "www.fanfiction.net", "expiry": 1720984497, "httpOnly": false, "name": "cookies", "path": "/", "sameSite": "Lax", "secure": true, "value": "yes"}}
2023-08-16 21:41:57.232 CEST DEBUG urllib3.connectionpool http://localhost:35903 "POST /session/8c8ed1f3a720b4b0cf6ee41895d58afb/cookie HTTP/1.1" 200 0
2023-08-16 21:41:57.232 CEST DEBUG selenium.webdriver.remote.remote_connection Remote response: status=200 | data={"value":null} | headers=HTTPHeaderDict({'Content-Length': '14', 'Content-Type': 'application/json; charset=utf-8', 'cache-control': 'no-cache'})
2023-08-16 21:41:57.232 CEST DEBUG selenium.webdriver.remote.remote_connection Finished Request
2023-08-16 21:41:57.232 CEST DEBUG ProxiedBrowser(init) Added 6 cookie(s)
2023-08-16 21:41:57.232 CEST INFO main Chrome is initialized & ready to works!
2023-08-16 21:41:57.232 CEST INFO main Listening on 127.0.0.1:8888

and on the other terminal:

sedmikraska~/K/f/n/Pride_and_Prejudice$ fanficfare -u A\ sunrise\ and\ a\ conversation-ffnet_12620565.epub
Updating A sunrise and a conversation-ffnet_12620565.epub, URL: https://www.fanfiction.net/s/12620565/1/
Traceback (most recent call last):
  File "/usr/bin/fanficfare", line 11, in <module>
    load_entry_point('FanFicFare==3.19.0', 'console_scripts', 'fanficfare')()
  File "/usr/lib/python3.6/site-packages/fanficfare/cli.py", line 310, in main
    passed_personalini)
  File "/usr/lib/python3.6/site-packages/fanficfare/cli.py", line 423, in do_download
    adapter.getStoryMetadataOnly()
  File "/usr/lib/python3.6/site-packages/fanficfare/adapters/base_adapter.py", line 297, in getStoryMetadataOnly
    self.doExtractChapterUrlsAndMetadata(get_cover=get_cover)
  File "/usr/lib/python3.6/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 111, in doExtractChapterUrlsAndMetadata
    raise e
  File "/usr/lib/python3.6/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 104, in doExtractChapterUrlsAndMetadata
    data = self._fetchUrl(url)
  File "/usr/lib/python3.6/site-packages/fanficfare/adapters/adapter_fanfictionnet.py", line 85, in _fetchUrl
    usecache=usecache)
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1338, in _fetchUrl
    referer=referer)[0]
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1327, in _fetchUrlOpened
    referer=referer)
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1228, in _fetchUrlOpened
    raise(excpt)
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1208, in _fetchUrlOpened
    referer=referer)
  File "/usr/lib/python3.6/site-packages/fanficfare/configurable.py", line 1167, in _fetchUrlRawOpened
    float(self.getConfig('connect_timeout',30.0)))
  File "/usr/lib64/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib64/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
sedmikraska~/K/f/n/Pride_and_Prejudice$

fanfiction_proxy_log.txt

nsapa commented 1 year ago

I going to be on holiday next week: I will works on the open issues (minimal version of undetectedchromedriver, making a release with binary for Windows, checking what is happing on MacOS, ...)

Look like FFF didn't use nsapa_proxy in your screendump.