search

termux-user-repository / tur

A place for all types of Termux packages.
Other
421 stars 81 forks source link

[Package]: Flaresolverr #631

Closed Vojtak42 closed 11 months ago

Vojtak42 commented 11 months ago

Name

Flaresolverr

Home page URL

No response

Source code URL

https://github.com/FlareSolverr/FlareSolverr

Packaging policy acknowledgement

Additional information

Officially arm64 is only supported on Docker (also due to chromium dependency) but there is (untested) guide from author for installing it on Arm64: https://github.com/FlareSolverr/FlareSolverr/issues/88. Thanks in advance I would really appreciate it.

licy183 commented 11 months ago

It is hard to package this. Maybe it will be able to package using venv as pyinstaller doesn't work on Termux.

romanovj commented 11 months ago

you can proot into docker container, here is my output (aarch64, termux from f-droid, no root)

2023-10-12 16:37:17 INFO     FlareSolverr 3.3.6
2023-10-12 16:37:17 INFO     Testing web browser installation...
2023-10-12 16:37:17 INFO     Platform: Linux-4.19.136-perf+-aarch64-with-glibc2.31
2023-10-12 16:37:17 INFO     Chrome / Chromium path: /usr/bin/chromium
2023-10-12 16:37:17 INFO     Chrome / Chromium major version: 116
2023-10-12 16:37:17 INFO     Launching web browser...
2023-10-12 16:37:20 INFO     FlareSolverr User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
2023-10-12 16:37:20 INFO     Test successful!
2023-10-12 16:37:20 INFO     Serving on http://0.0.0.0:8191

2023-10-12 16:48:45 INFO     Incoming request => POST /v1 body: {'cmd': 'request.get', 'url': 'https://iptorrents.com/', 'maxTimeout': 60000}
2023-10-12 16:48:49 INFO     Challenge detected. Title found: Just a moment...
2023-10-12 16:48:59 INFO     Challenge solved!
2023-10-12 16:48:59 INFO     Response in 14.253 s
2023-10-12 16:48:59 INFO     127.0.0.1 POST http://localhost:8191/v1 200 OK

use proot-distro to proot into debian/ubuntu, update, install skopeo(for downloading) and umoci (for unpacking)

cd /data/data/com.termux/files/home
skopeo copy docker://ghcr.io/flaresolverr/flaresolverr:v3.3.6 oci:flaresolverr:v3.3.6
umoci unpack --image flaresolverr:v3.3.6 rootfs

proot into this rootfs from termux and execute:

su flaresolverr
python -u /app/flaresolverr.py
Vojtak42 commented 11 months ago

you can proot into docker container, here is my output (aarch64, termux from f-droid, no root)

2023-10-12 16:37:17 INFO     FlareSolverr 3.3.6
2023-10-12 16:37:17 INFO     Testing web browser installation...
2023-10-12 16:37:17 INFO     Platform: Linux-4.19.136-perf+-aarch64-with-glibc2.31
2023-10-12 16:37:17 INFO     Chrome / Chromium path: /usr/bin/chromium
2023-10-12 16:37:17 INFO     Chrome / Chromium major version: 116
2023-10-12 16:37:17 INFO     Launching web browser...
2023-10-12 16:37:20 INFO     FlareSolverr User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36
2023-10-12 16:37:20 INFO     Test successful!
2023-10-12 16:37:20 INFO     Serving on http://0.0.0.0:8191

2023-10-12 16:48:45 INFO     Incoming request => POST /v1 body: {'cmd': 'request.get', 'url': 'https://iptorrents.com/', 'maxTimeout': 60000}
2023-10-12 16:48:49 INFO     Challenge detected. Title found: Just a moment...
2023-10-12 16:48:59 INFO     Challenge solved!
2023-10-12 16:48:59 INFO     Response in 14.253 s
2023-10-12 16:48:59 INFO     127.0.0.1 POST http://localhost:8191/v1 200 OK

use proot-distro to proot into debian/ubuntu, update, install skopeo(for downloading) and umoci (for unpacking)

cd /data/data/com.termux/files/home
skopeo copy docker://ghcr.io/flaresolverr/flaresolverr:v3.3.6 oci:flaresolverr:v3.3.6
umoci unpack --image flaresolverr:v3.3.6 rootfs

proot into this rootfs from termux and execute:

su flaresolverr
python -u /app/flaresolverr.py

Thanks, but su flaresolverr doesn't work for me (root@localhost:/data/data/com.termux/files/home/rootfs/rootfs/app# su flaresolverr su: user flaresolverr does not exist or the user entry does not contain all the required fields). And when i cd to the app folder and run python3 -u flaresolverr.py, it can't import certifi and when i try to install it, i get this:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/_vendor/platformdirs/android.py", line 85, in _android_folder
    from jnius import autoclass
ModuleNotFoundError: No module named 'jnius'
                                                                  During handling of the above exception, another exception occurred:

Traceback (most recent call last):                                  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3/dist-packages/pip/__main__.py", line 29, in <module>
    from pip._internal.cli.main import main as _main
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/main.py", line 9, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/autocompletion.py", line 10, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/usr/lib/python3/dist-packages/pip/_internal/cli/cmdoptions.py", line 23, in <module>
    from pip._internal.cli.parser import ConfigOptionParser         File "/usr/lib/python3/dist-packages/pip/_internal/cli/parser.py", line 12, in <module>                                               from pip._internal.configuration import Configuration, ConfigurationError                                                         File "/usr/lib/python3/dist-packages/pip/_internal/configuration.py", line 26, in <module>                                            from pip._internal.utils.logging import getLogger
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/logging.py", line 27, in <module>
    from pip._internal.utils.misc import ensure_dir
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/misc.py", line 39, in <module>                                               from pip._internal.locations import get_major_minor_version
  File "/usr/lib/python3/dist-packages/pip/_internal/locations/__init__.py", line 14, in <module>
    from . import _distutils, _sysconfig
  File "/usr/lib/python3/dist-packages/pip/_internal/locations/_distutils.py", line 19, in <module>
    from .base import get_major_minor_version
  File "/usr/lib/python3/dist-packages/pip/_internal/locations/base.py", line 12, in <module>
    USER_CACHE_DIR = appdirs.user_cache_dir("pip")
  File "/usr/lib/python3/dist-packages/pip/_internal/utils/appdirs.py", line 17, in user_cache_dir                                      return _appdirs.user_cache_dir(appname, appauthor=False)
  File "/usr/lib/python3/dist-packages/pip/_vendor/platformdirs/__init__.py", line 114, in user_cache_dir
    return PlatformDirs(appname=appname, appauthor=appauthor, version=version, opinion=opinion).user_cache_dir
  File "/usr/lib/python3/dist-packages/pip/_vendor/platformdirs/android.py", line 43, in user_cache_dir
    return self._append_app_name_and_version(_android_folder(), "cache")
  File "/usr/lib/python3/dist-packages/pip/_vendor/platformdirs/android.py", line 97, in _android_folder                                raise OSError("Cannot find path to android app folder")
OSError: Cannot find path to android app folder

EDIT: I successfully installe certifi from source

romanovj commented 11 months ago

@Vojtak42 after downloading and unpacking container you should exit from debian/ubuntu and proot from native termux into new rootfs by using script

#!/data/data/com.termux/files/usr/bin/bash
cd $(dirname $0)
## unset LD_PRELOAD in case termux-exec is installed
unset LD_PRELOAD
command="proot"
## uncomment following line if you are having FATAL: kernel too old message.
#command+=" -k 4.14.81"
command+=" --link2symlink"
command+=" -0"
command+=" -r rootfs"
command+=" -b /dev"
command+=" -b /proc"
command+=" -b /sys"
command+=" -b rootfs/tmp:/dev/shm"
#command+=" -b /data/data/com.termux"
#command+=" -b /:/host-rootfs"
#command+=" -b /sdcard"
#command+=" -b /storage"
command+=" -b /mnt"
command+=" -w /root"
command+=" /usr/bin/env -i"
command+=" HOME=/root"
command+=" PATH=/usr/local/sbin:/usr/local/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/games:/usr/local/games"
command+=" TERM=$TERM"
command+=" LANG=C.UTF-8"
command+=" /bin/bash --login"
com="$@"
if [ -z "$1" ];then
    exec $command
else
    $command -c "$com"
fi

place this script in ~/rootfs folder inside which you have another rootfs folder

romanovj commented 11 months ago

you shouldn't install anything

post output

/app/chromedriver

Vojtak42 commented 11 months ago

/app/chromedriver: error while loading shared libraries: libatomic.so.1: cannot open shared object file: No such file or directory

Vojtak42 commented 11 months ago

The python -u flaresolverr.py outputs this:

2023-10-12 19:39:28 INFO     FlareSolverr 3.3.6
2023-10-12 19:39:28 INFO     Testing web browser installation...
2023-10-12 19:39:28 INFO     Platform: Linux-4.14.180-perf-g4b73fd3-aarch64-with-glibc2.31
2023-10-12 19:39:28 INFO     Chrome / Chromium path: /usr/bin/chromium
2023-10-12 19:39:28 INFO     Chrome / Chromium major version: 117
2023-10-12 19:39:28 INFO     Launching web browser...
Traceback (most recent call last):
  File "/app/utils.py", line 296, in get_user_agent
    driver = get_webdriver()
             ^^^^^^^^^^^^^^^
  File "/app/utils.py", line 177, in get_webdriver
    driver = uc.Chrome(options=options, browser_executable_path=browser_executable_path,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/undetected_chromedriver/__init__.py", line 474, in __init__
    super(Chrome, self).__init__(
  File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(
  File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/chromium/webdriver.py", line 53, in __init__
    self.service.start()
  File "/usr/local/lib/python3.11/site-packages/selenium/webdriver/common/service.py", line 116, in start
    raise WebDriverException(f"Can not connect to the Service {self._path}")
selenium.common.exceptions.WebDriverException: Message: Can not connect to the Service /app/chromedriver

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/flaresolverr.py", line 105, in <module>
    flaresolverr_service.test_browser_installation()
  File "/app/flaresolverr_service.py", line 72, in test_browser_installation
    user_agent = utils.get_user_agent()
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/utils.py", line 300, in get_user_agent
    raise Exception("Error getting browser User-Agent. " + str(e))
Exception: Error getting browser User-Agent. Message: Can not connect to the Service /app/chromedriver
romanovj commented 11 months ago

wtf you did with rootfs? 

ldd /app/chromedriver
        linux-vdso.so.1 (0x000000798b3ac000)
        libatomic.so.1 => /usr/lib/aarch64-linux-gnu/libatomic.so.1
.....
Vojtak42 commented 11 months ago

Before i installed chromium using apt, the only different thing was that there was: Chrome / Chromium major version: 116

Vojtak42 commented 11 months ago

I will probably tomorrow start from beginning and i will see.

romanovj commented 11 months ago

from root user apt reinstall libatomic1:arm64

or just delete and unpack again, do not update/upgrade anything

Vojtak42 commented 11 months ago

I reinstalled everything and now i get 

/app/chromedriver
Starting ChromeDriver 116.0.5845.180 (fa9889b94260f950a6ab6bfc140732263651f355-refs/branch-heads/5845@{#1746}) on port 9515
Only local connections are allowed.
Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
ChromeDriver was started successfully.

But same error with flaresolverr.

romanovj commented 11 months ago

disable vpn if you have

start with LOG_LEVEL=debug HOST=localhost python -u /app/flaresolverr.py

romanovj commented 11 months ago

upd. do not resolve localhost echo 127.0.0.1 localhost > /etc/hosts

also you can try to change dns (default 1.1.1.1)

echo nameserver 8.8.8.8 > /etc/resolv.conf error Exception: Error getting browser User-Agent. Message: Can not connect to the Service /app/chromedriver

echo nameserver 9.9.9.9 > /etc/resolv.conf ok, but isn't working with random vpn

echo nameserver 192.168.1.1 > /etc/resolv.conf ok, ok with vpn (it's my router ip)

PS. I don't know why it's try to resolve localhost and why CF and google dns aren't returning 127.0.0.1 for localhost PPS HOST=127.0.0.1 doesn't change anything

#google
nslookup localhost 8.8.8.8                                    Server:         8.8.8.8
Address:        8.8.8.8#53

** server can't find localhost: NXDOMAIN

#CF
flaresolverr@localhost:~$ nslookup localhost 1.1.1.1
Server:         1.1.1.1
Address:        1.1.1.1#53

** server can't find localhost: NXDOMAIN

#quad dns
flaresolverr@localhost:~$ nslookup localhost 9.9.9.9
Server:         9.9.9.9
Address:        9.9.9.9#53

Non-authoritative answer:
Name:   localhost
Address: 127.0.0.1
Name:   localhost
Address: ::1

#adguard 
nslookup localhost 94.140.14.14
Server:         94.140.14.14
Address:        94.140.14.14#53

Name:   localhost
Address: 127.0.0.1
Name:   localhost
Address: ::1
romanovj commented 11 months ago

also add 'dumb-init' in order to kill zombie processes

LOG_LEVEL=debug HOST=localhost dumb-init python -u /app/flaresolverr.py

Vojtak42 commented 11 months ago

Thank you! It works well with this: echo 127.0.0.1 localhost > /etc/hosts