Closed step21 closed 2 years ago
I ran into the same issue. This happens, when 2captcha is under high load and results in
[2022/01/20 16:08:29|abstract_crawler.py|DEBUG ]: Captcha promise: 503 Service Unavailable
Just wait until https://2captcha.com/public_statistics/service-load is back to normal load and restart the script again.
Cool, thanks for providing more input. I am running the script with supervisor anyway, so it restarts automatically. Ideally, this would be captured and a wait added so that 2captcha is not re-tried too often. (though I am not sure how often this is queried atm)
@step21 how did you setup supervisor with pip? I have installed this package on a VPS and I can execute this job with this script:
#!/bin/bash
pipenv shell
/home/adam/.local/share/virtualenvs/flathunter-SBM5Nxub/bin/python /home/adam/flathunter/flathunt.py 2>> /home/adam/flathunter/job.log
When I try to run this script using supervisor I get this issue:
Traceback (most recent call last):
File "/home/adam/flathunter/flathunt.py", line 95, in <module>
main()
File "/home/adam/flathunter/flathunt.py", line 68, in main
config = Config(config_handle.name)
File "/home/adam/flathunter/flathunter/config.py", line 29, in __init__
self.__searchers__ = [CrawlImmobilienscout(self),
File "/home/adam/flathunter/flathunter/crawl_immobilienscout.py", line 43, in __init__
self.driver = self.configure_driver(self.driver_executable_path, self.driver_arguments)
File "/home/adam/flathunter/flathunter/abstract_crawler.py", line 53, in configure_driver
driver = webdriver.Chrome(executable_path=driver_path, options=chrome_options)
File "/home/adam/.local/share/virtualenvs/flathunter-SBM5Nxub/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/home/adam/.local/share/virtualenvs/flathunter-SBM5Nxub/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/home/adam/.local/share/virtualenvs/flathunter-SBM5Nxub/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/adam/.local/share/virtualenvs/flathunter-SBM5Nxub/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/adam/.local/share/virtualenvs/flathunter-SBM5Nxub/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Any hints?
My supervisor file:
[program:flathunter]
command=/home/<user>/.local/bin/pipenv run python /home/<user>/flathunter/flathunt.py
directory=/home/<user>/flathunter/
user=<user>
autostart=true
autorestart=true
redirect_stderr=true
stdout_logfile=/home/<user>/flathunter/supervisor_flathunter.log
It uses pipenv instead of only pip, which does not replace pip but is the best venv tool I know. Though as you have a problem with google-chrome/chromedriver, I suspect the problem may be another one. Quick googling suggests that this f.e. happens when you runs as the wrong user or wrong permissions. If I had to guess, I would guess you didn't set a user, so supervisor is executing the process as its own user or at least not the one you want. It also helps to set a working directory for the process if you didn't do that yet.
@step21 okay thank you! What is the working directory? Is it where directory
points to? And should this only include the .conf
file? Thats all I found when searching for working directory: http://supervisord.org/running.html?highlight=working%20directory#running-supervisord
You have to differentiate between the main supervisor config file (which is what the link refers to) and the individual service config file. With any program you or supervisor run, it often has a "current working directory" which in a terminal is the directory you are currently in or also in python is available via os.getcwd()
. In many cases this will not matter, but it will if f.e. you try to open a file without an absolute path, and also if you specify a log file without an absolute path, where it will be. In my config example above, I just set it to the flathunter directory.
@step21 sorry for asking once again. Tried to setup supervisor as you said, so I installed pipenv, now I get:
Traceback (most recent call last):
File "/home/adam/flathunter/flathunt.py", line 15, in <module>
from flathunter.hunter import Hunter
File "/home/adam/flathunter/flathunter/hunter.py", line 5, in <module>
from flathunter.config import Config
File "/home/adam/flathunter/flathunter/config.py", line 4, in <module>
import yaml
ModuleNotFoundError: No module named 'yaml'
`` `
@iwasherefirst2 Are you running flathunter using the pipenv run
command? If you run pipenv shell
on the command line and then try typing import yaml
into the python interpreter, does that work for you?
@codders yes, I have the command command=/usr/local/bin/pipenv run python /home/adam/flathunter/flathunt.py
in supervisor.
If I run this on the console it works on my VPN, but the supervisor fails. Also import yaml works in python interpreter when calling pipenv shell.
Sorry for only commenting now. This might seem/suggest that your shell environment is not the same for supervisor and when running via shell. Something similar f.e. happened to me for example when re-deploying for someone else, when the username was different, or when using su without switches to make it use a login shell. In my case, supervisor explicitly runs the flathunter as a normal user. If you run it as root, or in testing mixed it and sometimes run as root, sometimes as user, this might be the problem.
Anyway, closing this as the original problem is solved IIRC. If you still experience this issue @iwasherefirst2 please open a new one.
After some time, it crashed with the following error. I will try to narrow it down further or maybe just restart via supervisor.