search

scrapinghub / portia

Visual scraping for Scrapy
BSD 3-Clause "New" or "Revised" License
9.22k stars 1.41k forks source link

HTTPConnectionPool(host='localhost', port=6800): Max retries exceeded with url: /addversion.json #899

Closed imba-tjd closed 3 years ago

imba-tjd commented 4 years ago

I run it by docker run -v ~/portia_projects:/app/data/projects:rw -p 9001:9001 scrapinghub/portia (exactly the same command in Readme)

When I choose Deploy a project, I got this error:

图片

QQ截图20200523172346

The following content is produced by curl -d "" xxx:9001/api/projects/TEST/deploy because I can't copy the response in FF's network tab. Thus I can't promise it's 100% matching.

Response ``` ConnectionError at /api/projects/TEST/deploy HTTPConnectionPool(host='localhost', port=6800): Max retries exceeded with url: /addversion.json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) Request Method: POST Request URL: http://tjd:9001/api/projects/TEST/deploy Django Version: 2.2.3 Python Executable: /usr/bin/python3 Python Version: 3.5.2 Python Path: ['/app/portia_server', '/app/portia_server', '/app/slyd', '/app/slybot', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages'] Server time: Sat, 23 May 2020 09:24:58 +0000 Installed Applications: ['db_repo.apps.DbRepoConfig', 'storage.apps.StorageConfig', 'portia_orm.apps.PortiaOrmConfig', 'portia_api.apps.PortiaApiConfig', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'whitenoise.runserver_nostatic', 'django.contrib.staticfiles'] Installed Middleware: ['django.middleware.security.SecurityMiddleware', 'whitenoise.middleware.WhiteNoiseMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'portia_orm.middleware.ORMDataStoreMiddleware'] Traceback: File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py" in _new_conn 160. (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py" in create_connection 80. raise err File "/usr/local/lib/python3.5/dist-packages/urllib3/util/connection.py" in create_connection 70. sock.connect(sa) During handling of the above exception ([Errno 111] Connection refused), another exception occurred: File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py" in urlopen 603. chunked=chunked) File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py" in _make_request 355. conn.request(method, url, **httplib_request_kw) File "/usr/lib/python3.5/http/client.py" in request 1106. self._send_request(method, url, body, headers) File "/usr/lib/python3.5/http/client.py" in _send_request 1151. self.endheaders(body) File "/usr/lib/python3.5/http/client.py" in endheaders 1102. self._send_output(message_body) File "/usr/lib/python3.5/http/client.py" in _send_output 934. self.send(msg) File "/usr/lib/python3.5/http/client.py" in send 877. self.connect() File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py" in connect 183. conn = self._new_conn() File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py" in _new_conn 169. self, "Failed to establish a new connection: %s" % e) During handling of the above exception (: Failed to establish a new connection: [Errno 111] Connection refused), another exception occurred: File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py" in send 449. timeout=timeout File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py" in urlopen 641. _stacktrace=sys.exc_info()[2]) File "/usr/local/lib/python3.5/dist-packages/urllib3/util/retry.py" in increment 399. raise MaxRetryError(_pool, url, error or ResponseError(cause)) During handling of the above exception (HTTPConnectionPool(host='localhost', port=6800): Max retries exceeded with url: /addversion.json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))), another exception occurred: File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/exception.py" in inner 34. response = get_response(request) File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py" in _get_response 115. response = self.process_exception_by_middleware(e, request) File "/usr/local/lib/python3.5/dist-packages/django/core/handlers/base.py" in _get_response 113. response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/local/lib/python3.5/dist-packages/django/views/decorators/csrf.py" in wrapped_view 54. return view_func(*args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/rest_framework/viewsets.py" in view 95. return self.dispatch(request, *args, **kwargs) File "/usr/lib/python3.5/contextlib.py" in inner 30. return func(*args, **kwds) File "/app/portia_server/portia_api/resources/route.py" in dispatch 72. return super(JsonApiRoute, self).dispatch(request, *args, **kwargs) File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py" in dispatch 494. response = self.handle_exception(exc) File "/app/portia_server/portia_api/resources/route.py" in handle_exception 75. response = super(JsonApiRoute, self).handle_exception(exc) File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py" in handle_exception 454. self.raise_uncaught_exception(exc) File "/usr/local/lib/python3.5/dist-packages/rest_framework/views.py" in dispatch 491. response = handler(request, *args, **kwargs) File "/app/portia_server/portia_api/resources/projects.py" in deploy 172. data = self._deploy() File "/app/portia_server/portia_api/resources/projects.py" in _deploy 278. return Deployer(self.project).deploy() File "/app/portia_server/portia_api/utils/deploy/scrapyd.py" in deploy 63. requests.post(url, data=data, files=files, auth=auth) File "/usr/local/lib/python3.5/dist-packages/requests/api.py" in post 116. return request('post', url, data=data, json=json, **kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/api.py" in request 60. return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py" in request 533. resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/sessions.py" in send 646. r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.5/dist-packages/requests/adapters.py" in send 516. raise ConnectionError(e, request=request) Exception Type: ConnectionError at /api/projects/TEST/deploy Exception Value: HTTPConnectionPool(host='localhost', port=6800): Max retries exceeded with url: /addversion.json (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) Request information: USER: LocalUser(root) GET: No GET data POST: No POST data FILES: No FILES data COOKIES: No cookie data META: CONTENT_LENGTH = '0' CONTENT_TYPE = 'application/x-www-form-urlencoded' DEBIAN_FRONTEND = 'noninteractive' DJANGO_SETTINGS_MODULE = 'portia_server.settings' GATEWAY_INTERFACE = 'CGI/1.1' HOME = '/root' HOSTNAME = '535fd8e67341' HTTP_ACCEPT = '*/*' HTTP_CONNECTION = 'close' HTTP_HOST = 'tjd:9001' HTTP_USER_AGENT = 'curl/7.55.1' HTTP_X_FORWARDED_FOR = '117.147.12.62' HTTP_X_REAL_IP = '117.147.12.62' PATH = '/opt/qt59/5.9.1/gcc_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' PATH_INFO = '/api/projects/TEST/deploy' PWD = '/app/slyd' PYTHONPATH = '/app/portia_server:/app/slyd:/app/slybot' QT_MIRROR = 'http://ftp.fau.de/qtproject/official_releases/qt/5.9/5.9.1/qt-opensource-linux-x64-5.9.1.run' QUERY_STRING = '' REMOTE_ADDR = '127.0.0.1' REMOTE_HOST = '' REQUEST_METHOD = 'POST' RUN_MAIN = 'true' SCRIPT_NAME = '' SERVER_NAME = 'localhost' SERVER_PORT = '8000' SERVER_PROTOCOL = 'HTTP/1.0' SERVER_SOFTWARE = 'WSGIServer/0.2' SHLVL = '1' TZ = 'UTC' _ = '/app/portia_server/manage.py' wsgi.errors = <_io.TextIOWrapper name='' mode='w' encoding='ANSI_X3.4-1968'> wsgi.file_wrapper = '' wsgi.input = wsgi.multiprocess = False wsgi.multithread = True wsgi.run_once = False wsgi.url_scheme = 'http' wsgi.version = '(1, 0)' Settings: Using settings module portia_server.settings ABSOLUTE_URL_OVERRIDES = {} ADMINS = [] ALLOWED_HOSTS = ['*'] APPEND_SLASH = True AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend'] AUTH_PASSWORD_VALIDATORS = '********************' AUTH_USER_MODEL = 'auth.User' BASE_DIR = '/app/portia_server' BLACKLIST_URLS = set() CACHES = {'default': {'BACKEND': 'django.core.cache.backends.locmem.LocMemCache'}} CACHE_MIDDLEWARE_ALIAS = 'default' CACHE_MIDDLEWARE_KEY_PREFIX = '********************' CACHE_MIDDLEWARE_SECONDS = 600 CAPABILITIES = {'deploy_projects': True, 'rename_projects': True, 'create_projects': True, 'rename_templates': True, 'delete_projects': True, 'version_control': False, 'rename_spiders': True} CSRF_COOKIE_AGE = 31449600 CSRF_COOKIE_DOMAIN = None CSRF_COOKIE_HTTPONLY = False CSRF_COOKIE_NAME = 'csrftoken' CSRF_COOKIE_PATH = '/' CSRF_COOKIE_SAMESITE = 'Lax' CSRF_COOKIE_SECURE = False CSRF_FAILURE_VIEW = 'django.views.csrf.csrf_failure' CSRF_HEADER_NAME = 'HTTP_X_CSRFTOKEN' CSRF_TRUSTED_ORIGINS = [] CSRF_USE_SESSIONS = False CUSTOM = {} DATABASES = {'default': {'PORT': '', 'TEST': {'CHARSET': None, 'COLLATION': None, 'NAME': None, 'MIRROR': None}, 'PASSWORD': '********************', 'CONN_MAX_AGE': 0, 'HOST': '', 'ENGINE': 'django.db.backends.sqlite3', 'AUTOCOMMIT': True, 'OPTIONS': {}, 'USER': '', 'ATOMIC_REQUESTS': False, 'TIME_ZONE': None, 'NAME': '/app/portia_server/db.sqlite3'}} DATABASE_ROUTERS = [] DATA_UPLOAD_MAX_MEMORY_SIZE = 2621440 DATA_UPLOAD_MAX_NUMBER_FIELDS = 1000 DATETIME_FORMAT = 'N j, Y, P' DATETIME_INPUT_FORMATS = ['%Y-%m-%d %H:%M:%S', '%Y-%m-%d %H:%M:%S.%f', '%Y-%m-%d %H:%M', '%Y-%m-%d', '%m/%d/%Y %H:%M:%S', '%m/%d/%Y %H:%M:%S.%f', '%m/%d/%Y %H:%M', '%m/%d/%Y', '%m/%d/%y %H:%M:%S', '%m/%d/%y %H:%M:%S.%f', '%m/%d/%y %H:%M', '%m/%d/%y'] DATE_FORMAT = 'N j, Y' DATE_INPUT_FORMATS = ['%Y-%m-%d', '%m/%d/%Y', '%m/%d/%y', '%b %d %Y', '%b %d, %Y', '%d %b %Y', '%d %b, %Y', '%B %d %Y', '%B %d, %Y', '%d %B %Y', '%d %B, %Y'] DEBUG = True DEBUG_PROPAGATE_EXCEPTIONS = False DECIMAL_SEPARATOR = '.' DEFAULT_CHARSET = 'utf-8' DEFAULT_CONTENT_TYPE = 'text/html' DEFAULT_EXCEPTION_REPORTER_FILTER = 'django.views.debug.SafeExceptionReporterFilter' DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage' DEFAULT_FROM_EMAIL = 'webmaster@localhost' DEFAULT_INDEX_TABLESPACE = '' DEFAULT_TABLESPACE = '' DEPLOY_VERSION = None DISALLOWED_USER_AGENTS = [] EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend' EMAIL_HOST = 'localhost' EMAIL_HOST_PASSWORD = '********************' EMAIL_HOST_USER = '' EMAIL_PORT = 25 EMAIL_SSL_CERTFILE = None EMAIL_SSL_KEYFILE = '********************' EMAIL_SUBJECT_PREFIX = '[Django] ' EMAIL_TIMEOUT = None EMAIL_USE_LOCALTIME = False EMAIL_USE_SSL = False EMAIL_USE_TLS = False FILE_CHARSET = 'utf-8' FILE_UPLOAD_DIRECTORY_PERMISSIONS = None FILE_UPLOAD_HANDLERS = ['django.core.files.uploadhandler.MemoryFileUploadHandler', 'django.core.files.uploadhandler.TemporaryFileUploadHandler'] FILE_UPLOAD_MAX_MEMORY_SIZE = 2621440 FILE_UPLOAD_PERMISSIONS = None FILE_UPLOAD_TEMP_DIR = None FIRST_DAY_OF_WEEK = 0 FIXTURE_DIRS = [] FORCE_SCRIPT_NAME = None FORMAT_MODULE_PATH = None FORM_RENDERER = 'django.forms.renderers.DjangoTemplates' IGNORABLE_404_URLS = [] INSTALLED_APPS = ['db_repo.apps.DbRepoConfig', 'storage.apps.StorageConfig', 'portia_orm.apps.PortiaOrmConfig', 'portia_api.apps.PortiaApiConfig', 'django.contrib.auth', 'django.contrib.contenttypes', 'django.contrib.sessions', 'whitenoise.runserver_nostatic', 'django.contrib.staticfiles'] INTERNAL_IPS = [] LANGUAGES = [('af', 'Afrikaans'), ('ar', 'Arabic'), ('ast', 'Asturian'), ('az', 'Azerbaijani'), ('bg', 'Bulgarian'), ('be', 'Belarusian'), ('bn', 'Bengali'), ('br', 'Breton'), ('bs', 'Bosnian'), ('ca', 'Catalan'), ('cs', 'Czech'), ('cy', 'Welsh'), ('da', 'Danish'), ('de', 'German'), ('dsb', 'Lower Sorbian'), ('el', 'Greek'), ('en', 'English'), ('en-au', 'Australian English'), ('en-gb', 'British English'), ('eo', 'Esperanto'), ('es', 'Spanish'), ('es-ar', 'Argentinian Spanish'), ('es-co', 'Colombian Spanish'), ('es-mx', 'Mexican Spanish'), ('es-ni', 'Nicaraguan Spanish'), ('es-ve', 'Venezuelan Spanish'), ('et', 'Estonian'), ('eu', 'Basque'), ('fa', 'Persian'), ('fi', 'Finnish'), ('fr', 'French'), ('fy', 'Frisian'), ('ga', 'Irish'), ('gd', 'Scottish Gaelic'), ('gl', 'Galician'), ('he', 'Hebrew'), ('hi', 'Hindi'), ('hr', 'Croatian'), ('hsb', 'Upper Sorbian'), ('hu', 'Hungarian'), ('hy', 'Armenian'), ('ia', 'Interlingua'), ('id', 'Indonesian'), ('io', 'Ido'), ('is', 'Icelandic'), ('it', 'Italian'), ('ja', 'Japanese'), ('ka', 'Georgian'), ('kab', 'Kabyle'), ('kk', 'Kazakh'), ('km', 'Khmer'), ('kn', 'Kannada'), ('ko', 'Korean'), ('lb', 'Luxembourgish'), ('lt', 'Lithuanian'), ('lv', 'Latvian'), ('mk', 'Macedonian'), ('ml', 'Malayalam'), ('mn', 'Mongolian'), ('mr', 'Marathi'), ('my', 'Burmese'), ('nb', 'Norwegian Bokmål'), ('ne', 'Nepali'), ('nl', 'Dutch'), ('nn', 'Norwegian Nynorsk'), ('os', 'Ossetic'), ('pa', 'Punjabi'), ('pl', 'Polish'), ('pt', 'Portuguese'), ('pt-br', 'Brazilian Portuguese'), ('ro', 'Romanian'), ('ru', 'Russian'), ('sk', 'Slovak'), ('sl', 'Slovenian'), ('sq', 'Albanian'), ('sr', 'Serbian'), ('sr-latn', 'Serbian Latin'), ('sv', 'Swedish'), ('sw', 'Swahili'), ('ta', 'Tamil'), ('te', 'Telugu'), ('th', 'Thai'), ('tr', 'Turkish'), ('tt', 'Tatar'), ('udm', 'Udmurt'), ('uk', 'Ukrainian'), ('ur', 'Urdu'), ('vi', 'Vietnamese'), ('zh-hans', 'Simplified Chinese'), ('zh-hant', 'Traditional Chinese')] LANGUAGES_BIDI = ['he', 'ar', 'fa', 'ur'] LANGUAGE_CODE = 'en-us' LANGUAGE_COOKIE_AGE = None LANGUAGE_COOKIE_DOMAIN = None LANGUAGE_COOKIE_NAME = 'django_language' LANGUAGE_COOKIE_PATH = '/' LOCALE_PATHS = [] LOGGING = {} LOGGING_CONFIG = 'logging.config.dictConfig' LOGIN_REDIRECT_URL = '/accounts/profile/' LOGIN_URL = '/accounts/login/' LOGOUT_REDIRECT_URL = None MANAGERS = [] MEDIA_ROOT = '/app/data/projects' MEDIA_URL = '' MESSAGE_STORAGE = 'django.contrib.messages.storage.fallback.FallbackStorage' MIDDLEWARE = ['django.middleware.security.SecurityMiddleware', 'whitenoise.middleware.WhiteNoiseMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'django.middleware.clickjacking.XFrameOptionsMiddleware', 'portia_orm.middleware.ORMDataStoreMiddleware'] MIGRATION_MODULES = {} MONTH_DAY_FORMAT = 'F j' NUMBER_GROUPING = 0 PASSWORD_HASHERS = '********************' PASSWORD_RESET_TIMEOUT_DAYS = '********************' PORTIA_STORAGE_BACKEND = 'storage.backends.FsStorage' PREPEND_WWW = False PROJECT_DEPLOYER = 'portia_api.utils.deploy.scrapyd.ScrapydDeploy' REST_FRAMEWORK = {'DEFAULT_AUTHENTICATION_CLASSES': ('portia_server.backends.LocalAuthentication',), 'URL_FORMAT_OVERRIDE': None, 'EXCEPTION_HANDLER': 'portia_api.jsonapi.exceptions.jsonapi_exception_handler'} ROOT_URLCONF = 'portia_server.urls' SCRAPINGHUB_APIKEY = '********************' SCRAPYD_CFG_DEFAULT = None SCRAPYD_PASSWORD = '********************' SCRAPYD_URL = 'http://localhost:6800/' SCRAPYD_USERNAME = None SECRET_KEY = '********************' SECURE_BROWSER_XSS_FILTER = False SECURE_CONTENT_TYPE_NOSNIFF = False SECURE_HSTS_INCLUDE_SUBDOMAINS = False SECURE_HSTS_PRELOAD = False SECURE_HSTS_SECONDS = 0 SECURE_PROXY_SSL_HEADER = None SECURE_REDIRECT_EXEMPT = [] SECURE_SSL_HOST = None SECURE_SSL_REDIRECT = False SERVER_EMAIL = 'root@localhost' SESSION_CACHE_ALIAS = 'default' SESSION_COOKIE_AGE = 1209600 SESSION_COOKIE_DOMAIN = None SESSION_COOKIE_HTTPONLY = True SESSION_COOKIE_NAME = 'sessionid' SESSION_COOKIE_PATH = '/' SESSION_COOKIE_SAMESITE = 'Lax' SESSION_COOKIE_SECURE = False SESSION_ENGINE = 'django.contrib.sessions.backends.db' SESSION_EXPIRE_AT_BROWSER_CLOSE = False SESSION_FILE_PATH = None SESSION_SAVE_EVERY_REQUEST = False SESSION_SERIALIZER = 'django.contrib.sessions.serializers.JSONSerializer' SETTINGS_MODULE = 'portia_server.settings' SHORT_DATETIME_FORMAT = 'm/d/Y P' SHORT_DATE_FORMAT = 'm/d/Y' SIGNING_BACKEND = 'django.core.signing.TimestampSigner' SILENCED_SYSTEM_CHECKS = [] STATICFILES_DIRS = [] STATICFILES_FINDERS = ['django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder'] STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage' STATIC_ROOT = '/app/portiaui/dist' STATIC_URL = '/' TEMPLATES = [] TEST_NON_SERIALIZED_APPS = [] TEST_RUNNER = 'django.test.runner.DiscoverRunner' THOUSAND_SEPARATOR = ',' TIME_FORMAT = 'P' TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] TIME_ZONE = 'UTC' USE_I18N = True USE_L10N = True USE_THOUSAND_SEPARATOR = False USE_TZ = True USE_X_FORWARDED_HOST = False USE_X_FORWARDED_PORT = False WSGI_APPLICATION = 'portia_server.wsgi.application' X_FRAME_OPTIONS = 'SAMEORIGIN' YEAR_MONTH_FORMAT = 'F Y' You're seeing this error because you have DEBUG = True in your Django settings file. Change that to False, and Django will display a standard page generated by the handler for this status code ```

Besides, I can't enter anything in Enter a website address when it's behind nginx L7 proxy.

图片

alessandroias commented 3 years ago

did you managed to solve this issue? and if you did: HOW?

imba-tjd commented 3 years ago

No. The developers has decided not to maintain this project anymore, so I closed this.

alessandroias commented 3 years ago

Thanks anyway for the reply. I have a similar error in a django-scrapyd application, but I can't manage to find a solution. It is driving me crazy. Have a good day, and happy coding

Dr-Eagle commented 2 years ago

hello @alessandroias did you find a solution ?

alessandroias commented 2 years ago

hello @alessandroias did you find a solution ?

Kinda, I changed the way I was dockerizing the app, and used docker network.

Dr-Eagle commented 2 years ago

can you explain how you manage to solve it thank you