search

scylladb / scylla-cluster-tests

Tests for Scylla Clusters
GNU Affero General Public License v3.0
55 stars 93 forks source link

InvalidURL: Failed to parse: http://[10.142.0.19]:9090/api/v1/status/config #1537

Closed amoskong closed 4 years ago

amoskong commented 4 years ago

Prerequisites

Versions

Logs

Description

[Description of the bug or feature]

Steps to Reproduce

  1. [First Step]
  2. [Second Step]
  3. [and so on...]

Expected behavior: [What you expected to happen]

Actual behavior: [What actually happened]

09:23:23  ======================================================================
09:23:23  ERROR: test_rolling_upgrade (upgrade_test.UpgradeTest)
09:23:23  ----------------------------------------------------------------------
09:23:23  Traceback (most recent call last):
09:23:23    File "/sct/sdcm/tester.py", line 138, in wrapper
09:23:23      args[0].tearDown()
09:23:23    File "/sct/sdcm/tester.py", line 1496, in tearDown
09:23:23      self.finalize_test()
09:23:23    File "/sct/sdcm/tester.py", line 1385, in finalize_test
09:23:23      self.stop_resources()
09:23:23    File "/sct/sdcm/tester.py", line 1422, in stop_resources
09:23:23      coredumps=db_cluster_coredumps,
09:23:23    File "/sct/sdcm/db_stats.py", line 548, in update_test_details
09:23:23      update_data['results'] = self.get_prometheus_stats()
09:23:23    File "/sct/sdcm/db_stats.py", line 482, in get_prometheus_stats
09:23:23      prometheus_db_stats = PrometheusDBStats(host=self.monitors.nodes[0].external_address)
09:23:23    File "/sct/sdcm/db_stats.py", line 135, in __init__
09:23:23      self.config = self.get_configuration()
09:23:23    File "/sct/sdcm/db_stats.py", line 159, in get_configuration
09:23:23      result = self.request(url="http://[{0.host}]:{0.port}/api/v1/status/config".format(self))
09:23:23    File "/sct/sdcm/utils/common.py", line 93, in inner
09:23:23      return func(*args, **kwargs)
09:23:23    File "/sct/sdcm/db_stats.py", line 147, in request
09:23:23      response = requests.get(url)
09:23:23    File "/jenkins/.local/lib/python2.7/site-packages/requests/api.py", line 75, in get
09:23:23      return request('get', url, params=params, **kwargs)
09:23:23    File "/jenkins/.local/lib/python2.7/site-packages/requests/api.py", line 60, in request
09:23:23      return session.request(method=method, url=url, **kwargs)
09:23:23    File "/jenkins/.local/lib/python2.7/site-packages/requests/sessions.py", line 519, in request
09:23:23      prep = self.prepare_request(req)
09:23:23    File "/jenkins/.local/lib/python2.7/site-packages/requests/sessions.py", line 462, in prepare_request
09:23:23      hooks=merge_hooks(request.hooks, self.hooks),
09:23:23    File "/jenkins/.local/lib/python2.7/site-packages/requests/models.py", line 313, in prepare
09:23:23      self.prepare_url(url, params)
09:23:23    File "/jenkins/.local/lib/python2.7/site-packages/requests/models.py", line 381, in prepare_url
09:23:23      raise InvalidURL(*e.args)
09:23:23  InvalidURL: Failed to parse: http://[10.142.0.19]:9090/api/v1/status/config
09:23:23  
09:23:23  ----------------------------------------------------------------------
09:23:23  Ran 1 test in 1797.239s
amoskong commented 4 years ago 
08:53:04   > git fetch --tags --progress git@github.com:scylladb/scylla-cluster-tests.git +refs/heads/*:refs/remotes/origin/* # timeout=10
08:53:05  Checking out Revision 2a79203c605dc6c9f22c107436727facf6f6348d (refs/remotes/origin/branch-3.2)
08:53:05   > git rev-parse refs/remotes/origin/branch-3.2^{commit} # timeout=10
08:53:05   > git rev-parse refs/remotes/origin/origin/branch-3.2^{commit} # timeout=10
08:53:06  Commit message: "Chaning AbortRepair from non-disruptive nemesis to Disruptive. (#1530)"
amoskong commented 4 years ago

Hi @fruch , it seems a ip list is wrongly assigned to PrometheusDBStats.host

But I didn't problem in sct code.

fruch commented 4 years ago

This is weird, we test those url when introduced IPv6 support. And this is the first time I see this issue happening

It reproduceable ?

amoskong commented 4 years ago

I only see this problem once, I will try to reproduce by rerun the job.

On Tue, Dec 10, 2019 at 10:17 PM Israel Fruchter notifications@github.com wrote:

This is weird, we test those url when introduced IPv6 support. And this is the first time I see this issue happening

It reproduceable ?

amoskong commented 4 years ago

I also see this problem in https://jenkins.scylladb.com/job/enterprise-2019.1/job/rolling-upgrade/job/rolling-upgrade-centos7/50/execution/node/56/log/

The job used new branch-2019.1 (92667e1594891289a852c46ae98b4b8abedf8247)

amoskong commented 4 years ago

sct master:

$ git grep 'http://\['
longevity_test.py:            endpoint_url = 'http://[{}]:{}'.format(self.db_cluster.nodes[0].external_address, alternator_port)
refresh_test.py:        cmd = 'curl http://[%s]:9090/api/v1/query_range?query=scylla_storage_proxy_coordinator_read_timeouts&start=%s&end=%s&step=60s' % (
sdcm/cluster.py:            set_grafana_url("http://[{0.external_address}]:{1.grafana_port}".format(node, self))
sdcm/cluster.py:        set_grafana_url("http://[{0.external_address}]:{1.grafana_port}".format(node, self))
sdcm/cluster.py:        annotations_url = "http://[{node_ip}]:{grafana_port}/api/annotations"
sdcm/cluster.py:        annotations_url = "http://[{node_ip}]:{grafana_port}/api/annotations"
sdcm/db_stats.py:        self.range_query_url = "http://[{0.host}]:{0.port}/api/v1/query_range?query=".format(self)
sdcm/db_stats.py:        result = self.request(url="http://[{0.host}]:{0.port}/api/v1/status/config".format(self))
sdcm/db_stats.py:        url = "http://[{0.host}]:{0.port}/api/v1/query_range?query=".format(self)
sdcm/db_stats.py:        url = "http://[{0.host}]:{0.port}/api/v1/admin/tsdb/snapshot".format(self)

Why do we also add square brackets for ipv4 address?

fruch commented 4 years ago

Cause we tested it, and it should work also for IPv4.

It's not clear why it's not working in those cases. cause it's not failing consistently on this

amoskong commented 4 years ago

requests.get(url="http://[127.0.0.1]:80").content works in my localhost.

I just checked the requests version of master and branch-2019.1, it's same

(requests==2.20.0)

amoskong commented 4 years ago

Failed again: https://jenkins.scylladb.com/job/scylla-staging/job/amos/job/rolling-upgrade-2019.1/job/rolling-upgrade-centos7/11/console

I found it's caused by urllib3 version.

We need to assign urllib3 version in rquirements-python.txt


diff --git a/requirements-python.txt b/requirements-python.txt
index c2dc8ae..c7c179c 100644
--- a/requirements-python.txt
+++ b/requirements-python.txt
@@ -24,3 +24,4 @@ tenacity==5.0.4
 https://github.com/fruch/repodataParser/archive/py3.zip ; python_version > '3'
 pytest==4.6.4
 tcconfig==0.23.0
+urllib3==1.24.3
amoskong commented 4 years ago

Quote from: https://pypi.org/project/urllib3/

1.25 (2019-04-22)
Require and validate certificates by default when using HTTPS (Pull #1507)
Upgraded urllib3.utils.parse_url() to be RFC 3986 compliant. (Pull #1487)
Added support for key_password for HTTPSConnectionPool to use encrypted key_file without creating your own SSLContext object. (Pull #1489)
Add TLSv1.3 support to CPython, pyOpenSSL, and SecureTransport SSLContext implementations. (Pull #1496)
Switched the default multipart header encoder from RFC 2231 to HTML 5 working draft. (Issue #303, PR #1492)
Fixed issue where OpenSSL would block if an encrypted client private key was given and no password was given. Instead an SSLError is raised. (Pull #1489)
Added support for Brotli content encoding. It is enabled automatically if brotlipy package is installed which can be requested with urllib3[brotli] extra. (Pull #1532)
Drop ciphers using DSS key exchange from default TLS cipher suites. Improve default ciphers when using SecureTransport. (Pull #1496)
Implemented a more efficient HTTPResponse.__iter__() method. (Issue #1483)

Implement RFC 3986 URL parsing #1487 https://github.com/urllib3/urllib3/pull/1487/files/aff6ef473c6b20e24218c7bbfede70fda5a57c57

https://tools.ietf.org/html/rfc3986

amoskong commented 4 years ago

Quote from rfc3986:

   A host identified by an Internet Protocol literal address, version 6
   [RFC3513] or later, is distinguished by enclosing the IP literal
   within square brackets ("[" and "]").  This is the only place where
   square bracket characters are allowed in the URI syntax.

Strictly, we can't use square brackets for ipv4 address.

fruch commented 4 years ago

This is not nice, since curl for example isn't that strict....

fruch commented 4 years ago

Can you raise as an issue in urllib asking a waver on that ? For usability purposes ?

amoskong commented 4 years ago

Can you raise as an issue in urllib asking a waver on that ? For usability purposes ?

I think we need to change our code in future. urllib3's new change is correct, they won't change it back.

bentsi commented 4 years ago

@amoskong I opened a task for that in Trello (https://trello.com/c/soUGS7tZ/1457-remove-brackets-usage-when-using-ipv4)

amoskong commented 4 years ago

Bad news:

The issue still occurred sometimes.

I checked manually, urllib3 in docker container is 1.24.3, unless the urllib3 is updated when test is running.

[jenkins@public-jenkins-builder1-qavpc scylla-cluster-tests]$ docker run -it scylladb/hydra:v0.55-2019.1 bash
[root@48ee32963f4a /]# pip show urllib3
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Name: urllib3
Version: 1.24.3
Summary: HTTP library with thread-safe connection pooling, file post, and more.
Home-page: https://urllib3.readthedocs.io/
Author: Andrey Petrov
Author-email: andrey.petrov@shazow.net
License: MIT
Location: /usr/lib/python2.7/site-packages
Requires: 
Required-by: requests, elasticsearch, botocore