search

joelee2012 / api4jenkins

Python sync/async client library for Jenkins API
https://api4jenkins.readthedocs.io/
Apache License 2.0
89 stars 22 forks source link

Port number is ignored #9

Closed delenius closed 3 years ago

delenius commented 3 years ago

When using a non-default port, e.g.

J = Jenkins('http://localhost:9090/', auth=('admin', 'admin'))
job = J.get_job('MetaWorker')
item = job.build()

the library still uses port 8080.

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /job/MetaWorker/build (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fae50594e20>: Failed to establish a new connection: [Errno 61] Connection refused'))

joelee2012 commented 3 years ago

hi @delenius, thanks for reporting issue, however i can't reproduce:

>>> from api4jenkins import Jenkins
>>> j = Jenkins('http://localhost:9090/', auth=('admin', '123456'))
>>> job = j.get_job('test1')
>>> item = job.build()
>>> build = item.get_build()
>>> build.result
'SUCCESS'
>>> for line in build.console_text():
...     print(line)
... 
b'Started by user admin'
b'Running as SYSTEM'
b'Building in workspace /var/jenkins_home/workspace/test1'
b'[test1] $ /bin/sh -xe /tmp/jenkins1111554451879568386.sh'
b'+ echo this is test'
b'this is test'
b'Finished: SUCCESS'

are you able to run pip list to gather python packages are installed in your env and provide full trace log. thanks

delenius commented 3 years ago

Here is the pip list output. As you can see, I tried a couple of other jenkins libraries too.

$ pip list
Package         Version
--------------- -------------------
api4jenkins     1.0
argcomplete     1.12.1
catkin-pkg      0.4.23
certifi         2020.6.20
chardet         3.0.4
docutils        0.16
empy            3.3.4
idna            2.10
ifcfg           0.21
jenkins         1.0.2
jenkinsapi      0.3.11
lark-parser     0.10.1
lxml            4.6.1
multi-key-dict  2.0.3
netifaces       0.10.9
numpy           1.19.2
pbr             5.5.1
pip             20.2.4
pydot           1.4.1
pygraphviz      1.6
pyparsing       2.4.7
python-dateutil 2.8.1
python-jenkins  1.7.0
pytz            2020.4
PyYAML          5.3.1
requests        2.25.0
setuptools      50.3.0.post20201006
six             1.15.0
urllib3         1.26.2
wheel           0.35.1
joelee2012 commented 3 years ago

hi @delenius , i still can't reproduce the issue, here i create a pure env for testing:

(testing) ➜  .local pip list
Package     Version  
----------- ---------
api4jenkins 1.0      
certifi     2020.12.5
chardet     3.0.4    
idna        2.10     
pip         19.0.3   
requests    2.25.0   
setuptools  40.8.0   
urllib3     1.26.2   
You are using pip version 19.0.3, however version 20.3.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(testing) ➜  .local python
Python 3.7.3 (default, Apr 24 2020, 18:51:23) 
[Clang 11.0.3 (clang-1103.0.32.62)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from api4jenkins import Jenkins
>>> j = Jenkins('http://localhost:9090/', auth=('admin', '123456'))
>>> job = j.get_job('test')
>>> item = job.build()
>>> build = item.get_build()
>>> for line in build.console_text():
...     print(line)
... 
b'Started by user admin'
b'Running as SYSTEM'
b'Building in workspace /var/jenkins_home/workspace/test'
b'[test] $ /bin/sh -xe /tmp/jenkins5995599370908134910.sh'
b'+ echo this is test'
b'this is test'
b'Finished: SUCCESS'

as you can see i installed requests, urllib3 same as yours, can you try to do same with following steps:

python3 -m venv testing
. ./testing/bin/activate
pip install api4jenkins
python

then run code what i did in above with your user and password. if any issue happens, please paste the full log.

delenius commented 3 years ago

Yeah, same problem.

Perhaps I should mention that Jenkins is running inside a VM, on port 8080, but I am forwarding it to port 9090 on the host (via VMWare Fusion 12).

I can browse the Jenkins Web UI on the host at http://localhost:9090. Maybe one of the http libraries is getting confused.

$ python
Python 3.9.1 (default, Dec 10 2020, 10:36:35)
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from api4jenkins import Jenkins
>>> j = Jenkins('http://localhost:9090/', auth=('admin','admin'))
>>> job = j.get_job('MetaWorker')
>>> item = job.build()
Traceback (most recent call last):
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/Cellar/python@3.9/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x10e4e7700>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /job/MetaWorker/build (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10e4e7700>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/api4jenkins/job.py", line 120, in build
    resp = self.handle_req('POST', entry, params=parameters)
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/api4jenkins/item.py", line 51, in handle_req
    return self.jenkins.send_req(method, self.url + entry, **kwargs)
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/api4jenkins/requester.py", line 26, in send
    resp = session.request(method, url, **kwargs, **kw)
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/elenius/repos/nscore/uuv_client/testing/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /job/MetaWorker/build (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10e4e7700>: Failed to establish a new connection: [Errno 61] Connection refused'))
>>>
joelee2012 commented 3 years ago

it's strange, as we can see it got job successful which means it connected the Jenkins on 9090.

joelee2012 commented 3 years ago

Not sure if it's because you are using python 3.9 which may not be supported by urllib3 on mac, see https://github.com/urllib3/urllib3/pull/1778, can you try python3.7 or python3.8

delenius commented 3 years ago

Same result with Python 3.8.5 (I was actually using that initially, via a conda environment):

$ python
Python 3.8.5 (default, Sep  4 2020, 02:22:02)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from api4jenkins import Jenkins
>>> j = Jenkins('http://localhost:9090/', auth=('admin','admin'))
>>> job = j.get_job('MetaWorker')
>>> item = job.build()
Traceback (most recent call last):
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
    raise err
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/connection.py", line 200, in connect
    conn = self._new_conn()
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fc9a81c3190>: Failed to establish a new connection: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/connectionpool.py", line 783, in urlopen
    return self.urlopen(
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /job/MetaWorker/build (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc9a81c3190>: Failed to establish a new connection: [Errno 61] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/api4jenkins/job.py", line 120, in build
    resp = self.handle_req('POST', entry, params=parameters)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/api4jenkins/item.py", line 51, in handle_req
    return self.jenkins.send_req(method, self.url + entry, **kwargs)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/api4jenkins/requester.py", line 26, in send
    resp = session.request(method, url, **kwargs, **kw)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/elenius/miniconda3/envs/nscore/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /job/MetaWorker/build (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc9a81c3190>: Failed to establish a new connection: [Errno 61] Connection refused'))
>>>
delenius commented 3 years ago

... and same result with Python 3.7.9.

joelee2012 commented 3 years ago

hi @delenius i can reproduce this issue now, it's because the Jenkins URL in system configuration is http://localhost:8080/, which should http://localhost:9090/, please change it accordingly and try.

delenius commented 3 years ago

@joelee2012 your message was cut off...

joelee2012 commented 3 years ago

it's was typo 😄 , have you tested it ?

delenius commented 3 years ago

@joelee2012 , I don't understand what you mean by "system configuration". Where do I change it? Another curious thing: If I change the url to e.g. http://localhost:90/, then I get an error like 

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=90)

i.e. the new port is accepted. But if I use http://localhost:9090/, then it uses 8080:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080)

I can verify on the command line that I can access the URL, e.g.

curl --user "admin:admin" http://localhost:9090/

returns a bunch of HTML.

delenius commented 3 years ago

It looks like the issue does happen in your library. The port number is lost on get_job:

$ python
Python 3.8.5 (default, Sep  4 2020, 02:22:02)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from api4jenkins import Jenkins
>>> J = Jenkins('http://localhost:9090',auth=('admin','admin'))
>>> job = J.get_job('MetaWorker')
>>> job.url
'http://localhost:8080/job/MetaWorker/'

I have not yet been able to track down exactly where the port number gets lost.

delenius commented 3 years ago

Ooh, I think I understand the problem now. The library uses the job URL returned from the server, which has a different port number, because the server doesn't know that it is being mapped to a different port. Right? This seems like a flaw in the Jenkins api. A lot of APIs get forwarded, especially when used in VMs or containers.

I might have to use urllib directly, and contruct these URLs myself.

delenius commented 3 years ago

Ah, I figured out what you mean by "system configuration". After changing the URL in the Jenkins settings to the "external" URL (i.e. port 9090), everything works. Thanks for the help, closing this issue!