search

jenkinsci / jenkins-charm

Juju charm to deploy and scale Jenkins
Other
17 stars 36 forks source link

/srv/mnt/jenkins storage dir is hard-coded, deviates from the default /var/lib/jenkins, is not configurable, and is broken #35

Closed ryan-beisner closed 5 years ago

ryan-beisner commented 7 years ago

/srv/mnt directory is hard-coded, deviates from the default /var/lib/jenkins, is not configurable, and appears to be broken in implementation.

See commits related to: https://github.com/jenkinsci/jenkins-charm/pull/18

I've deployed with "--storage jenkins=10G" as indicated by README, yet, the charm tries to create files in /var/lib/jenkins.

I'm finding usage roadblocks with and without storage defined, on the current revision of the Jenkins charm (cs:xenial/jenkins-4).

Attached full unit log. But here are some interesting snippets show the /srv/mnt/jenkins vs /var/lib/jenkins issues:

2017-04-12 16:12:28 INFO juju-log Invoking reactive handler: reactive/jenkins.py:181:attach
2017-04-12 16:12:28 INFO juju-log maintenance: Configuring Jenkins storage
2017-04-12 16:12:28 INFO juju-log Setting Jenkins to use storage at /srv/mnt/jenkins
2017-04-12 16:12:28 INFO juju-log active: Ready
2017-04-12 16:12:28 INFO juju-log Invoking reactive handler: reactive/apt.py:36:update
2017-04-12 16:12:29 INFO juju-log active: Updating apt cache
2017-04-12 16:13:09 INFO juju-log Writing file /var/lib/jenkins/config.xml jenkins:nogroup 444
2017-04-12 16:13:09 INFO juju-log Invoking reactive handler: reactive/apt.py:47:ensure_package_status
2017-04-12 16:13:09 INFO juju-log Unholding packages pwgen,daemon,default-jre-headless,python3-testtools,jenkins
2017-04-12 16:13:09 INFO juju-log Marking {'pwgen', 'daemon', 'default-jre-headless', 'python3-testtools', 'jenkins'} as unhold
2017-04-12 16:13:09 INFO jenkins-storage-attached pwgen was already not hold.
2017-04-12 16:13:09 INFO jenkins-storage-attached daemon was already not hold.
2017-04-12 16:13:09 INFO jenkins-storage-attached default-jre-headless was already not hold.
2017-04-12 16:13:09 INFO jenkins-storage-attached python3-testtools was already not hold.
2017-04-12 16:13:09 INFO jenkins-storage-attached jenkins was already not hold.
2017-04-12 16:13:09 INFO juju-log Invoking reactive handler: reactive/jenkins.py:98:configure_tools
2017-04-12 16:13:10 INFO juju-log maintenance: Installing tools
2017-04-12 16:13:10 INFO juju-log Invoking reactive handler: reactive/jenkins.py:109:configure_admin
2017-04-12 16:13:10 INFO juju-log maintenance: Configuring Jenkins public url
2017-04-12 16:13:10 INFO juju-log Writing file /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml jenkins:nogroup 444
2017-04-12 16:13:10 INFO juju-log maintenance: Configuring admin user
2017-04-12 16:13:10 INFO juju-log Configuring user for jenkins
2017-04-12 16:13:10 INFO juju-log Retrying '_make_client' 7 more times (delay=5)
2017-04-12 16:13:15 INFO juju-log Retrying '_make_client' 6 more times (delay=10)
2017-04-12 16:13:26 INFO juju-log Writing file /var/lib/jenkins/.admin_token root:root 600
2017-04-12 16:13:26 INFO juju-log Writing file /var/lib/jenkins/.admin_password root:root 600
2017-04-12 16:13:27 INFO juju-log Retrying '_make_client' 7 more times (delay=5)
2017-04-12 16:13:32 INFO juju-log Retrying '_make_client' 6 more times (delay=10)
2017-04-12 16:13:42 INFO juju-log Retrying '_make_client' 5 more times (delay=15)
2017-04-12 16:13:57 INFO juju-log Retrying '_make_client' 4 more times (delay=20)
2017-04-12 16:14:17 INFO juju-log Retrying '_make_client' 3 more times (delay=25)
2017-04-12 16:14:43 INFO juju-log Retrying '_make_client' 2 more times (delay=30)
2017-04-12 16:15:13 INFO juju-log Retrying '_make_client' 1 more times (delay=35)
2017-04-12 16:15:48 INFO jenkins-storage-attached Traceback (most recent call last):
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/jenkins/__init__.py", line 431, in jenkins_open
2017-04-12 16:15:48 INFO jenkins-storage-attached     response = urlopen(req, timeout=self.timeout).read()
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/lib/python3.5/urllib/request.py", line 163, in urlopen
2017-04-12 16:15:48 INFO jenkins-storage-attached     return opener.open(url, data, timeout)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/lib/python3.5/urllib/request.py", line 472, in open
2017-04-12 16:15:48 INFO jenkins-storage-attached     response = meth(req, response)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
2017-04-12 16:15:48 INFO jenkins-storage-attached     'http', request, response, code, msg, hdrs)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/lib/python3.5/urllib/request.py", line 510, in error
2017-04-12 16:15:48 INFO jenkins-storage-attached     return self._call_chain(*args)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
2017-04-12 16:15:48 INFO jenkins-storage-attached     result = func(*args)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
2017-04-12 16:15:48 INFO jenkins-storage-attached     raise HTTPError(req.full_url, code, msg, hdrs, fp)
2017-04-12 16:15:48 INFO jenkins-storage-attached urllib.error.HTTPError: HTTP Error 401: Invalid password/token for user: ubuntu
2017-04-12 16:15:48 INFO jenkins-storage-attached 
2017-04-12 16:15:48 INFO jenkins-storage-attached During handling of the above exception, another exception occurred:
2017-04-12 16:15:48 INFO jenkins-storage-attached 
2017-04-12 16:15:48 INFO jenkins-storage-attached Traceback (most recent call last):
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/var/lib/juju/agents/unit-osci-master-0/charm/hooks/jenkins-storage-attached", line 19, in <module>
2017-04-12 16:15:48 INFO jenkins-storage-attached     main()
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/__init__.py", line 78, in main
2017-04-12 16:15:48 INFO jenkins-storage-attached     bus.dispatch()
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 434, in dispatch
2017-04-12 16:15:48 INFO jenkins-storage-attached     _invoke(other_handlers)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 417, in _invoke
2017-04-12 16:15:48 INFO jenkins-storage-attached     handler.invoke()
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/charms/reactive/bus.py", line 291, in invoke
2017-04-12 16:15:48 INFO jenkins-storage-attached     self._action(*args)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/var/lib/juju/agents/unit-osci-master-0/charm/reactive/jenkins.py", line 126, in configure_admin
2017-04-12 16:15:48 INFO jenkins-storage-attached     users.configure_admin()
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "lib/charms/layer/jenkins/users.py", line 34, in configure_admin
2017-04-12 16:15:48 INFO jenkins-storage-attached     paths.LAST_EXEC, "{}\n".format(api.version()).encode("utf-8"),
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "lib/charms/layer/jenkins/api.py", line 47, in version
2017-04-12 16:15:48 INFO jenkins-storage-attached     self.wait()
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "lib/charms/layer/jenkins/api.py", line 43, in wait
2017-04-12 16:15:48 INFO jenkins-storage-attached     self._make_client()
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/charmhelpers/core/decorators.py", line 40, in _retry_on_exception_inner_2
2017-04-12 16:15:48 INFO jenkins-storage-attached     return f(*args, **kwargs)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "lib/charms/layer/jenkins/api.py", line 132, in _make_client
2017-04-12 16:15:48 INFO jenkins-storage-attached     client.get_whoami()
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/jenkins/__init__.py", line 575, in get_whoami
2017-04-12 16:15:48 INFO jenkins-storage-attached     response = self.jenkins_open(Request(self._build_url(WHOAMI_URL)))
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/jenkins/__init__.py", line 430, in jenkins_open
2017-04-12 16:15:48 INFO jenkins-storage-attached     self.maybe_add_crumb(req)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/jenkins/__init__.py", line 303, in maybe_add_crumb
2017-04-12 16:15:48 INFO jenkins-storage-attached     self._build_url(CRUMB_URL)), add_crumb=False)
2017-04-12 16:15:48 INFO jenkins-storage-attached   File "/usr/local/lib/python3.5/dist-packages/jenkins/__init__.py", line 448, in jenkins_open
2017-04-12 16:15:48 INFO jenkins-storage-attached     e.code, e.msg)
2017-04-12 16:15:48 INFO jenkins-storage-attached jenkins.JenkinsException: Error in request. Possibly authentication failed [401]: Invalid password/token for user: ubuntu
2017-04-12 16:15:48 ERROR juju.worker.uniter.operation runhook.go:107 hook "jenkins-storage-attached" failed: exit status 1
ryan-beisner commented 7 years ago

unit-osci-master-0.log.txt

freeekanayaka commented 7 years ago

@tkuhlman would you mind giving a look at this PR? As far as I understand it's a regression from the introduction of storage support. @ryan-beisner do you confirm that?

tkuhlman commented 7 years ago

To clarify a bit /srv/mnt/jenkins is the directory where the storage is mounted by Juju and then /var/lib/jenkins is linked to this so anything writing to /var/lib/jenkins should be fine. Likely that is not the real cause of the problem here. If you have a running instance it should be easy enough to verify the link.

I tried to deploy cs:xenial/jenkins-4 both with and without adding "--storage jenkins=10G" both worked fine for me.

The error above is an error logging into the new Jenkins installation. The charm does set the initial password and possibly something went wrong there. I noticed in your logs above you are running Jenkins version 2.7.4 where when I didn't specify I would get version 2.46.1. I tried again with a simple config specifying that the release be http://pkg.jenkins-ci.org/debian/binary/jenkins_2.7_all.deb and it still worked fine.

Clearly I am missing some details in trying to replicate this. What config options are you using? Also what version of Juju are you running?

mthaddon commented 5 years ago

I'm seeing the same problem using jenkins 2.150.1.

I think it's unrelated to storage, and is actually about api authentication. My instance at the time has:

$ ls -lh /var/lib/jenkins
lrwxrwxrwx 1 root root 16 Feb  6 15:52 /var/lib/jenkins -> /srv/mnt/jenkins

The call that's failing is _make_client from lib/charms/layer/jenkins/api.py and if I add some manual logging into the charm, I can see that value of "token" is "This user currently does not have a legacy token".

Need to do some more digging to figure out why this user does not have a legacy token, how we can authenticate to the API without one, and then can apply a fix from there.

mthaddon commented 5 years ago

Ah, which is https://github.com/jenkinsci/jenkins-charm/issues/47

mthaddon commented 5 years ago

Closing this out as #47 is now resolved.