search

chef-boneyard / opscode-pushy-server

Chef Push Jobs Server
https://docs.chef.io/push_jobs.html
Apache License 2.0
16 stars 10 forks source link

Seeing 500 issues related to SSL connection issues #119

Closed cparedes closed 7 years ago

cparedes commented 8 years ago

System details:

I've attempted to use chef-server-ctl reconfigure and opscode-push-jobs-server reconfigure with no change in behavior. I've restarted everything as well after attempting a reconfigure.

I've verified my erchef nginx certs using openssl s_client, and saw that it was valid with the CA certs already on my system.

I'm getting these errors every time I attempt to query the server with either knife node status or knife job list:

(client side)

knife job list
ERROR: Server returned error 500 for https://SERVER_URL/organizations/REDACTED/pushy/jobs, retrying 1/5 in 4s
ERROR: Server returned error 500 for https://SERVER_URL/organizations/REDACTED/pushy/jobs, retrying 2/5 in 6s

(server side)

2015-10-31 02:31:04.681 [error] <0.336.0> SSL: certify: ssl_handshake.erl:239:Fatal error: certificate unknown
2015-10-31 02:31:04.682 [error] <0.296.0> Webmachine error at path "/organizations/REDACTED/pushy/jobs" : {throw,{error,{conn_failed,{error,"certificate unknown"}}},[{pushy_http_common,fetch_authenticated,2,[{file,"src/pushy_http_common.erl"},{line,44}]},{pushy_org,fetch_org_id,1,[{file,"src/pushy_org.erl"},{line,38}]},{pushy_object,fetch_org_id,1,[{file,"src/pushy_object.erl"},{line,45}]},{pushy_wm_base,verify_request_signature,2,[{file,"src/pushy_wm_base.erl"},{line,157}]},{pushy_wm_base,is_authorized,2,[{file,"src/pushy_wm_base.erl"},{line,135}]},{webmachine_resource,resource_call,3,[{file,"..."},...]},...]}

Not sure what else I could be missing. I've checked https://tickets.opscode.com/browse/CHEF-5144 and seems it could be similar? Saw that the Erlang version bundled with opscode-push-jobs-server is R15B03.

cparedes commented 8 years ago

After chatting with support folks, I've tested 2.0.0~alpha.3 with erlang R16B03-1 and everything seems to work. The issue in R15B03 might have to do with the kind of SSL cert I'm passing in (wildcard cert, sha256 + RSA encryption)...

e100 commented 8 years ago

I can confirm that 2.0.0~alpha.3 also resolved this problem for me.

cparedes commented 8 years ago

Just as a side note too, my hunch is that the reason why I saw the error in the first place was because one of the chain certificates was probably encoded in sha384 or sha512, which is not supported in R15B03. Might be worth checking if that's the case, if your server cert is sha256 but still exhibits the same behavior on the current stable version.

e100 commented 8 years ago

In my case one of the chain certs is signed with sha384

mengesb commented 8 years ago

So I'm experiencing this same issue; could someone help with a solution? I'm using a comodo wildcard ssl cert and I think that I have the same problems as searches lead me back to here, and I'm not quite sure how I'd update the erlang processor in an embedded app setup like this. Instructions would be excellent.

2016-02-28 02:40:44 =ERROR REPORT====
SSL: certify: ssl_handshake.erl:239:Fatal error: certificate unknown
2016-02-28 02:40:44 =ERROR REPORT====
webmachine error: path="/organizations/<ORG>/pushy/config/<HOST>"
{throw,{error,{conn_failed,{error,"certificate unknown"}}},[{pushy_http_common,fetch_authenticated,2,[{file,"src/pushy_http_common.erl"},{line,44}]},{pushy_org,fetch_org_id,1,[{file,"src/pushy_org.erl"},{line,38}]},{pushy_object,fetch_org_id,1,[{file,"src/pushy_object.erl"},{line,45}]},{pushy_wm_base,verify_request_signature,2,[{file,"src/pushy_wm_base.erl"},{line,157}]},{pushy_wm_base,is_authorized,2,[{file,"src/pushy_wm_base.erl"},{line,135}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,183}]},{webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,141}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}]}
opscode-push-jobs-server 1.1.6+20141204195817

Component                  Installed Version                          Version GUID                                   
---------------------------------------------------------------------------------------------------------------------
autoconf                   2.68                                       md5:c3b5247592ce694f7097873aa07d66fe           
automake                   1.11.2                                     md5:79ad64a9f6e83ea98d6964cef8d8a0bc           
berkshelf2                 2.0.18                                     
bundler                    1.5.3                                      
cacerts                    2014.08.20                                 md5:c9f4f7f4d6a5ef6633e893577a09865e           
chef-gem                   11.12.2                                    
curl                       7.36.0                                     md5:643a7030b27449e76413d501d4b8eb57           
erlang                     R15B03-1                                   md5:eccd1e6dda6132993555e088005019f2           
gdbm                       1.9.1                                      md5:59f6e4c4193cb875964ffbe8aa384b58           
libedit                    20120601-3.0                               md5:e50f6a7afb4de00c81650f7b1a0f5aea           
libffi                     3.0.13                                     md5:45f3b6dbc9ee7c7dfbbbc5feba571529           
libgcc                     0.0.1                                      
libiconv                   1.14                                       md5:e34509b1623cec449dfeb73d7ce9c6c6           
liblzma                    5.0.5                                      md5:19d924e066b6fff0bc9d1981b4e53196           
libtool                    2.4                                        md5:b32b04148ecdd7344abc6fe8bd1bb021           
libuuid                    2.21                                       md5:4222aa8c2a1b78889e959a4722f1881a           
libxml2                    2.9.1                                      md5:9c0cfef285d5c4a5c80d00904ddab380           
libxslt                    1.1.28                                     md5:9667bf6f9310b957254fdcf6596600b7           
libyaml                    0.1.6                                      md5:5fe00cda18ca5daeb43762b80c38e06e           
libzmq                     v2.1.11                                    git:73f167eeb5ce9d26678399a574918f9813976024   
makedepend                 1.0.5                                      md5:efb2d7c7e22840947863efaedc175747           
ncurses                    5.9                                        md5:8cb9c412e5f2d96bc6f459aa8c6282a1           
nokogiri                   1.6.2.1                                    
oc-pushy-pedant            1.0.9                                      git:889d1c9e5c36df0babb4727f2467d7a004e36aec   
omnibus-ctl                0.0.7                                      git:0dae72b0f55f804294e004632ffaea4418d094a5   
openssl                    1.0.1i                                     md5:c8dc151a671b9b92ff3e4c118b174972           
opscode-pushy-server       1.1.0                                      git:4683cfad2e90d8a8bc3cd6bf154bb4e4dd3e6a49   
opscode-pushy-server-ctl   1.1.6+20141204195817                       
pkg-config                 0.28                                       md5:aa3c86e67551adc3ac865160e34a2a0d           
preparation                1.0.0                                      
pushy-server-cookbooks     1.1.6+20141204195817                       
pushy-server-schema        1.0.0                                      git:2557b0c5b61d19b66b2c05764b8c8ef895a360af   
pushy-server-scripts       1.1.6+20141204195817                       
rebar                      93621d0d0c98035f79790ffd24beac94581b0758   git:93621d0d0c98035f79790ffd24beac94581b0758   
ruby                       1.9.3-p547                                 md5:7531f9b1b35b16f3eb3d7bea786babfd           
rubygems                   1.8.24                                     md5:3a555b9d579f6a1a1e110628f5110c6b           
runit                      2.1.1                                      md5:8fa53ea8f71d88da9503f62793336bc3           
util-macros                1.18.0                                     md5:fd0ba21b3179703c071bbb4c3e5fb0f4           
version-manifest           0.0.1                                      
xproto                     7.0.25                                     md5:a47db46cb117805bd6947aa5928a7436           
zlib                       1.2.6                                      md5:618e944d7c7cd6521551e30b32322f4a
e100 commented 8 years ago

Go here: https://packagecloud.io/chef/current Find push-jobs-server package version 2.0.0~alpha.3 or newer for your platform and install it.

You will also need to update your push-jobs-clients to 2.0

mengesb commented 8 years ago

My attempt at upgrading to opscode-push-jobs-server-2.0.0~alpha.4+20160226080810 is resulting in HTTP 400 messages:

<push.client.public.ip> - - [28/Feb/2016:04:31:50 +0000]  "GET /organizations/<ORG>/pushy/config/<HOST> HTTP/1.1" 400 "0.001" 26 "-" "Chef Client/12.4.3 (ruby-2.1.6-p336; ohai-8.5.1; x86_64-linux; +http://opscode.com)" "<chef.server.local.ip>:10003" "400" "0.001" "12.4.3" "algorithm=sha1;version=1.0;" "<HOST>" "2016-02-28T04:31:54Z" "2jmj7l5rSw0yVb/vlWAYkK/YBwk=" 1053
mengesb commented 8 years ago

OK now there are 404s after updating the push-job-client installations to a 2.x.x build:

==> /var/log/opscode/opscode-erchef/erchef.log <==
2016-02-28 04:44:17.302 [error] {<<"method=GET; path=/organizations/<ORG>/groups/pushy_job_readers; status=404; ">>,"Not Found"}

==> /var/log/opscode/opscode-erchef/crash.log <==
2016-02-28 04:44:17 =ERROR REPORT====
{<<"method=GET; path=/organizations/<ORG>/groups/pushy_job_readers; status=404; ">>,"Not Found"}

==> /var/log/opscode/opscode-erchef/current <==
2016-02-28_04:44:17.31472 [error] {<<"method=GET; path=/organizations/<ORG>/groups/pushy_job_readers; status=404; ">>,"Not Found"}

==> /var/log/opscode/opscode-erchef/requests.log.1 <==
2016-02-28T04:44:17Z erchef@127.0.0.1 method=GET; path=/organizations/<ORG>/principals/<USER>; status=200; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAHc6gAAAAAAAAAA; org_name=<ORG>; req_time=3; rdbms_time=0; rdbms_count=3; req_api_version=1; 
2016-02-28T04:44:17Z erchef@127.0.0.1 method=GET; path=/organizations/<ORG>/groups/pushy_job_readers; status=404; req_id=g3IAA2QAEGVyY2hlZkAxMjcuMC4wLjEBAAHdCgAAAAAAAAAA; org_name=mengesio; msg=group_not_found; couchdb_groups=false; couchdb_organizations=false; couchdb_containers=false; couchdb_acls=false; 503_mode=false; couchdb_associations=false; couchdb_association_requests=false; req_time=3; rdbms_time=0; rdbms_count=3; user=pivotal; req_api_version=1;

re-running chef-client on my clients resulted in the cookbook failing due to 'no version info' for several erb resource templates. Not sure what to do about the 404's now.

mengesb commented 8 years ago

Looks like my client isn't able to successfully run:

2016-02-28_04:53:13.29198 INFO: [<HOST>] Starting client ...
2016-02-28_04:53:13.29203 INFO: [<HOST>] Retrieving configuration from https://chef-01.<DOMAIN>/organizations/<ORG>//pushy/config/<HOST>: ...
2016-02-28_04:53:13.29214 /opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/opscode-pushy-client-2.0.2/lib/pushy_client.rb:209:in `rest': uninitialized constant Chef::REST (NameError)
2016-02-28_04:53:13.29216   from /opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/opscode-pushy-client-2.0.2/lib/pushy_client.rb:220:in `get_config'
2016-02-28_04:53:13.29217   from /opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/opscode-pushy-client-2.0.2/lib/pushy_client.rb:94:in `start'
2016-02-28_04:53:13.29217   from /opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/opscode-pushy-client-2.0.2/lib/pushy_client/cli.rb:138:in `run_application'
2016-02-28_04:53:13.29218   from /opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/chef-12.8.0/lib/chef/application.rb:58:in `run'
2016-02-28_04:53:13.29218   from /opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/opscode-pushy-client-2.0.2/bin/pushy-client:8:in `<top (required)>'
2016-02-28_04:53:13.29218   from /opt/push-jobs-client/bin/pushy-client:53:in `load'
2016-02-28_04:53:13.29219   from /opt/push-jobs-client/bin/pushy-client:53:in `<main>'

Unfortunately I don't see an 'alpha' designated build on el6:

# yum --showduplicates --disablerepo=* --enablerepo=chef_current,chef-stable,chef_stable list push-jobs-client
Loaded plugins: fastestmirror, presto
Loading mirror speeds from cached hostfile
Installed Packages
push-jobs-client.x86_64          2.0.2+20160226200016-1.el6     @chef_current
Available Packages
push-jobs-client.x86_64          1.3.3-1.el6                                   chef-stable  
push-jobs-client.x86_64          1.3.4-1.el6                                   chef-stable  
push-jobs-client.x86_64          2.0.2+20160226200016-1.el6     chef_current

The prior installed version before 2.0.2 was installed was push-jobs-client-1.3.4-1.el6.x86_64

mengesb commented 8 years ago

OK so I finally fixed it. Seems there's a problem with the rest function usage/definition in pushy_client.rb at line 209.

Upgraded opscode-push-jobs-server' to2.0.0~alpha.4+20160226080810`

  1. Installed chef_current packagecloud.io repository: curl -s https://packagecloud.io/install/repositories/chef/current/script.rpm.sh | sudo bash
  2. Ensured it was disabled: sudo sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/chef_current.repo
  3. Installed 2.0.0~alpha.4+20160226080810 version of opscode-push-jobs-server: sudo yum --disablerepo=* --enablerepo=chef_current install -y opscode-push-jobs-server-2.0.0~alpha.4+20160226080810
  4. Run opscode-push-jobs-server-ctl reconfigure

Here's what I've done to get push-jobs-cient working at all clients:

  1. Installed chef_current packagecloud.io repository: curl -s https://packagecloud.io/install/repositories/chef/current/script.rpm.sh | sudo bash
  2. Ensured it was disabled: sudo sed -i "s/enabled=1/enabled=0/g" /etc/yum.repos.d/chef_current.repo
  3. Installed 2.0.2+20160226200016-1.el6 version of push-jobs-client: sudo yum --disablerepo=* --enablerepo=chef_current install -y push-jobs-client-2.0.2+20160226200016-1.el6

  4. Hacked /opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/opscode-pushy-client-2.0.2/lib/pushy_client.rb

    --- /a/opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/opscode-pushy-client-2.0.2/lib/pushy_client.rb    2016-02-28 05:41:01.924003375 +0000
+++ /b/opt/push-jobs-client/embedded/lib/ruby/gems/2.1.0/gems/opscode-pushy-client-2.0.2/lib/pushy_client.rb    2016-02-28 05:41:23.952003106 +0000
@@ -206,7 +206,10 @@
private

def rest
-    @rest ||= Chef::REST.new(chef_server_url, client_name, client_key)
+    @rest ||= begin
+      require 'chef/rest'
+      Chef::REST.new(chef_server_url, client_name, client_key)
+    end
end

def get_config
  5. Restarted the client just to be safe.
mengesb commented 8 years ago

Submitted the PR on opscode-pushy-client

https://github.com/chef/opscode-pushy-client/pull/84

charlesjohnson commented 7 years ago

As there have been no updates to this issue in 162 days, and there have been new major releases of both the push-jobs server and client in the interim, I'm going to close this ticket as stale.

@mengesb if this issue is still valid, please re-open, we'll be happy to assist further.