Open cpoole opened 8 years ago
tailing the opscode-erchef* gives me more info:
==> /var/log/opscode/opscode-erchef/erchef.log <==
2016-04-21 12:23:12.560 [error] LDAP search failed unexpectedly: noSuchObject
which seems to come from https://github.com/chef/chef-server/blob/bb28b489960c8fae6ac061bf2dab5800142b22a3/src/oc_erchef/apps/oc_chef_wm/src/oc_chef_wm_authn_ldap.erl
Did more digging: Chef server appears to ignore the ssl_enabled and tls_enabled flags in the chef-server.rb file.
With tls_enabled set to false and ssl_enabled set to true the chef server's client hello only offers TLS cipher suites.
This might be the real root of the cause, since foxpass claims to only support SSL
Important packet trace shows chef-server closes the connection with a TLSv1.2 Record Layer: Encrypted Alert. This is likely the close_notify alert to end the session, meaning chef is successfully reaching ldap but is either not successfully binding or is executing the search incorrectly
Timeout for connections is 600
tcpick: reading from synconnections.pcap
1 SYN-SENT 192.168.10.244:58885 > 54.210.170.147:ldaps
1 SYN-RECEIVED 192.168.10.244:58885 > 54.210.170.147:ldaps
1 ESTABLISHED 192.168.10.244:58885 > 54.210.170.147:ldaps
R.......e..X...$.(.&.*.k.j.=.#.'.%.).g.@.<.
...9.8.....5...........
. ...3.2...../............... ...{.........ldap.foxpass.com.
............. .
...........................................
....Q...M..?.Vr..2....eWa&4.o...L..G...)Jo. .Jw/.Fx......[.z&..
.....0G1.0H..=....U....US1.0...U.\..-0..)0.......... L!0
..........0..0.1.0...U....ldap.foxpass.com0.."00..
......VO.N.......3..!w..P..y..]``>w.E.....r5.b....2.\..'.-.Y.~uM?..U+.........xr..0H.Fe....,.]"...~./@.~..._.T.....&.k.?.....CF:.a...1..'....E3:!...!)....#..E...;..9r..V.4.e!I...hW.K.d<y.p......$...c.h..].1._%.t.q.J.a..;TyMr........(.........55......w.<:d8k8...........H0..D0...U.#..0........F.4..F..|[....Y0W..+........K0I0...+.....0...http://gv.symcd.com0&..+.....0...http://gv.symcb.com/gv.crt0...U...........0...U.%..0...+.........+.......0...U....0...ldap.foxpass.com0+...,..-..N.*-.r.g...P.lr.M.=Ui..}n(y.'..-.E....e0z...@.&.S..$S.>.`5.{.M7.....p....I.K..-`|.....gCys://www.rapidssl.com/legal0
.....0B1.0H...= ..U....US1.0...U...8r.,T-....D..9hs.o....h.R.........fi..........Z8...;@>].L........za..\.V..}.....R.b.ZOlr..#..)0..%0..
220520213932Z0G1.0.U......U....US1.0...U.A0..
..........0..1 0...U....RapidSSL SHA256 CA - G30.."0
......T..X].,V.....}.....Z....T....].....
..e_...D|....J...C.....&.a..f1".44_??./D_.......K..........iR..d......p.....j,...i..
..cZhq.{0.......>H.Uz.................c.../..W.<H..h!./..?.......S..rO`.B.....W.....~.N.!.S.6.H....H.).d..THS...e..)3).)`H....WX.5..i.Y.........0...0...U.#..0....z.h.....d..}.}e...N0...U...........F.4..F..|[....Y0...U.......0.......0...U...........05..U....0,0*.(.&.$http://g.symcb.com/crls/gtglobal.crl0...+........"0 0...+.....0...http://g.symcd.com0L..U. .E0C0A.
0.._a,..........bLp&............~Nd...aU..:.]%....JtVO.U@pu%.3..K.]S.nE.....I......Y...,;..Z..{.5{..3.{.q"B...oO.....y..J.w .......T..{
...........=..b.?.sxC.....p.........Kx'YkuKC..]...v.Z..g.M?3....%...',..B.x.V.K....O.p..K.K........[..$....C.\..YX...............9'..K...'"......9~..Sgk...xsP#z.X....N..1...&.c.|......#..T... .Y.......6.4.`VL|...*r....C....p#i...$.y..Y..!.>...
..........Pl.C.$.h@vL!,.....z/B..h.....9./....Q.......4.:S..Y...yr.z.. ..w.wyxm2^...VA.E...
..........P........},<..4,.z....".Y....2.{.Mt...h.....f..L.D..Zq...m.R..NY.z8.1.`.e.7Q..f.~
....p..c..F.ec.%....).. .$m.u-w...5n...!....g.1.M....o'.P.....(.LT..+S......
.$..........PH....~8a.
....|0vD..v.$cQ.,
....`.._..fm......[.2K...Tk....(..5.F(". mfJ.w.v..*...z.5m.e..*o..;R..;t !....
.$.
m%UJG...6......".*
.6.n ..x.(...z(....r...[GBz.C...d.`<G.,....J.a..4..U.U.x@....M.)2/M......xt$&..
....@*..%p.nP_.'.d.V....@..E..<6...5..b...Q....c.Q.{..)...s`.*.m....R
....@.J..............Ij.......].N..T..)X.J.o..F].c..j.. G{.(Lx.."g...
1 FIN-WAIT-1 192.168.10.244:58885 > 54.210.170.147:ldaps
1 FIN-WAIT-2 192.168.10.244:58885 > 54.210.170.147:ldaps
1 TIME-WAIT 192.168.10.244:58885 > 54.210.170.147:ldaps
1 CLOSED 192.168.10.244:58885 > 54.210.170.147:ldaps
tcpick: done reading from synconnections.pcap
25 packets captured
1 tcp sessions detected
Following is the client hello showing chef server only offers TLS despite disabling it in the settings
Secure Sockets Layer
SSL Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 256
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 252
Version: TLS 1.2 (0x0303)
Random
gmt_unix_time: Apr 21, 2016 16:33:45.000000000 PDT
random_bytes: 3d499f73fcb2838ec45bbddf4e0fe01b79ec0d522e8294c4...
Session ID Length: 0
Cipher Suites Length: 88
Cipher Suites (44 suites)
Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 (0xc026)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 (0xc02a)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 (0xc025)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008)
Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012)
Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
Cipher Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc003)
Cipher Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA (0xc00d)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA (0xc007)
Cipher Suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA (0xc011)
Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
Cipher Suite: TLS_DHE_RSA_WITH_DES_CBC_SHA (0x0015)
Cipher Suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA (0xc002)
Cipher Suite: TLS_ECDH_RSA_WITH_RC4_128_SHA (0xc00c)
Cipher Suite: TLS_RSA_WITH_DES_CBC_SHA (0x0009)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Extensions Length: 123
Extension: server_name
Type: server_name (0x0000)
Length: 21
Server Name Indication extension
Server Name list length: 19
Server Name Type: host_name (0)
Server Name length: 16
Server Name: ldap.foxpass.com
Extension: elliptic_curves
Type: elliptic_curves (0x000a)
Length: 58
Elliptic Curves Length: 56
Elliptic curves (28 curves)
Elliptic curve: sect571r1 (0x000e)
Elliptic curve: sect571k1 (0x000d)
Elliptic curve: secp521r1 (0x0019)
Elliptic curve: Unknown (0x001c)
Elliptic curve: sect409k1 (0x000b)
Elliptic curve: sect409r1 (0x000c)
Elliptic curve: Unknown (0x001b)
Elliptic curve: secp384r1 (0x0018)
Elliptic curve: sect283k1 (0x0009)
Elliptic curve: sect283r1 (0x000a)
Elliptic curve: Unknown (0x001a)
Elliptic curve: secp256k1 (0x0016)
Elliptic curve: secp256r1 (0x0017)
Elliptic curve: sect239k1 (0x0008)
Elliptic curve: sect233k1 (0x0006)
Elliptic curve: sect233r1 (0x0007)
Elliptic curve: secp224k1 (0x0014)
Elliptic curve: secp224r1 (0x0015)
Elliptic curve: sect193r1 (0x0004)
Elliptic curve: sect193r2 (0x0005)
Elliptic curve: secp192k1 (0x0012)
Elliptic curve: secp192r1 (0x0013)
Elliptic curve: sect163k1 (0x0001)
Elliptic curve: sect163r1 (0x0002)
Elliptic curve: sect163r2 (0x0003)
Elliptic curve: secp160k1 (0x000f)
Elliptic curve: secp160r1 (0x0010)
Elliptic curve: secp160r2 (0x0011)
Extension: ec_point_formats
Type: ec_point_formats (0x000b)
Length: 2
EC point formats Length: 1
Elliptic curves point formats (1)
EC point format: uncompressed (0)
Extension: signature_algorithms
Type: signature_algorithms (0x000d)
Length: 26
Data (26 bytes)
for comparison here is a successfull tcp trace for ldapsearch
Timeout for connections is 600
tcpick: reading from synconnections.pcap
1 SYN-SENT 192.168.10.244:48189 > 52.91.119.240:ldaps
1 SYN-RECEIVED 192.168.10.244:48189 > 52.91.119.240:ldaps
1 ESTABLISHED 192.168.10.244:48189 > 52.91.119.240:ldaps
....p...l..W.q..K....4.x...B..}_t..p.K.......0.3.g.E.9.k.....2.@.D.8.j.....f./.<.A.5.=...
..............
..........
.....0G1.0H.......U....US1.0...U..)0...........{L!0.. ...dA......6...
..........0..0.1.0...U....ldap.foxpass.com0.."00..
......VO.N.......3..!w..P..y..]``>w.E.....r5.b....2.\..'.-.Y.~uM?..U+.........xr..0H.Fe....,.]"...~./@.~..._.T.....&.k.?.....CF:.a...1..'....E3:!...!)....#..E...;..9r..V.4.e!I...hW.K.d<y.p......$...c.h..].1._%.t.q.J.a..;TyMr........(.........55......w.<:d8k8...........H0..D0...U.#..0........F.4..F..|[....Y0W..+........K0I0...+.....0...http://gv.symcd.com0&..+.....0...http://gv.symcb.com/gv.crt0...U...........0...U.%..0...+.........+.......0...U....0...ldap.foxpass.com0+...,..-..N.*-.r.g...P.lr.M.=Ui..}n(y.'..-.E....e0z...@.&.S..$S.>.`5.{.M7.....p....I.K..-`|.....gCys://www.rapidssl.com/legal0
.....0B1.0H...= ..U....US1.0...U...8r.,T-....D..9hs.o....h.R.........fi..........Z8...;@>].L........za..\.V..}.....R.b.ZOlr..#..)0..%0..
220520213932Z0G1.0.U......U....US1.0...U.A0..
..........0..1 0...U....RapidSSL SHA256 CA - G30.."0
......T..X].,V.....}.....Z....T....].....
..e_...D|....J...C.....&.a..f1".44_??./D_.......K..........iR..d......p.....j,...i..
..cZhq.{0.......>H.Uz.................c.../..W.<H..h!./..?.......S..rO`.B.....W.....~.N.!.S.6.H....H.).d..THS...e..)3).)`H....WX.5..i.Y.........0...0...U.#..0....z.h.....d..}.}e...N0...U...........F.4..F..|[....Y0...U.......0.......0...U...........05..U....0,0*.(.&.$http://g.symcb.com/crls/gtglobal.crl0...+........"0 0...+.....0...http://g.symcd.com0L..U. .E0C0A.
0.._a,..........bLp&............~Nd...aU..:.]%....JtVO.U@pu%.3..K.]S.nE.....I......Y...,;..Z..{.5{..3.{.q"B...oO.....y..J.w .......T..{
...........^t.D.Q..3(.s]. ..vn..R............R.mCn..x..Y.........N.j{.-4.h(..F............A,.E.U/.W.".U!F_....Z...TY...W.YR. .. p.J".....4Q......g...C_X,Qw1..z.3.C:.=..g.:...k...._....B....8 .....3i....d....^\..^....|..u.........&..6b"...w......'Ws.n..pD.U..Y(....
.6
......
.........y..Y.ui.......'....Ji.....96_@.j)"...*...V{...m.g.......Z...~.y'.Z..w.9...u..|.9O..J~....Ck.T\..]q.....D..4..K.{...52..(Y..XI..M.I./..@.iec....QK."|r.v....Ek...`....1.q.......{.-FI.h....lp.;.j..I.W|#.h...xYASBz......(E..Dc..y.!.bpxl...;..R.c.... DSg...
..........@.g'...K..<...*.M...j].0..).&..Y....
.5
i..bn....GQZ...(.{.u...fr
..............6.>..X.;.h<...........6.2kx....D...|.B*........Ng.2.G'.h.....I=..&9MB{=NG.....N.j...;.8.LBU.a.D.....Tu...]..........2..
....P..@F.8....:QcS..Te7.t....lQ....;...a^$.=.?..4*. ).d..q0z...nh9...f...|.ao.}F...\
......C..:..........S..s_......,..w.k....?\....c.....\+.'o.U.b\...A"Xj.J....{.Lsp]|KfQ..f.. .t.....l.~ n.H..fC.b.A.vu...Q.E....l.BO.K6..w. 7.=......d.eFD1n...#.=..k...K......#.._..^...%..,4....r.y.
.......m.~.t........N....r.Ds...OmCN.X8.^.B.
v."'.Re.<k<.pF........s_./R$.lK.....JC .......\...(.r3.C....}........' ...Z......H.-h..E....H....{..?b....#[.TAc.......qg.Y..g..68F.o.1.8......S.
.....g.:t/....,9.ix..\....B.@........O.Q....R.>T.R#.\.......Q....u@.cW...7'.\.|.i...s.....E....V...+..U.).bv.e.....'cg~.f...a.]#..}.t.y(.......@... E...q.....b..Xp..H..$
|..o.>]...z]N......\.-.......+./.=9.
......Q.0(U.K...}...o:..'w.
U.S~.#.mosW..:......d7.6...+0. ....%...r.&...cj'....[........0.+i...,..&w.......U...V.eWa<.w....3."......
........>.=.9d5.CI=1.B.].......^..E..7..W.Fu. ...$.a..4..9.q.cm.H...s..I........r{...d....g..9.M.Sq...J..c...k.aDmY.d.....<IQ*.6!/'.2..........[}...K
1 FIN-WAIT-1 192.168.10.244:48189 > 52.91.119.240:ldaps
1 FIN-WAIT-2 192.168.10.244:48189 > 52.91.119.240:ldaps
1 TIME-WAIT 192.168.10.244:48189 > 52.91.119.240:ldaps
1 CLOSED 192.168.10.244:48189 > 52.91.119.240:ldaps
tcpick: done reading from synconnections.pcap
29 packets captured
1 tcp sessions detected
there are four more packets sent and obviously more encrypted LDAP packets are sent back and forth
Same here! Getting a 504. The LDAP server is an OpenLDAP.
@gfoligna did you ever get a resolution?
I'm a bit surprised that this issue isn't being addressed at all...
I too am experiencing this.
Instead of ssl/tls_enabled, does it behave when you set:
ldap['enable_ssl'] = true
ldap['enable_tls'] = true
@cpoole @gfoligna did the suggestion above resolve this issue for you?
same issue here.. @marcparadise that did not work for me.
@marcparadise I'm still researching why the current code looks like it does, but here is what I've found so far:
There are basically two ways the eldap module supports creating a secure connection: (1) Using ssl:connect from the outset to create an encrypted(i.e. LDAPS) or (2) Using tcp:connect from the outset to create an unencrypted connection and then calling start_tls to upgrade the connection (i.e. STARTTLS).
In our config enable_ssl
means we will use the first method and corresponds to adding {ssl, true}
to the options for eldap:open/2
. enable_tls
means we will use thee second method and call eldap:start_tls/3
after making a connection.
As far as I can see there are no differences between what protocols or ciphers would be offered in either case. They should offer any protocol that the ssl application is configured to support (since we don't pass any custom ssl options) which, in erlang 17.5, should include "sslv3, tlsv1.0, tlsv1.1, tlsv1.2". I think the naming confusion in the options is a misunderstanding of what "start_tls" means.
If we wanted user-controllable ssl protocols we'd have to offer up a way to have them set custom sslopts() during the open or start_tls calls.
Now, my research here indicates that you should be at least getting SSLv3 offered; however, I'll need to look more carefully at the data @cpoole offered to figure out whether you are and if not, why not.
hey everyone, we gave up and just went with hosted chef and manually creating accounts. I have since torn down the proof of concept server.
I'm sure I can stand this up quickly again and make some more trial connections if need be. I can probably get log entries from foxpass as well... but the fact that this is a 504 makes me suspicious that the requests are not reaching foxpass's application servers
@cpoole Thanks for the offer but I wouldn't go out of your way, it is easy enough for us to set up a test locally. Any users currently hitting problems with LDAP should also feel free to let us know what they are seeing.
sounds good. My specific use case was with the hosted LDAP provider foxpass (great service btw). Their founder might have some insight as well... paging @aren
Happy to help debug. aren@foxpass.com.
I finally got my test instance back up to not impact production. What information can I provide to help move this along? Our setup is LDAP secure (636) with self signed certs
UPDATE Ok, I finally got this working Here are the settings I had to use:
ldap['base_dn'] = 'ASK_LDAP_ADMIN' ldap['bind_dn'] = 'ASK_LDAP_ADMIN' ldap['bind_password'] = 'ASK_LDAP_ADMIN' ldap['host'] = 'ASK_LDAP_ADMIN' ldap['port'] = '636' ldap['ssl_enabled'] = 'true'
ldap['system_adjective'] = 'NOT_REALLY_USED'
ldap['tls_enabled'] = 'false'
@cpoole Sorry for the late reply on this. We will try to pull this example into our tests see if the same issue still exists with the latest chef-server. If so we can try to schedule to fix soon on the roadmap. We have done a lot of work around testing ldap setup and that should make the setup of this test easier.
following the guide to install chef-server standalone on ubuntu 14.04 https://docs.chef.io/install_server.html
When configuring the chef-server.rb
If I execute ldap search from the same box as chef-server is running on everything works properly. however when I attempt to log in to chef the following appears in the logs.
these errors show up in the chef logs milliseconds after clicking login through the management console so a 504 seems like an improper error code.
I have spoken with foxpath and there is no inbound connection to their ldap service, something in chef server is failing to connect to ldap and is throwing the error.