search

JuliaWeb / HTTP.jl

HTTP for Julia
https://juliaweb.github.io/HTTP.jl/stable/
Other
632 stars 175 forks source link

TLS Renegotiation is disabled, causing EOFError() at knb.ecoinformatics.org #342

Closed oxinabox closed 3 years ago

oxinabox commented 5 years ago

Julia Version 1.0.0 HTTP v0.7.1

I am trying to fetch https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N

so I run

HTTP.request("GET", "https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N")

Which has the following result:

julia> HTTP.request("GET", "https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N")
ERROR: IOError(EOFError() during request(https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N))

Stacktrace:
 [1] readuntil(::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, ::typeof(HTTP.Parsers.find_end_of_header)) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\IOExtras.jl:168
 [2] readheaders(::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, ::HTTP.Messages.Response) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\Messages.jl:469
 [3] startread(::HTTP.Streams.Stream{HTTP.Messages.Response,HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}}) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\Streams.jl:149
 [4] macro expansion at .\task.jl:263 [inlined]
 [5] macro expansion at C:\Users\White\.julia\packages\HTTP\YjRCz\src\StreamRequest.jl:56 [inlined]
 [6] macro expansion at .\task.jl:244 [inlined]
 [7] #request#1(::Nothing, ::Nothing, ::Int64, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Type{HTTP.StreamRequest.StreamLayer}, ::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, ::HTTP.Messages.Request, ::Array{UInt8,1}) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\StreamRequest.jl:54
 [8] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction,),Tuple{Nothing}}, ::typeof(HTTP.request), ::Type{HTTP.StreamRequest.StreamLayer}, ::HTTP.ConnectionPool.Transaction{MbedTLS.SSLContext}, ::HTTP.Messages.Request, ::Array{UInt8,1}) at .\none:0
 [9] #request#1(::Nothing, ::Type, ::Int64, ::Base.Iterators.Pairs{Symbol,Nothing,Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{Nothing}}}, ::Function, ::Type{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\ConnectionRequest.jl:52
 [10] #request at .\none:0 [inlined]
 [11] #request#1 at C:\Users\White\.julia\packages\HTTP\YjRCz\src\ExceptionRequest.jl:19 [inlined]
 [12] (::getfield(HTTP, Symbol("#kw##request")))(::NamedTuple{(:iofunction,),Tuple{Nothing}}, ::typeof(HTTP.request), ::Type{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at .\none:0
 [13] (::getfield(Base, Symbol("###44#45#46")){ExponentialBackOff,getfield(HTTP.RetryRequest, Symbol("##2#3")){Bool,HTTP.Messages.Request},typeof(HTTP.request)})(::Base.Iterators.Pairs{Symbol,Nothing,Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{Nothing}}}, ::Function, ::Type, ::Vararg{Any,N} where N) at .\error.jl:216
 [14] #request#1(::Int64, ::Bool, ::Base.Iterators.Pairs{Symbol,Nothing,Tuple{Symbol},NamedTuple{(:iofunction,),Tuple{Nothing}}}, ::Function, ::Type{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}, ::HTTP.URIs.URI, ::HTTP.Messages.Request, ::Array{UInt8,1}) at .\none:0
 [15] #request at .\none:0 [inlined]
 [16] #request#1(::VersionNumber, ::String, ::Nothing, ::Nothing, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Type{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\MessageRequest.jl:47
 [17] request at C:\Users\White\.julia\packages\HTTP\YjRCz\src\MessageRequest.jl:28 [inlined]
 [18] #request#1(::Int64, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\RedirectRequest.jl:24
 [19] request(::Type{HTTP.RedirectRequest.RedirectLayer{HTTP.MessageRequest.MessageLayer{HTTP.RetryRequest.RetryLayer{HTTP.ExceptionRequest.ExceptionLayer{HTTP.ConnectionRequest.ConnectionPoolLayer{HTTP.StreamRequest.StreamLayer}}}}}}, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\RedirectRequest.jl:21
 [20] #request#3(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::String, ::HTTP.URIs.URI, ::Array{Pair{SubString{String},SubString{String}},1}, ::Array{UInt8,1}) at C:\Users\White\.julia\packages\HTTP\YjRCz\src\HTTP.jl:302
 [21] #request#4 at C:\Users\White\.julia\packages\HTTP\YjRCz\src\HTTP.jl:302 [inlined]
 [22] request at C:\Users\White\.julia\packages\HTTP\YjRCz\src\HTTP.jl:312 [inlined] (repeats 2 times)
 [23] top-level scope at none:0

Note that

bash> curl -I https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N

and

bash> curl https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N

work.

as does openning it in a webbrowser.

Originally raised in https://github.com/oxinabox/DataDepsGenerators.jl/issues/67

oxinabox commented 5 years ago

Taking a look at

oxinabox@motsugo ~> curl -D - https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N | cat -A
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0HTTP/1.1 200 OK^M$
Date: Thu, 25 Oct 2018 03:24:01 GMT^M$
Server: Apache/2.4.7 (Ubuntu)^M$
Vary: User-Agent^M$
Set-Cookie: JSESSIONID=E3F0DF506200EA58F63B37D2B688256C; Path=/knb/; Secure^M$
Content-Disposition: inline; filename="doi:10.5063/F1T43R7N.xml"^M$
X-Frame-Options: sameorigin^M$
Access-Control-Allow-Origin: ^M$
Access-Control-Allow-Headers: Authorization, Content-Type, Origin, Cache-Control^M$
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS^M$
Access-Control-Allow-Credentials: true^M$
Transfer-Encoding: chunked^M$
Content-Type: text/xml^M$
^M$
<?xml version="1.0" encoding="UTF-8"?><eml:eml xmlns:eml="eml://ecoinformatics.org/eml-2.1.1" xmlns:xsi="http://w
ww.w3.org/2001/XMLSchema-instance" packageId="doi:10.5063/F1T43R7N" scope="system" system="knb" xsi:schemaLocatio
n="eml://ecoinformatics.org/eml-2.1.1 eml.xsd"> <access authSystem="knb" order="allowFirst" scope="document"> <al
low> <principal>public</principal>$
...

The ^M escape code is \r, and $ is \r\n

So we are looking at headers ended with \r\n\r\n.

samoconnor commented 5 years ago

It looks like the server sends a TLS "Hello Request" right after the GET request is sent...

untitled
samoconnor commented 5 years ago

Turning of host cert validation in curl seems to cause the same behaviour

$ curl --insecure -v https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N
*   Trying 128.111.54.111...
* TCP_NODELAY set
* Connected to knb.ecoinformatics.org (128.111.54.111) port 443 (#0)
* WARNING: disabling hostname validation also disables SNI.
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: knb.ecoinformatics.org
* Server certificate: Let's Encrypt Authority X3
* Server certificate: DST Root CA X3
> GET /knb/d1/mn/v2/object/doi:10.5063/F1T43R7N HTTP/1.1
> Host: knb.ecoinformatics.org
> User-Agent: curl/7.54.0
> Accept: */*
>
* SSLRead() return error -9841
* Closing connection 0
curl: (56) SSLRead() return error -9841
$
samoconnor commented 5 years ago

This is a log compiled with MbedTLS.jl/src/debug.jl -> DEBUG_LEVEL=3:

MBTLS: 2018-10-25T15:32:37.192 8013 🤝 ...
MBTLS: 2018-10-25T15:32:37.213 8013 f_send ➡️  429
MBTLS: 2018-10-25T15:32:37.312 8013 f_recv WANT_READ
MBTLS: 2018-10-25T15:32:37.312 8013 🤝  ⌛️
MBTLS: 2018-10-25T15:32:37.422 8013 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:37.425 8013 f_recv ⬅️  65
MBTLS: 2018-10-25T15:32:37.425 8013 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:37.425 8013 f_recv ⬅️  2773
MBTLS: 2018-10-25T15:32:37.426 8013 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:37.426 8013 f_recv ⬅️  333
MBTLS: 2018-10-25T15:32:37.426 8013 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:37.426 8013 f_recv ⬅️  4
MBTLS: 2018-10-25T15:32:37.433 8013 f_send ➡️  75
MBTLS: 2018-10-25T15:32:37.433 8013 f_send ➡️  6
MBTLS: 2018-10-25T15:32:37.433 8013 f_send ➡️  45
MBTLS: 2018-10-25T15:32:37.433 8013 f_recv WANT_READ
MBTLS: 2018-10-25T15:32:37.434 8013 🤝  ⌛️
MBTLS: 2018-10-25T15:32:37.866 8013 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:37.866 8013 f_recv ⬅️  218
MBTLS: 2018-10-25T15:32:37.866 8013 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:37.866 8013 f_recv ⬅️  1
MBTLS: 2018-10-25T15:32:37.866 8013 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:37.866 8013 f_recv ⬅️  40
MBTLS: 2018-10-25T15:32:37.866 8013 🤝  ✅
DEBUG: 2018-10-25T15:32:38.244 8013 ➡️  "GET /knb/d1/mn/v2/object/doi:10.5063/F1T43R7N HTTP/1.1\r\n" (write)
MBTLS: 2018-10-25T15:32:38.255 8013 ssl_write ➡️  56
MBTLS: 2018-10-25T15:32:38.255 8013 f_send ➡️  85
MBTLS: 2018-10-25T15:32:38.251 5783 wait_for_encrypted_data ⌛️
DEBUG: 2018-10-25T15:32:38.255 8013 ➡️  "Host: knb.ecoinformatics.org\r\n" (write)
MBTLS: 2018-10-25T15:32:38.255 8013 ssl_write ➡️  30
MBTLS: 2018-10-25T15:32:38.255 8013 f_send ➡️  59
DEBUG: 2018-10-25T15:32:38.255 8013 ➡️  "Content-Length: 0\r\n" (write)
MBTLS: 2018-10-25T15:32:38.255 8013 ssl_write ➡️  19
MBTLS: 2018-10-25T15:32:38.255 8013 f_send ➡️  48
DEBUG: 2018-10-25T15:32:38.255 8013 ➡️  "\r\n" (write)
MBTLS: 2018-10-25T15:32:38.255 8013 ssl_write ➡️  2
MBTLS: 2018-10-25T15:32:38.255 8013 f_send ➡️  31
MBTLS: 2018-10-25T15:32:38.686 5783 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:38.686 5783 f_recv ⬅️  28
MBTLS: 2018-10-25T15:32:38.686 5783 f_send ➡️  31
MBTLS: 2018-10-25T15:32:38.686 5783 f_recv WANT_READ
MBTLS: 2018-10-25T15:32:38.686 5783 ssl_read ⬅️  -26880 (WANT_READ)
MBTLS: 2018-10-25T15:32:38.686 5783 ssl_read ⌛️ 0
MBTLS: 2018-10-25T15:32:38.686 5783 wait_for_decrypted_data 📥  0
MBTLS: 2018-10-25T15:32:38.686 5783 wait_for_encrypted_data ⌛️
MBTLS: 2018-10-25T15:32:38.864 5783 f_recv ⬅️  5
MBTLS: 2018-10-25T15:32:38.864 5783 f_recv ⬅️  26
MBTLS: 2018-10-25T15:32:38.864 5783 ssl_read ⬅️  -30592
MBTLS: 2018-10-25T15:32:38.864 5783 ssl_abandon 💥
MBTLS: 2018-10-25T15:32:38.87  5783 ssl_read 💥
MBTLS: 2018-10-25T15:32:38.87  5783 ssl_abandon 💥
MBTLS: 2018-10-25T15:32:39.808 8013 eof ⌛️
MBTLS: 2018-10-25T15:32:39.809 8013 close iswritable=false

The code -30592 means fatal message from the server

samoconnor commented 5 years ago

This patch fixes the problem:

diff --git a/src/MbedTLS.jl b/src/MbedTLS.jl
index e12a7fd..60d292d 100644
--- a/src/MbedTLS.jl
+++ b/src/MbedTLS.jl
@@ -133,6 +133,7 @@ function SSLConfig(verify::Bool; log_secrets=nothing)
         MbedTLS.dbg!(conf, tls_dbg)
     end
     MbedTLS.ca_chain!(conf)
+    MbedTLS.ssl_conf_renegotiation!(conf, MbedTLS.MBEDTLS_SSL_RENEGOTIATION_ENABLED)
     conf
 end

diff --git a/src/ssl.jl b/src/ssl.jl
index fa93d20..5279ea9 100644
--- a/src/ssl.jl
+++ b/src/ssl.jl
@@ -468,6 +468,12 @@ function ca_chain!(config::SSLConfig, chain=crt_parse_file(joinpath(dirname(@__F
         config.data, chain.data, C_NULL)
 end

+function ssl_conf_renegotiation!(config::SSLConfig, x)
+    ccall((:mbedtls_ssl_conf_renegotiation, libmbedtls), Cvoid,
+        (Ptr{Cvoid}, Cint),
+        config.data, x)
+end
+
 function own_cert!(config::SSLConfig, cert::CRT, key::PKContext)
     config.cert = cert
     config.key = key
samoconnor commented 5 years ago

However .... https://tls.mbed.org/api/ssl_8h.html#aad4f50fc1c0a018fd5eb18fd9621d0d3

It is recommended to always disable renegotation unless you know you need it and you know what you're doing. In the past, there have been several issues associated with renegotiation or a poor understanding of its properties.

samoconnor commented 5 years ago

It seems that renegotiation is obsoleted in TLS 1.3 : https://ldapwiki.com/wiki/TLS%20Renegotiation

Because of ?: https://www.sslshopper.com/article-ssl-and-tls-renegotiation-vulnerability-discovered.html

oxinabox commented 5 years ago

This affects all DataOne sites, using the DataOne API v2. Do I need to raise an issue against DataOne as well?

samoconnor commented 5 years ago

We could add an option to MbedTLS.jl to allow optional enabling of renegotiation. And, we could try to print out a warning that says "hey, you might need to enabled renegotiation" when we hit const MBEDTLS_ERR_SSL_FATAL_ALERT_MESSAGE=-30592 in ssl_unsafe_read...

I suspect that the renegotiation is somehow related to this API's use of client certificates as a means of authentication. I would be interesting to try the request with a token based auth header and see if that stops the server from trying to renegotiate.

But, yes, it is worth filing an issue with DataOne to say "why are you forcing us to enable a feature that MbedTLS recommends disabling and which seems to be related to a vulnerability?"

samoconnor commented 5 years ago

Also, we probably make the change below to ensure that the exception thrown on the @async monitoring task is not lost. At present, we correctly identify the EOF condition, but we loose the cause.

https://github.com/JuliaWeb/MbedTLS.jl/pull/183

So, if the @async monitoring tasks does a zero-byte read that finds that the server sent a fatal alert, the exception will be rethrown the next time the user calls eof or one of the read functions.

samoconnor commented 5 years ago

With https://github.com/JuliaWeb/MbedTLS.jl/pull/183, instead of just EOFError I get: 

julia> using HTTP; using MbedTLS ; HTTP.request("GET", "https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N"; retry=false, verbose=3, sslconfig=MbedTLS.SSLConfig(false, log_secrets="/Users/sam/stuff/ssl_secrets"))
DEBUG: 2018-10-25T17:40:00.479 073d ➡️  "GET /knb/d1/mn/v2/object/doi:10.5063/F1T43R7N HTTP/1.1\r\n" (write)
DEBUG: 2018-10-25T17:40:00.502 073d ➡️  "Host: knb.ecoinformatics.org\r\n" (write)
DEBUG: 2018-10-25T17:40:00.502 073d ➡️  "Content-Length: 0\r\n" (write)
DEBUG: 2018-10-25T17:40:00.502 073d ➡️  "\r\n" (write)
ERROR: IOError(MbedTLS error code -30592: SSL - A fatal alert message was received from our peer during request(https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N))
samoconnor commented 5 years ago

(Note: I'm using this branch https://github.com/JuliaWeb/MbedTLS.jl/pull/181#issuecomment-432487935 that allows logging TLS secrets to a file for use by wireshark. It is very useful to be able to see what is going on inside a SSL trace!)

samoconnor commented 5 years ago

Ok, so with this PR https://github.com/JuliaWeb/MbedTLS.jl/pull/183/files, we now get an error message that suggests using ssl_conf_renegotiation!:

HTTP.request("GET", "https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N"; retry=false, verbose=3)
[ Info: Recompiling stale cache file /Users/sam/.julia/compiled/v1.0/HTTP/zXWya.ji for HTTP [cd3eb016-35fb-5094-929b-558a96fad6f3]
DEBUG: 2018-10-25T20:13:14.458 787f ➡️  "GET /knb/d1/mn/v2/object/doi:10.5063/F1T43R7N HTTP/1.1\r\n" (write)
DEBUG: 2018-10-25T20:13:14.485 787f ➡️  "Host: knb.ecoinformatics.org\r\n" (write)
DEBUG: 2018-10-25T20:13:14.485 787f ➡️  "Content-Length: 0\r\n" (write)
DEBUG: 2018-10-25T20:13:14.485 787f ➡️  "\r\n" (write)
ERROR: IOError(MbedTLS error code -30592:
SSL - A fatal alert message was received from our peer
(You may need to enable `ssl_conf_renegotiation!`.
See https://github.com/JuliaWeb/HTTP.jl/issues/342#issuecomment-432921180)
during request(https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N))

ssl_conf_renegotiation! can be used like this:

julia> using HTTP

julia> using MbedTLS

julia> conf = MbedTLS.SSLConfig(true)
MbedTLS.SSLConfig()

julia> MbedTLS.ssl_conf_renegotiation!(conf, MbedTLS.MBEDTLS_SSL_RENEGOTIATION_ENABLED)

julia> HTTP.request("GET", "https://knb.ecoinformatics.org/knb/d1/mn/v2/object/doi:10.5063/F1T43R7N";
                    retry=false, verbose=3, sslconfig=conf)

DEBUG: 2018-10-25T20:15:11.752 7fcf ➡️  "GET /knb/d1/mn/v2/object/doi:10.5063/F1T43R7N HTTP/1.1\r\n" (write)
DEBUG: 2018-10-25T20:15:11.781 7fcf ➡️  "Host: knb.ecoinformatics.org\r\n" (write)
DEBUG: 2018-10-25T20:15:11.782 7fcf ➡️  "Content-Length: 0\r\n" (write)
DEBUG: 2018-10-25T20:15:11.782 7fcf ➡️  "\r\n" (write)
DEBUG: 2018-10-25T20:15:13.438 7fcf ⬅️  "HTTP/1.1 200 OK\r\n" (readavailable)
DEBUG:                                 "Date: Thu, 25 Oct 2018 09:15:11 GMT\r\n"
DEBUG:                                 "Server: Apache/2.4.7 (Ubuntu)\r\n"
DEBUG:                                 "Vary: User-Agent\r\n"
DEBUG:                                 "Set-Cookie: JSESSIONID=F33582CD4B269DE2ED92A5710D85E2AC; Path=/knb/; Secure\r\n"
DEBUG:                                 "Content-Disposition: inline; filename=\"doi:10.5063/F1T43R7N.xml\"\r\n"
DEBUG:                                 "X-Frame-Options: sameorigin\r\n"
DEBUG:                                 "Access-Control-Allow-Origin: \r\n"
DEBUG:                                 "Access-Control-Allow-Headers: Authorization, Content-Type, Origin, Cache-Control\r\n"
DEBUG:                                 "Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS\r\n"
DEBUG:                                 "Access-Control-Allow-Credentials: true\r\n"
DEBUG:                                 "Transfer-Encoding: chunked\r\n"
DEBUG:                                 "Content-Type: text/xml\r\n"
DEBUG:                                 "\r\n"
DEBUG:                                 "1ff8\r\n"
DEBUG: 2018-10-25T20:15:13.465 7fcf ♻️  "1ff8\r\n" (unread!)
DEBUG: 2018-10-25T20:15:13.654 7fcf ⬅️  "1ff8\r\n" (readavailable)
DEBUG: 2018-10-25T20:15:13.655 7fcf ⬅️  "<?xml version=\"1.0\" encoding=\"UTF-8\"?> ...
oxinabox commented 5 years ago

Transcribing this from DataOne's Slack

@oxinabox [10:28 AM]

re: https://twitter.com/DataONEorg/status/1057335092912312320

@oxinabox_frames We've been talking about this in DataONE. It'd be great if you could join our slack channel and connect with the dev team :) https://slack.dataone.org/

which is re:

@DataONEorg use of TLS Renegotiation in the API, is problematic. MbedTLS recommends against accepting it as there are related vulnerabilities. But it is required to access the API for KNB (@nceas) and @arcticdatactr and I believe all DataOne sites?

which is in regards to https://github.com/JuliaWeb/HTTP.jl/issues/342

@davev [12:29 PM]

Can you provide more information on the MbedTLS recommendation? wrt CVE-2009-3555, insecure renegotiation was apparently addressed by a patch at OpenSSL 0.9.8m

@oxinabox [8:46 AM]

I can't but if you post on the HTTP.jl github issue I linked above Sam Connor probably can.

342

@bryce [9:07 AM]

Hrm, this seems related to the Apache config overrides we apply for web browser user agents (to avoid verifying client certs) and API clients (to allow POST requests above the TLS reneg limits). This works because we SSLClientVerify None httr as a UA:

julia> HTTP.setuseragent!("httr")
julia> x = HTTP.request("GET", "https://dev.nceas.ucsb.edu/knb/d1/mn/v2/object/00005f4b-f1ff-40a4

@metamattj [5:02 PM]

yeah, we’ve been moving towards limiting the need for SSLClientVerify to only clients that are likely to use X.509 certs because of the Safari bug in reneg

@davev [5:04 PM]

Also note that TLS1.3 does not allow renegotiation at all. It's not clear yet how Apache is supporting optional SSLClientVerify with TLS1.3

@chris_dataone [5:05 PM]

So, the one immediate workaround for @oxinabox that I can see is to just set the user agent string to a known browser string like chrome or the like. This will cause Apache to not enter the renegotiation. The issue is that Apache is forcing renegotiation even though our SSLVerifyClient optional directive is not in a Location nor Directory context. I can’t figure out why it is doing that.

@metamattj [5:06 PM]

but that totally depends on our current config

@chris_dataone [5:06 PM]

What do you mean?

@metamattj [5:07 PM]

it relies on the fact that we are not allowinfg client certs in safari and httr and other specific situations (edited) which very well might change (edited) our problem is that reneg is only partially working, and not at all in Safari

@oxinabox [5:08 PM]

Why are client certs even useful in this case? What is the security story?

@chris_dataone [5:08 PM]

It does. This would all be fixed if WebKit folks would address the underlying Safari cert mishandling issue

@metamattj [5:08 PM]

yep. although we are also having the issues in R, which is why we have the httr agent filter too so now R only uses the JWT tokens for auth and it has been great!

@davev [5:09 PM]

does R use openssl?

@chris_dataone [5:09 PM]

@oxinabox - for the DataONE API, we use X509 certificates for Member Node to Member Node and Coordinating Node to Memeber Node communication

@metamattj [5:10 PM]

R can use openssl. there are a bunch of http libs that use different combos of underlying security libs I forget what httr uses but I could look its also a rapidly changing landscape there on the http client front

@davev [5:12 PM]

I suspect ssl libs all understandably set renegotiation to False by default after CVE-2009-3555

@metamattj [5:12 PM]

I remember now — httr uses libcurl, which uses the underlying system C lib for each platform

@davev [5:12 PM]

There's more than likely a way to enable it, which will work until webservers support TLS1.3

@metamattj [5:14 PM]

I’m sure there is, but we’ve been very happy to not use the client X.09 certs, which nobody in the R community really ever grokked; the JWT tokens have been much better, and no longer require any advanced SSL support

@davev [5:15 PM]

yeah, jwt is generally a much better solution

@metamattj [5:15 PM]

if we had the oath refresh mechanism working it would be even better :slightly_smiling_face: