Insufficient handling of possible valid Accept headers

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Parsing of the Mime header misses some legitimate accept headers, namely 
quoted tokens which can contain commas, spaces, escaped quotes, and semi-colons.

I've written an implementation in ruby that handles all cases with some speed 
optimizations for the common case where quoted tokens do not occur. I'm just 
going to paste it here in-line instead of attempting to patch every version in 
this library.

ACCEPT_HEADER_REGEXP = /(\S+="(?:\\"|.)*?"[^\s,]*|\S+=[^\s,]+|[^\s;,]+[^\s,]*)/
ACCEPT_PARAMETER_REGEXP = /([^\s;=,]+)=("(?:\\"|.)*?"|[^;]+)/

def parse(header)
  # Only use the complex regexp if the header contains quoted tokens
  formats = if header.index('"')
    header.scan(ACCEPT_HEADER_REGEXP).map(&:first)
  else
    header.split(/,\s*/)
  end

  formats.map { |format|
    # Set default quality
    params = {'q' => 1.0}

    # Split into type and following parameters
    type, accept_params = format.split(";", 2)

    # Correct a standard wildcard error
    type = "*/*" if type == "*"

    if accept_params
      # Only use a complex regexp if the parameters contain quoted tokens
      accept_params = if accept_params.index('"')
        accept_params.scan(ACCEPT_PARAMETER_REGEXP)
      else
        accept_params.split(";").map { |a| a.split("=") }
      end

      accept_params.each { |(key, val)|
        val = if key == 'q'
          val.to_f
        elsif val[0] == '"' and val[-1] == '"'
          val[1..-2].gsub(/\\(.)/, "\\1")
        else
          val
        end
        params[key] = val
      }
    end
    [*type.split("/"), params]
  }
end

Original issue reported on code.google.com by kel...@bigcartel.com on 28 Mar 2014 at 3:34