COSE::Key.deserialize can throw NoMemoryError

[lgarron]

If you run COSE::Key.deserialize(SecureRandom.random_bytes(64)), then there is a fairly high chance of a NoMemoryError. I think it would be nice if a compact untrusted input always throws a recoverable error.

The following repro code crashes for me pretty reliably:

require "cose"
require 'securerandom'

errors = {}

100.times do |i|
  begin
    COSE::Key.deserialize(SecureRandom.random_bytes(64))
  rescue => ex
    errors[ex.class.name] = ex
  end
end

puts errors

Sample output:

ruby(49760,0x108de85c0) malloc: can't allocate region
*** mach_vm_map(size=597202949621485568) failed (error code=3)
ruby(49760,0x108de85c0) malloc: *** set a breakpoint in malloc_error_break to debug
ruby(49760,0x108de85c0) malloc: can't allocate region
*** mach_vm_map(size=597202949621485568) failed (error code=3)
ruby(49760,0x108de85c0) malloc: *** set a breakpoint in malloc_error_break to debug
/Users/lgarron/.data/gem/gems/cose-0.7.0/lib/cose/key.rb:24:in `decode': failed to allocate memory (NoMemoryError)
  from /Users/lgarron/.data/gem/gems/cose-0.7.0/lib/cose/key.rb:24:in `deserialize'
  from registrations.rb:8:in `block in <main>'
  from registrations.rb:6:in `times'
  from registrations.rb:6:in `<main>'

[lgarron]

After digging into this a bit more, looks like this is all in the CBOR decoding (e.g. https://github.com/cabo/cbor-ruby/issues/11).

This might be a hard problem to solve safely for the general case of CBOR decoding, but it would be nice if trying to decode limited-format public keys was more predictable and resource-constrained.

[lgarron]

FWIW, based 100 million test runs, it seems the errors that COSE::Key.deserialize can throw include at least:

• CBOR::MalformedFormatError
• TypeError
• EOFError
• COSE::UnknownKeyType
• ArgumentError
• NoMemoryError
• URI::InvalidURIError
• NoMethodError
• IndexError
• RegexpError
• FloatDomainError

[grzuy]

Hi @lgarron,

Thanks for reporting and identifying the issue in cbor. Can't reproduce any more after updating cbor gem to 0.5.9.6.