Buffer inconsistent throws/coercion

[trevnorris]

Working on #4964 and noticed that Buffers have inconsistencies of when arguments are thrown and when they're coerced. This is a proposal to solidify how they should be handled. It has taken care to follow existing practices by browsers in regards to generic argument handling and that of the Typed Array specification:

• Required numeric arguments throw if undefined.
• Optional numeric arguments are set as default if undefined.
• Arguments expected to be numeric will coerce all other values.
• Any double will be floored to int.
• Throw for all out of range indexes (index > buffer.length || index < 0) for any read/write methods.
• buffer.slice() and buffer.copy() should follow the specification for arraybuffer.slice() (with a few modification for .copy()).

As an example of how to handle this, here are a couple verification functions:

inline uint32_t GetOptionalUintArg(Handle<Value> arg) {
  int32_t val_i = arg->Int32Value();
  if (val_i < 0)
    return ThrowRangeError("index is less than zero");
  return static_cast<uint32_t>(val_i);
}

inline uint32_t GetRequiredUintArg(Handle<Value> arg) {
  if (arg->IsUndefined())
    return ThrowTypeError("argument is undefined");
  return GetOptionalUintArg(arg);
}

[bnoordhuis]

I second this motion.

Arguments expected to be numeric will coerce all other values.

My beef with this would be that it could hide silly type errors in core, i.e. that it could mask expensive type coercions. Not insurmountable: if all type checking logic is centralized, I can always patch my tree to throw instead of coerce and run the tests (and hope they excise that particular behavior.)

[isaacs]

My beef with this would be that it could hide silly type errors in core

I'm inclined to agree with this. Where would we be coercing to a numeric type?

[trevnorris]

@bnoordhuis I feel the same way. My take on it is that type coercion in js is usually cheap, and if an unexpected value goes through it'll show up as a DEOPT while testing. I understand this isn't optimal, but can't think of a much better way.

For cc-land, not completely sure. The best idea I've had is to create a global set of argument parsing functions, like the two above, which would make it easy to patch for testing.

@isaacs For any functions that users aren't supposed to touch, no need for the coercion. It's geared towards the public API. And it's not just for numbers specifically (though I forgot to mention that). It's for any primitive. Remember 0468861? It's pretty standard to auto-coerce any expected primitive argument to be what is required.

Here's an example:

var dv = new DataView(new ArrayBuffer(8));
dv.setFloat64('0','1.1');
dv.getFloat64('0');
// output: 1.1

[trevnorris]

These are easy primitive coercion methods that follow ECMA 262.

• !! - convert value to Boolean (sec. 9.2)
• + - convert value to Number (sec. 9.3)
• ~~ - convert value to Int32 (sec. 9.5)
• +'' - convert value to string (sec. 9.8)

Here is an example using several of the above:

// use: fillRange(val, [start][, end][, noAssert]);
function fillRange(val, start, end, noAssert) {
  val = +val;
  // null-ish will default to 0
  start = ~~start;
  // null-ish will default to "data_length"
  end = end == null ? data_length : ~~end;

  // js api's generally fix this for developers
  if (end > start)
    end = start;

  // important to throw for index check when js is trying to
  // access raw memory (e.g. Buffers/Typed Arrays)
  if (!noAssert && (start < 0 || end > total_length))
    throw new RangeError("out of range index");

  // ...
}

Now as far as the c++ side, definitely want @bnoordhuis opinion on that. I'd like to be able to replicate the above for consistency, but value coercion is much more expensive on the native side. What about making a few argument value parsing functions included in node.h. Possibly setting a #define for debug mode so it'll throw instead of auto-coercing the values?

[bnoordhuis]

What about making a few argument value parsing functions included in node.h. Possibly setting a #define for debug mode so it'll throw instead of auto-coercing the values?

I'm okay with that but said functions should go into node_internals.h, not node.h.

[trevnorris]

Sounds good to me. Other than that, have any comments on any other arguments that should be thrown?

[trevnorris]

@isaacs Want your opinion on one thing. Right now the Buffer .slice() will coerce all arguments. This sort of make sense because it follows the same pattern as Array. Though since the user isn't just getting a copy back, imho, out of range indexes should throw. So, change tha API, or make an exception for that case?

[trevnorris]

Ok. Consensus has been reached.

General:

• Expected arguments throw if value === undefined.
• arguments.length is too expensive (in js) and so not used to check.
• Optional arguments are set to their defaults if === undefined.
• Primitives are coerced before value validation occurs. (e.g. length = +length)

General Numeric rules:

• Expected Number values are coerced:
  • JS with +arg.
  • CC with NumberValue(arg).
• Expected Int/Uint values are coerced:
  • JS with ~~arg.
  • CC with Int32Value(arg)
• When Int/Uint are expected values are allowed to overflow wrap, but coercion must be done before checking.
• Uint values are first coerced to Int to check for negative values. Then cast to size_t.

Other general rules:

• String arguments are coerced using + ''.
• Boolean arguments are coerced using !!.

Exception:

• Buffer instantiation expects Uint but evaluates as the floor of Number.

General Buffer method rules:

• If end > start then end = start.
• All write*/read* will throw if index (plus offset for written value) is out of range.
• Buffer.prototype.slice will behave like ArrayBuffer.prototype.slice.

One TBD:

• Buffer.prototype.copy is problematic for the following:
  • I was an idiot and added the following to the docs: "All values passed that are undefined/NaN or are out of bounds are set equal to their respective defaults."
  • Currently throws when "sourceEnd is negative", "sourceStart is greater than sourceEnd" or "attempt to copy after targetBuffer". Which could contradict the "out of bounds" line above. Depending on how you read it.

Other notes:

• In cc-land the function ParseArrayIndex has been added to node_internals.h used to parse all array indexes in cc.

Have a hard time believing it's come out this complicated, but this should address most, if not all, argument parsing concerns.

/cc @isaacs @bnoordhuis just for one last review

[bnoordhuis]

I'm not sure if your comment addresses this but:

Expected Int/Uint values are coerced:

Can lead to unexpected results (where 'unexpected' === 'most people won't realize that', not 'undefined'.)

> 0x7fffffff === ~~0x7fffffff  // 2147483647
true
> 0x80000000 === ~~0x80000000  // -2147483648
false

This is more of a general observation because you can't have buffers that big anyway.

[trevnorris]

Was thinking about that myself, but then realized that no one else does it as well:

new ArrayBuffer(~~(0xffffffff+3))

[trevnorris]

But I did make an exception for Buffer instantiation. I floor the Number which will leave it in floating point range. So in that one case it'll throw regardless of the size of n.

[domenic]

Sorry to jump in here, but

Optional arguments are set to their defaults if == null.

Is not really the way to go. It should be === undefined to match ES6 semantics of default arguments and all built-in ES5 functions.

Otherwise the general movement toward coercion is great and matches the ES5 built-ins.

[trevnorris]

Good assessment. There are parts of the API where null is a valid but not default parameter. Oversight on my part. Thanks.

[trevnorris]

@isaacs I'd suggest during v0.13 cleanup we make this standard across all of core as much as possible.

[jasnell]

@trevnorris ... is this still an issue?

[jasnell]

@trevnorris ... ping :-)

[trevnorris]

going to say no.