Fix issue #561 (missing check for long-long signed overflow)

[andrew-appel]

and also, fix a bug with extendM that was inadvertently introduced in P.R. #555.

closes #561