search

mozilla / fxa-auth-server

DEPRECATED - Migrated to https://github.com/mozilla/fxa
Mozilla Public License 2.0
399 stars 121 forks source link

Investigate scrypt malloc error #2972

Closed philbooth closed 5 years ago

philbooth commented 5 years ago

With node 10, we migrated to node's built-in implementation of scrypt, which requires you to specify the upper limit for memory available to the scrypt operation. We store ours in a const called DEFAULT_MAXMEM in lib/crypto/scrypt.js.

This error from Sentry seems to indicate that the limit we calculate for that is not sufficient on occasion:

https://sentry.prod.mozaws.net/operations/auth-prod/issues/5373526/?environment=prod

philbooth commented 5 years ago

Correction, we calculate ours according to the values for N and r, here:

let maxmem = DEFAULT_MAXMEM
if (N > DEFAULT_N || r > DEFAULT_R) {
  // Conservatively prevent `memory limit exceeded` errors. See the docs for more info:
  // https://nodejs.org/api/crypto.html#crypto_crypto_scrypt_password_salt_keylen_options_callback
  maxmem = MAXMEM_MULTIPLIER * (N || DEFAULT_N) * (r || DEFAULT_R)
}
philbooth commented 5 years ago

Actually, the linked error seems to imply that malloc failed, not that we hit our own limit. So maybe the limit calculation here is okay and the instance ran out of physcial resources instead.

jrgm commented 5 years ago

Closing. Root cause was likely a severe memory leak in systemd-journald now fixed by reverting to previous AMI images. - https://github.com/mozilla-services/cloudops-deployment/pull/3023