Investigate image optimization server 500 errors

blushi commented 3 years ago

Looking at server logs, it seems like there might be some memory limit issue:

2021-05-18T16:53:11.105590+00:00 heroku[web.1]: Process running mem=823M(160.8%)
2021-05-18T16:53:11.159303+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2021-05-18T16:53:14.000000+00:00 app[heroku-redis]: source=REDIS addon=redis-regular-00984 sample#active-connections=2 sample#load-avg-1m=0.39 sample#load-avg-5m=0.33 sample#load-avg-15m=0.29 sample#read-iops=0 sample#write-iops=0.15481 sample#memory-total=15628988kB sample#memory-free=8084132kB sample#memory-cached=3444692kB sample#memory-redis=26236752bytes sample#hit-rate=0.6881 sample#evicted-keys=0
2021-05-18T16:53:14.000000+00:00 app[heroku-redis]: source=REDIS_URL addon=redis-regular-00984 sample#used_memory_over_limit=2643792.0bytes message=Database memory close to the limit. Please upgrade your database plan or change your maxmemory policy.

cc/ @mhagel

Cause this in Safari: https://regen-network.slack.com/files/UBEFH4RT3/F0230FRUELS/image.png

mhagel commented 3 years ago

2021-05-25T19:46:10.422035+00:00 app[web.1]: lerna info Executing command in 1 package: "npm run start"

2021-05-25T19:46:10.715900+00:00 app[web.1]: registry-server: > registry-server@0.1.0 start

2021-05-25T19:46:10.716005+00:00 app[web.1]: registry-server: > ts-node Server.ts

2021-05-25T19:46:19.815260+00:00 app[web.1]: registry-server: TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined

2021-05-25T19:46:19.815688+00:00 app[web.1]: registry-server:     at new NodeError (node:internal/errors:363:5)

2021-05-25T19:46:19.815750+00:00 app[web.1]: registry-server:     at validateString (node:internal/validators:119:11)

2021-05-25T19:46:19.815798+00:00 app[web.1]: registry-server:     at Module.require (node:internal/modules/cjs/loader:1006:3)

2021-05-25T19:46:19.815835+00:00 app[web.1]: registry-server:     at require (node:internal/modules/cjs/helpers:93:18)

2021-05-25T19:46:19.815982+00:00 app[web.1]: registry-server:     at loadStore (/app/server/node_modules/keyv/src/index.js:18:15)

2021-05-25T19:46:19.816040+00:00 app[web.1]: registry-server:     at new Keyv (/app/server/node_modules/keyv/src/index.js:39:22)

2021-05-25T19:46:19.816088+00:00 app[web.1]: registry-server:     at Object.imageOptimizer [as default] (/app/server/middleware/imageOptimizer.ts:11:18)

2021-05-25T19:46:19.816129+00:00 app[web.1]: registry-server:     at Object.<anonymous> (/app/server/Server.ts:51:33)

2021-05-25T19:46:19.816219+00:00 app[web.1]: registry-server:     at Module._compile (node:internal/modules/cjs/loader:1109:14)

2021-05-25T19:46:19.816368+00:00 app[web.1]: registry-server:     at Module.m._compile (/app/server/node_modules/ts-node/src/index.ts:1056:23)

mhagel commented 3 years ago

Cleared the Redis cache. Seemed to fix it temporarily, but we need to do 3 things:

Implement a fallback image on the Impact cards (or just use unoptimized images)
Test and upgrade heroku-redis to a higher plan. I tried to do this, but it broke things - needs more attention.
Make server code less brittle. (When I tried to upgrade, registry-server couldn't find the new REDIS_URL variable. This caused the code to fail, and the whole server was down.

mhagel commented 3 years ago

new issue: https://app.zenhub.com/workspaces/regen-registry-5f8998bec8958d000f4609e2/issues/regen-network/regen-registry/518

blushi commented 3 years ago

Test and upgrade heroku-redis to a higher plan. I tried to do this, but it broke things - needs more attention.

Could you contact @aaronc about this?

regen-network / rnd-dev-team

Investigate image optimization server 500 errors #502