starlightprivate / flash2

nodejs app
0 stars 0 forks source link

Fix strange issues on heroku #274

Closed vodolaz095 closed 7 years ago

vodolaz095 commented 7 years ago

There was quite strange issues reported in sentry https://sentry.io/starlight-group/node-api/issues/245313329/

Corresponding loggly entry

{"buildId":"heroku","type":"http:error","env":"production","ip":"190.167.69.15","method":"GET","entryPoint":"/tacticalsales/assets/js/libs.js","path":"/tacticalsales/assets/js/libs.js","query":{},"body":{},"sessionId":"PGlmf4LOJRZYDgSJgvKDt93wcLAegP7L","userAgent":"Mozilla/5.0 (Linux; Android 5.1.1; SAMSUNG-SM-N910A Build/LMY47X; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/56.0.2924.87 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/117.0.0.18.47;]","message":" expressjs error : OOM command not allowed when used memory > 'maxmemory'.","stacktrace":"Error\n at HiredisReplyParser.returnError (/app/node_modules/ioredis/lib/redis/parser.js:25:25)\n at HiredisReplyParser.execute (/app/node_modules/ioredis/node_modules/redis-parser/lib/hiredis.js:28:18)\n at Socket.<anonymous> (/app/node_modules/ioredis/lib/redis/event_handler.js:107:22)\n at emitOne (events.js:96:13)\n at Socket.emit (events.js:191:7)\n at readableAddChunk (_stream_readable.js:178:18)\n at Socket.Readable.push (_stream_readable.js:136:10)\n at TCP.onread (net.js:559:20)","level":"error"}

It turned out that heroku redis database have ran out of memory. I connected to it via redis-cli, and see

ec2-34-206-56-13.compute-1.amazonaws.com:9309> info
# Server
redis_version:3.2.8
redis_git_sha1:db8a945c
redis_git_dirty:0
redis_build_id:49bd10fc73af440c
redis_mode:standalone
os:Linux 4.4.0-64-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.4
process_id:5
run_id:c611a60df01e9bf5d9470a8899613afb09483efe
tcp_port:9309
uptime_in_seconds:1870141
uptime_in_days:21
hz:10
lru_clock:14651031
executable:/usr/bin/redis-server
config_file:/etc/redis/heroku-redis.conf

# Clients
connected_clients:2
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:26230408
used_memory_human:25.02M
used_memory_rss:32620544
used_memory_rss_human:31.11M
used_memory_peak:30871808
used_memory_peak_human:29.44M
total_system_memory:16040235008
total_system_memory_human:14.94G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:26214400
maxmemory_human:25.00M
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.24
mem_allocator:jemalloc-4.0.3

# Persistence
loading:0
rdb_changes_since_last_save:80922
rdb_bgsave_in_progress:0
rdb_last_save_time:1489175898
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

# Stats
total_connections_received:54940
total_commands_processed:337984
instantaneous_ops_per_sec:0
total_net_input_bytes:59396472
total_net_output_bytes:121668259
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:9245
evicted_keys:0
keyspace_hits:29741
keyspace_misses:413
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:434.66
used_cpu_user:289.74
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=40639,expires=40639,avg_ttl=46614369

This lines explains it all: used_memory_human:25.02M used_memory_lua_human:37.00K maxmemory_human:25.00M

I prayed to Sergius_of_Radonezh, and started research.

It turned out, that the database was cluttered with initialized, but unused entries of sessions. I researched site, and found, that PHPSESSID cooke was not set. So express-session created all data in redis on each request, but user have not received the cookie with session, and have not used it. But site was working, because mobile browses usually do not use cookies.

I researched more, and found, that in cookie enabled browsers, the PHPSESSID cookie was not set.

In code the PHPSESSID cookie is secured one, it is used when application is working via HTTPS, not HTTP.

And if application is used via HTTP, the cookie is not send to user.

Previously, we had similiar issues, when cloudflare was working with heroku via HTTP, not HTTPS - because there was some issues with SSL certs installed in application in heroku settings by @asharma-ror .

Probably it have repeated - https://dashboard.heroku.com/apps/flash2dev/settings.

As half measures, i made cookie of PHPSESSID to be send as not secure, until @asharma-ror fix cloudflare issues with interaction with heroku via SSL


This change is Reviewable

vodolaz095 commented 7 years ago

i mean currently the code deployed on heroky is by pushes from my local PC. And it works We need either

  1. merge this PR to make clean deploy. OR

  2. fix cloudflare

  3. uncomment line with secure: isProtectedByCloudflare for cookies and add extra push to this PR.

merge this PR.

vodolaz095 commented 7 years ago

i tried to mitigate this issue in this way:

[vodolaz095@ivory ~]$ curl -v http://tacticalmastery.com/
*   Trying 104.20.194.45...
* Connected to tacticalmastery.com (104.20.194.45) port 80 (#0)
> GET / HTTP/1.1
> Host: tacticalmastery.com
> User-Agent: curl/7.47.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 01 Apr 2017 13:02:13 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=d40d90deb12dd4f0b845b9cad42c9707f1491051733; expires=Sun, 01-Apr-18 13:02:13 GMT; path=/; domain=.tacticalmastery.com; HttpOnly
< Cache-Control: max-age=3600
< Expires: Sat, 01 Apr 2017 14:02:13 GMT
< Location: https://tacticalmastery.com/
< X-Content-Type-Options: nosniff
< Server: cloudflare-nginx
< CF-RAY: 348bbdd394884e24-DME
< 
* Connection #0 to host tacticalmastery.com left intact

when uses opens site via HTTP (not S), he is instantly redirected to HTTPS version of site, without any important cookies (!!!!) being send via HTTP protocol