CSRF input randomly disappears

da1nerd commented 3 years ago

Describe the bug

I have a lucky app running in a docker container on Digital Ocean. Every once in awhile the csrf input field disappears. The result is a 403 error if you try to submit the form. This has happened multiple times, and each time it is easily resolved by restarting the container.

To Reproduce Steps to reproduce the behavior: unknown

I'm collecting a bunch of telemetry using DataDog and have a test that tries to log into the website every hour from three different regions within the U.S on chrome and firefox. This is currently how I am monitoring the problem so I can restarted the sever before it impacts users.

Versions (please complete the following information):

Lucky version (check in shard.lock): 0.25.0
Crystal version (crystal --version): 0.35.1
Docker Image: alpine
OS: Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-65-generic x86_64)

Additional context I don't know if the two csrf meta tags are also missing, because I didn't know they existed in the first place.

The error might occur within a few days to a couple of weeks after the last deployment.

This last incident occurred on March 15th, 16 days after the last deployment.
The previous incident occurred Feb 18th, 2 days after the last deployment.

System metrics The docker image is deployed on Digital Ocean with 2GB memory and 25 GB disk on Ubuntu 20.04.2 On the host: memory remains at a constant 60%, CPU hangs out at below 5%, and disk I/O is in the kB/s range. The only anomaly at the time of failure is a small spike in network traffic (300kB vs the usual 122 kB).

I assume the spike in CPU is related to the increased network traffic.

The actual docker container has no anomalies in CPU, memory, disk I/O, or network I/O.

Dockerfile Because the error occurs randomly it's possible it is related to some service or package on the container or host that is out of date. Here's my docker file for context. Compilation all happens in docker containers so result is reproducible.

# STAGE 1: Use node env to build node dependencies
FROM node:alpine as node_build
WORKDIR /tmp_build

# Install node packages in image
COPY package.json .
COPY yarn.lock .
RUN yarn install --no-progress --frozen-lockfile

# Compile assets with webpack
COPY public ./public
COPY webpack.mix.js .
COPY src ./src
RUN yarn prod

# STAGE 2: Use crystal env to build crystal dependencies
FROM crystallang/crystal:0.35.1-alpine as lucky_build
ENV SKIP_LUCKY_TASK_PRECOMPILATION="1"
WORKDIR /tmp_build
COPY shard.* ./
RUN  shards install --production
COPY . .
COPY --from=node_build /tmp_build/public/mix-manifest.json public/mix-manifest.json
RUN crystal build --static src/start_server.cr
RUN crystal build --static tasks.cr -o run_task

# FINAL: actual image with app binary and static assets
FROM alpine
RUN addgroup -g 1000 -S lucky && \
    adduser -u 1000 -S lucky -G lucky
WORKDIR /home/lucky/app

COPY --chown=lucky:lucky --from=node_build /tmp_build/public public
COPY --chown=lucky:lucky --from=lucky_build /tmp_build/start_server start_server
COPY --chown=lucky:lucky --from=lucky_build /tmp_build/run_task run_task

RUN chown -R lucky /home/lucky
USER lucky

CMD ["/home/lucky/app/start_server"]

da1nerd commented 3 years ago

Some additional context is that a restart was required on the host, there were 41 pending updates. I don't remember exactly, but I probably installed all the available updates during the last incident. I am updating everything now at any rate.

These are the packages that need to be updated (in case they shed any light).

bind9-dnsutils/focal-updates 1:9.16.1-0ubuntu2.7 amd64 [upgradable from: 1:9.16.1-0ubuntu2.6]
bind9-host/focal-updates 1:9.16.1-0ubuntu2.7 amd64 [upgradable from: 1:9.16.1-0ubuntu2.6]
bind9-libs/focal-updates 1:9.16.1-0ubuntu2.7 amd64 [upgradable from: 1:9.16.1-0ubuntu2.6]
containerd.io/focal 1.4.4-1 amd64 [upgradable from: 1.4.3-1]
datadog-agent/unknown 1:7.26.0-1 amd64 [upgradable from: 1:7.25.1-1]
dirmngr/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
docker-ce-cli/focal 5:20.10.5~3-0~ubuntu-focal amd64 [upgradable from: 5:20.10.3~3-0~ubuntu-focal]
docker-ce-rootless-extras/focal 5:20.10.5~3-0~ubuntu-focal amd64 [upgradable from: 5:20.10.3~3-0~ubuntu-focal]
docker-ce/focal 5:20.10.5~3-0~ubuntu-focal amd64 [upgradable from: 5:20.10.3~3-0~ubuntu-focal]
friendly-recovery/focal-updates 0.2.41ubuntu0.20.04.1 all [upgradable from: 0.2.41]
gnupg-l10n/focal-updates 2.2.19-3ubuntu2.1 all [upgradable from: 2.2.19-3ubuntu2]
gnupg-utils/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
gnupg2/focal-updates 2.2.19-3ubuntu2.1 all [upgradable from: 2.2.19-3ubuntu2]
gnupg/focal-updates 2.2.19-3ubuntu2.1 all [upgradable from: 2.2.19-3ubuntu2]
gpg-agent/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
gpg-wks-client/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
gpg-wks-server/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
gpg/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
gpgconf/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
gpgsm/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
gpgv/focal-updates 2.2.19-3ubuntu2.1 amd64 [upgradable from: 2.2.19-3ubuntu2]
grub-common/focal-updates 2.04-1ubuntu26.9 amd64 [upgradable from: 2.04-1ubuntu26.8]
grub-efi-amd64-bin/focal-updates 2.04-1ubuntu26.9 amd64 [upgradable from: 2.04-1ubuntu26.8]
grub-efi-amd64-signed/focal-updates 1.142.11+2.04-1ubuntu26.9 amd64 [upgradable from: 1.142.10+2.04-1ubuntu26.8]
grub-pc-bin/focal-updates 2.04-1ubuntu26.9 amd64 [upgradable from: 2.04-1ubuntu26.8]
grub-pc/focal-updates 2.04-1ubuntu26.9 amd64 [upgradable from: 2.04-1ubuntu26.8]
grub2-common/focal-updates 2.04-1ubuntu26.9 amd64 [upgradable from: 2.04-1ubuntu26.8]
linux-firmware/focal-updates 1.187.10 all [upgradable from: 1.187.9]
linux-headers-generic/focal-updates,focal-security 5.4.0.67.70 amd64 [upgradable from: 5.4.0.66.69]
linux-headers-virtual/focal-updates,focal-security 5.4.0.67.70 amd64 [upgradable from: 5.4.0.66.69]
linux-image-extra-virtual/focal-updates,focal-security 5.4.0.67.70 amd64 [upgradable from: 5.4.0.66.69]
linux-image-generic/focal-updates,focal-security 5.4.0.67.70 amd64 [upgradable from: 5.4.0.66.69]
linux-image-virtual/focal-updates,focal-security 5.4.0.67.70 amd64 [upgradable from: 5.4.0.66.69]
linux-virtual/focal-updates,focal-security 5.4.0.67.70 amd64 [upgradable from: 5.4.0.66.69]
pollinate/focal-updates 4.33-3ubuntu1.20.04.1 all [upgradable from: 4.33-3ubuntu1]
python3-software-properties/focal-updates 0.98.9.4 all [upgradable from: 0.98.9.3]
python3-twisted-bin/focal-updates 18.9.0-11ubuntu0.20.04.1 amd64 [upgradable from: 18.9.0-11]
python3-twisted/focal-updates 18.9.0-11ubuntu0.20.04.1 all [upgradable from: 18.9.0-11]
python3-update-manager/focal-updates 1:20.04.10.6 all [upgradable from: 1:20.04.10.5]
software-properties-common/focal-updates 0.98.9.4 all [upgradable from: 0.98.9.3]
update-manager-core/focal-updates 1:20.04.10.6 all [upgradable from: 1:20.04.10.5]

matthewmcgarvey commented 3 years ago

Getting a feel for the context of the issue:

luckyframework / lucky

CSRF input randomly disappears #1437