Create Dockerfile for EKS Onboarding Process with Ops

olivereri commented 2 years ago

Description

As a CMS Devops Engineer, I need to create a Dockerfile for CMS, so that I can define container image with Drupal and customizations installed for deployment on EKS.

Acceptance Criteria

[x] Create a Dockerfile that can generate a container image with all the necessary software and dependencies to run the content management system. Enough to get through the onboarding process.
[x] Image is successfully built from Dockerfile.

Implementation Details

An example of what this may look like can be found here: https://github.com/department-of-veterans-affairs/va.gov-cms/blob/master/.tugboat/config.yml

olivereri commented 2 years ago

Hey team! Please add your planning poker estimate with ZenHub @ElijahLynn @indytechcook @ndouglas

ElijahLynn commented 2 years ago

We should look at Geerling guys container images like Drupal Container.

ElijahLynn commented 2 years ago

Also possibly Lagoon's containers.

https://github.com/uselagoon/lagoon

ndouglas commented 2 years ago

I'm not doing anything else this afternoon, so I figured I'd pop on and share my opinions.

I don't think Jeff's Dockerfile is particularly helpful. It seems really barebones to me.
I don't think Lagoon's is either. I've done it before, but I don't particularly like mixing Drupal with Nginx because the Drupal project assumes Apache by default and it becomes extra effort to duplicate and then maintain the functionality of .htaccess, mod_rewrite, etc within Nginx. Lagoon might have an Apache-based image I'm missing, or there might be issues with my dismissal of Nginx in this context 🤷

ndouglas commented 2 years ago

I was on the subject so I thought I'd take a quick swing at this:

FROM php:7.3-apache AS cms_build

ENV CMS_REPO_ROOT="/var/www/cms"
ENV CMS_DOCROOT="$CMS_REPO_ROOT/docroot"
ENV PHP_VERSION="7.3"

# Copy Composer into container rather than installing it.
COPY --from=composer:2.1.8 /usr/bin/composer /usr/local/bin/composer

# php-extension-installer makes working with PHP modules a lot easier
COPY --from=mlocati/php-extension-installer /usr/bin/install-php-extensions /usr/local/bin/

# Install VA internal certificate.
ADD http://crl.pki.va.gov/PKI/AIA/VA/VA-Internal-S2-RCA1-v1.cer /usr/local/share/ca-certificates/
RUN openssl x509 -inform DER -in /usr/local/share/ca-certificates/VA-Internal-S2-RCA1-v1.cer -out /usr/local/share/ca-certificates/VA-Internal-S2-RCA1-v1.crt
RUN update-ca-certificates

# Use default production values.
RUN mv "$PHP_INI_DIR/php.ini-production" "$PHP_INI_DIR/php.ini"

# Inject PHP conf overrides (currently just `apc.enable=1`, need to ensure this is enabled on CLI too)
COPY .lando/zzz-lando-my-custom.ini $PHP_INI_DIR/conf.d/my-php.ini

# Change docroot.
RUN sed -ri -e 's#/var/www/html#${CMS_DOCROOT}#g' \
  /etc/apache2/sites-available/*.conf
RUN sed -ri -e 's#/var/www/#${CMS_DOCROOT}#g' \
  /etc/apache2/apache2.conf \
  /etc/apache2/conf-available/*.conf

RUN install-php-extensions \
  apcu \
  gd \
  memcache \
  opcache \
  pdo_mysql \
  xdebug \
  # Drush 10 requirement
  zip

# Install dependencies
RUN apt-get update \
  && apt-get install -y \
    # Git (for Composer/etc modules)
    git \
    # unzip (for unzipping Composer zip files)
    unzip

# Copy in the repository.
COPY . $CMS_REPO_ROOT
RUN chown -R www-data:www-data $CMS_REPO_ROOT/..

USER www-data

WORKDIR $CMS_REPO_ROOT

RUN composer install

USER root
RUN a2enmod rewrite

IDK if it works, but it builds. I'm out for the day, though 😃 There's a lot of room for improvement like possibly shifting to a multistage build and segregating the build/test dependencies, nuking Composer cache, etc, to get the final image size down. Segregating build/test dependencies compromises the goal of running the same image in all environments though, arguably, so IDK.

olivereri commented 2 years ago

I was able to build the image as well.

nuking Composer cache, etc, to get the final image size down.

Yeah, this a big one!

ndouglas commented 2 years ago

Belmore:va.gov-cms nathan.douglas$ du -d1 -h | sort -h
4.0K    ./.composer-cache
4.0K    ./hooks
8.0K    ./.vscode
 12K    ./.devcontainer
 12K    ./.lando
 24K    ./patches
 40K    ./.tugboat
 56K    ./simplesamlphp-config-metadata
 64K    ./.web
120K    ./.github
252K    ./scripts
580K    ./drush
784K    ./product-release-notes
1.8M    ./tests
2.6M    ./READMES
4.5M    ./bin
 11M    ./config
287M    ./.git
314M    ./node_modules
8.5G    ./.dumps
8.6G    ./docroot
 18G    .

Just occurred to me that I probably copied all kinds of crap into that image that shouldn't be copied. Need a .dockerignore

ndouglas commented 2 years ago

d'oh, it copies the images, etc in too, lol

olivereri commented 2 years ago

Not a huge problem. To speed it up committing the image to ECR (A6 AWS Account) I'm doing it on an EC2. Goal is to try and create a simple K8s manifest that pulls in this container image and deploys it with my local ArgoCD. Then I can report back what's up with the Dockerfile after I hook a DB up to it.

ndouglas commented 2 years ago

Yeah, I started with the approach of pulling the CMS repo in within the container as part of the build process, but that complicated my life so I just switched to copying it in under the assumption that the Dockerfile existed in the repo. Not positive which is the best approach in this situation.

olivereri commented 2 years ago

I like the latter approach because it probably jives well with how we might build the container with GHA. One workflow task checks out the repo, Dockerfile and all, then we build without the complication of trying to clone the repo in the container.

olivereri commented 2 years ago

Getting somewhere! Still missing a few pieces but I'm sure I can replicate what we do for Tugboat in K8s+ArgoCD to verify and tweak the Dockerfile.

argocd app create cms-drupal --repo https://github.com/olivereri/vsp-infra-cms.git --path cms --dest-server https://kubernetes.default.svc --dest-namespace default

ndouglas commented 2 years ago

yeah, that's what it looks like when it's not able to reach the DB server.

$databases['default']['default'] = array(
  'driver' => 'mysql',
  'database' => getenv('CMS_MARIADB_DATABASE') ?: 'drupal8',
  'username' => getenv('CMS_MARIADB_USERNAME') ?: 'drupal8',
  'password' => getenv('CMS_MARIADB_PASSWORD') ?: 'drupal8',
  'prefix' => '',
  // 'database' is the default DB container for Lando (local).
  'host' => getenv('CMS_MARIADB_HOST') ?: 'database',
  'port' => 3306,
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
);

I'll play with it this morning locally and open a PR including Dockerfile and .dockerignore to see what nightmares we encounter next 😃

ndouglas commented 2 years ago

I forked your repo to here just so I can play with it on my cluster, set up a DB, etc. I figure I'll toss in a CronJob, etc once I get the DB layer working.

ndouglas commented 2 years ago

Ideally most of it (outside the DB/Memcache stuff, which would be irrelevant) can be backported easily into EKS

olivereri commented 2 years ago

yeah, that's what it looks like when it's not able to reach the DB server.

I eventually found how settings.php does an include from docroot/sites/default/settings/ which contains a bunch of other specific settings.php file. Suppose I can create a new container that has a settings.eks.php with the values for a database service. The DB container I was thinking create a sperate Dockerfile using MariaDB 10.5 and load up latest CMS DB like Tugboat does:

        - curl --remote-name https://dsva-vagov-prod-cms-backup-sanitized.s3-us-gov-west-1.amazonaws.com/database/cms-prod-db-sanitized-latest.sql.gz
        - mysql -e "DROP DATABASE IF EXISTS tugboat; CREATE DATABASE tugboat";
        - zcat cms-prod-db-sanitized-latest.sql.gz | mysql tugboat
        - rm cms-prod-db-sanitized-latest.sql.gz

ndouglas commented 2 years ago

The Dockerfile's at least somewhat legit at this point -- this is the same image I built this morning. Just need to get the DB loaded and files copied to see what breaks next 😄

ndouglas commented 2 years ago

I'ma see if I can do that DB/file loading in an initContainer

ndouglas commented 2 years ago

IDK if you saw this @olivereri https://hub.docker.com/_/mariadb

Initializing a fresh instance

When a container is started for the first time, a new database with the specified name will be created and initialized with the provided configuration variables. Furthermore, it will execute files with extensions .sh, .sql, .sql.gz, and .sql.xz that are found in /docker-entrypoint-initdb.d. Files will be executed in alphabetical order. .sh files without file execute permission are sourced rather than executed. You can easily populate your mariadb services by mounting a SQL dump into that directory and provide custom images with contributed data. SQL files will be imported by default to the database specified by the MARIADB_DATABASE / MYSQL_DATABASE variable.

I used this mechanism before with the local dev environment for some D7 sites at my previous employer and it worked well.

olivereri commented 2 years ago

Not having a ton of luck with the initialization of the DB through /docker-entrypoint-initdb.d.

FROM mariadb:10.5
ENV MARIADB_ROOT_PASSWORD="drupal8"
ENV MARIADB_DATABASE="drupal8"
ENV MARIADB_USER="drupal8"
ADD cms-prod-db-sanitized-latest.sql.gz /docker-entrypoint-initdb.d

mk8s@Tritium:~/docker/cms/cms-db$ docker run cms-db:0.0.414 -it bash
2021-09-29 18:21:50+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.12+maria~focal started.
2021-09-29 18:21:50+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
        command was: mysqld -it bash --verbose --help --log-bin-index=/tmp/tmp.qIedQ1TfQ8
        2021-09-29 18:21:50 0 [Warning] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some options may be missing from the help text
2021-09-29 18:21:50 0 [ERROR] mysqld: unknown option '-i'

ndouglas commented 2 years ago

You gotta do docker run -it bash cms-db:0.0.414 because it'll interpret anything after IMAGE:TAG as arguments to the entrypoint. I don't think that'll fix the issue necessarily... that looks kind of like the volume is corrupted/incomplete.

ndouglas commented 2 years ago

You shouldn't need to do the docker-entrypoint-initdb.d thing inside the Dockerfile, though -- do something like docker run -v $(pwd)/dump:/docker-entrypoint-initdb.d cms-db:0.0.414 and it should load it up as part of the startup process.

olivereri commented 2 years ago

Welp, I was running the command wrong. This works: docker run -it cms-db:0.0.414 bash

do something like docker run -v $(pwd)/dump:/docker-entrypoint-initdb.d cms-db:0.0.414 and it should load it up as part of the startup process.

But what if I'm looking to create a container image that contains the CMS DB already loaded? So I can pull that image as part of a K8s mainfest and deploy with ArgoCD

ndouglas commented 2 years ago

Sorry, was off getting my 🍑 handed to me by another issue of my own creation 😩

Welp, I was running the command wrong. This works:

🤦 yeah, I switched the arguments above, my bad.

But what if I'm looking to create a container image that contains the CMS DB already loaded? So I can pull that image as part of a K8s mainfest and deploy with ArgoCD

I guess it depends on how ephemeral these environments are. Loading the DB into the image will work but not for something dev/stg/prod-like. It might be an option for local/CI though 🤔

What I'm doing (haven't gotten it to work yet, though):

kind: Deployment
apiVersion: apps/v1
metadata:
  name: mariadb
  namespace: cms-drupal
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      initContainers:
        - name: init-files
          image: curlimages/curl:7.79.1
          command: ['/bin/bash', '-c']
          args:
            - 'cd /data; curl --remote-name https://dsva-vagov-prod-cms-backup-sanitized.s3-us-gov-west-1.amazonaws.com/database/cms-prod-db-sanitized-latest.sql.gz'
          volumeMounts:
            - mountPath: /data
              subPath: init-database
              name: init-database
      containers:
        - name: mariadb
          image: mariadb:10.5
          ports:
            - containerPort: 3306
          env:
            - name: MYSQL_DATABASE
              value: drupal
            - name: MYSQL_USER
              value: drupal
            - name: MYSQL_PASSWORD
              value: drupal
            - name: MYSQL_RANDOM_ROOT_PASSWORD
              value: 'yes'
          volumeMounts:
            - mountPath: /var/lib/mysql/
              name: database
              subPath: mysql
            - mountPath: /docker-entrypoint-initdb.d
              name: init-database
              subPath: init-database
      volumes:
        - name: database
          persistentVolumeClaim:
            claimName: mariadb-pvc
        - name: init-database
          persistentVolumeClaim:
            claimName: mariadb-init-pvc

So the idea is that the initContainer pulls the DB dump, and then it'll get loaded by the DB container.

I guess the approach just depends on what you're trying to accomplish. Copying it into the container should definitely work, and so far mine isn't, so IDK 😛

olivereri commented 2 years ago

I guess the approach just depends on what you're trying to accomplish. Copying it into the container should definitely work, and so far mine isn't, so IDK

It was supposed to be something quick to get Drupal hooked up to a DB and test out the php7.3 container image. Even when I get MariaDB running I'm not sure if it even did the init steps. MariaDB doesn't appear to be running and won't start.

ndouglas commented 2 years ago

I'm trying this:

FROM mariadb:10.5
ENV MARIADB_ROOT_PASSWORD="drupal8"
ENV MARIADB_DATABASE="drupal8"
ENV MARIADB_USER="drupal8"
ADD https://dsva-vagov-prod-cms-backup-sanitized.s3-us-gov-west-1.amazonaws.com/database/cms-prod-db-sanitized-latest.sql.gz /docker-entrypoint-initdb.d/cms-prod-db-sanitized-latest.sql.gz

🔔nathan.douglas@Belmore:~/dblol$ docker run marialol
2021-09-29 19:49:45+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.12+maria~focal started.
2021-09-29 19:49:45+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2021-09-29 19:49:45+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.12+maria~focal started.
2021-09-29 19:49:45+00:00 [Note] [Entrypoint]: Initializing database files

PLEASE REMEMBER TO SET A PASSWORD FOR THE MariaDB root USER !
To do so, start the server, then issue the following command:

'/usr/bin/mysql_secure_installation'

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the MariaDB Knowledgebase at https://mariadb.com/kb or the
MySQL manual for more instructions.

Please report any problems at https://mariadb.org/jira

The latest information about MariaDB is available at https://mariadb.org/.
You can find additional information about the MySQL part at:
https://dev.mysql.com
Consider joining MariaDB's strong and vibrant community:
https://mariadb.org/get-involved/

2021-09-29 19:49:46+00:00 [Note] [Entrypoint]: Database files initialized
2021-09-29 19:49:46+00:00 [Note] [Entrypoint]: Starting temporary server
2021-09-29 19:49:46+00:00 [Note] [Entrypoint]: Waiting for server startup
2021-09-29 19:49:46 0 [Note] mysqld (mysqld 10.5.12-MariaDB-1:10.5.12+maria~focal) starting as process 113 ...
2021-09-29 19:49:46 0 [Note] InnoDB: Uses event mutexes
2021-09-29 19:49:46 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-09-29 19:49:46 0 [Note] InnoDB: Number of pools: 1
2021-09-29 19:49:46 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-09-29 19:49:46 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2021-09-29 19:49:46 0 [Note] InnoDB: Using Linux native AIO
2021-09-29 19:49:46 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2021-09-29 19:49:46 0 [Note] InnoDB: Completed initialization of buffer pool
2021-09-29 19:49:46 0 [Note] InnoDB: 128 rollback segments are active.
2021-09-29 19:49:46 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-09-29 19:49:46 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-09-29 19:49:46 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-09-29 19:49:46 0 [Note] InnoDB: 10.5.12 started; log sequence number 45094; transaction id 20
2021-09-29 19:49:46 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-09-29 19:49:46 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-09-29 19:49:46 0 [Note] InnoDB: Buffer pool(s) load completed at 210929 19:49:46
2021-09-29 19:49:46 0 [Warning] 'user' entry 'root@afc257347f37' ignored in --skip-name-resolve mode.
2021-09-29 19:49:46 0 [Warning] 'proxies_priv' entry '@% root@afc257347f37' ignored in --skip-name-resolve mode.
2021-09-29 19:49:46 0 [Note] Reading of all Master_info entries succeeded
2021-09-29 19:49:46 0 [Note] Added new Master_info '' to hash table
2021-09-29 19:49:46 0 [Note] mysqld: ready for connections.
Version: '10.5.12-MariaDB-1:10.5.12+maria~focal'  socket: '/run/mysqld/mysqld.sock'  port: 0  mariadb.org binary distribution
2021-09-29 19:49:47+00:00 [Note] [Entrypoint]: Temporary server started.
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
2021-09-29 19:49:49 5 [Warning] 'proxies_priv' entry '@% root@afc257347f37' ignored in --skip-name-resolve mode.
2021-09-29 19:49:49+00:00 [Note] [Entrypoint]: Creating database drupal8

2021-09-29 19:49:49+00:00 [Note] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/cms-prod-db-sanitized-latest.sql.gz
gzip: /docker-entrypoint-initdb.d/cms-prod-db-sanitized-latest.sql.gz: Permission denied

weird, I thought it'd correctly identify .sql.gz

ndouglas commented 2 years ago

Furthermore, it will execute files with extensions .sh, .sql, .sql.gz, and .sql.xz that are found in /docker-entrypoint-initdb.d.

I guess I just assumed that by "execute" they meant "will pipe into MySQL", not "will assume it's a script, given the well-known tendency of devops engineers to give their Bash scripts names ending in .sql.gz." 😡

olivereri commented 2 years ago

Oh jeez, so by adding -it and bash I skipped the entrypoint stuff? As soon as I just ran it like you did it started moving along. Still have some learning to do around Docker.

It's sitting here right now: 2021-09-29 19:57:15+00:00 [Note] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/cms-prod-db-sanitized-latest.sql.gz

ndouglas commented 2 years ago

FROM mariadb:10.5
ENV MARIADB_ROOT_PASSWORD="drupal8"
ENV MARIADB_DATABASE="drupal8"
ENV MARIADB_USER="drupal8"
ADD https://dsva-vagov-prod-cms-backup-sanitized.s3-us-gov-west-1.amazonaws.com/database/cms-prod-db-sanitized-latest.sql.gz /docker-entrypoint-initdb.d/cms-prod-db-sanitized-latest.sql.gz
RUN chown -R mysql:mysql /docker-entrypoint-initdb.d/

chowning the directory got it to work here -- 🤦 guess it's the kind of "permission denied" that means "read permission denied" and not "execute permission denied"

olivereri commented 2 years ago

I had downloaded the sql.gz file locally and chmod'd it before running the build. I wonder why the K8s Manifest file didn't work. It's essentially the same thing.

ndouglas commented 2 years ago

docker run --link elegant_borg:database -p 8080:80 druplol:test this is working but the login credentials aren't being created correctly, I think

ndouglas commented 2 years ago

maybe those commands don't get invoked if the DB is inited from the folder and we'll need to add some GRANT ALL type instructions

ndouglas commented 2 years ago

ugh

MariaDB [mysql]> SELECT User, Host, Password FROM mysql.user;
+-------------+-----------+-------------------------------------------+
| User        | Host      | Password                                  |
+-------------+-----------+-------------------------------------------+
| mariadb.sys | localhost |                                           |
| root        | localhost | *67D4DA2FE87470FC1F7D487BC0649B37910E3AFB |
| root        | %         | *67D4DA2FE87470FC1F7D487BC0649B37910E3AFB |
+-------------+-----------+-------------------------------------------+
3 rows in set (0.001 sec)

olivereri commented 2 years ago

I haven't gotten much farther. Created a container image from the container that loaded up the DB. Added a new database service to the K8s manifest and now I'm getting this:

The website encountered an unexpected error. Please try again later. PDOException: SQLSTATE[HY000] [2002] Connection refused in Drupal\Component\DependencyInjection\PhpArrayContainer->createService() (line 79 of /var/www/cms/docroot/core/lib/Drupal/Component/DependencyInjection/PhpArrayContainer.php).

Think something is up with the pod configuration.

olivereri commented 2 years ago

This is what I got:

MariaDB [drupal8]>  SELECT User, Host, Password FROM mysql.user;
+-------------+-----------+-------------------------------------------+
| User        | Host      | Password                                  |
+-------------+-----------+-------------------------------------------+
| mariadb.sys | localhost |                                           |
| root        | localhost | *D31856F750FA9B4D8D6B48DFD5654F678F151F4C |
| root        | %         | *D31856F750FA9B4D8D6B48DFD5654F678F151F4C |
| drupal8     | %         | *D31856F750FA9B4D8D6B48DFD5654F678F151F4C |
+-------------+-----------+-------------------------------------------+
4 rows in set (0.001 sec)

And this is the Docker file:

FROM mariadb:10.5
ENV MARIADB_ROOT_PASSWORD="drupal8"
ENV MARIADB_PASSWORD="drupal8"
ENV MARIADB_USER="drupal8"
ENV MARIADB_DATABASE="drupal8"
ADD cms-prod-db-sanitized-latest.sql.gz /docker-entrypoint-initdb.d

Wonder why the drupal8 user wasn't created. I suppose the only difference is I'm using an image after the entrypoint loaded up cms DB.

ndouglas commented 2 years ago

Yeah, that's weird. It's really the 💩 🍒 on top of the rest of this day 😁

So I did this:

MariaDB [drupal8]> CREATE USER drupal8@'%' IDENTIFIED BY 'drupal8';
Query OK, 0 rows affected (0.002 sec)
MariaDB [drupal8]> GRANT ALL PRIVILEGES ON drupal8.* TO drupal8@'%';
Query OK, 0 rows affected (0.001 sec)

and now I get:

Which I think is just a sign that the cache needs to be cleared.

ndouglas commented 2 years ago

TRUNCATE TABLE `cache_discovery_migration`;    
TRUNCATE TABLE `cache_toolbar`;                
TRUNCATE TABLE `cache_library`;                
TRUNCATE TABLE `cache_data`;                   
TRUNCATE TABLE `cache_bootstrap`;              
TRUNCATE TABLE `cache_migrate`;                
TRUNCATE TABLE `cache_dynamic_page_cache`;     
TRUNCATE TABLE `cache_graphql_results`;        
TRUNCATE TABLE `cache_rest`;                   
TRUNCATE TABLE `cache_entity`;                 
TRUNCATE TABLE `cache_render`;                 
TRUNCATE TABLE `cache_container`;              
TRUNCATE TABLE `cache_default`;                
TRUNCATE TABLE `cachetags`;                    
TRUNCATE TABLE `cache_graphql_definitions`;    
TRUNCATE TABLE `cache_config`;                 
TRUNCATE TABLE `cache_menu`;                   
TRUNCATE TABLE `cache_jsonapi_normalizations`; 
TRUNCATE TABLE `cache_tome_static`;            
TRUNCATE TABLE `cache_discovery`;              
TRUNCATE TABLE `cache_page`;

After that, though, I get the same error about the circular dependency.

olivereri commented 2 years ago

Oh boy!

ndouglas commented 2 years ago

lol

ndouglas commented 2 years ago

Playing with it this morning (in raw Docker, not K8s). This is interesting.

From a web request, I get the circular reference from Raven.

From drush cr, I get:

fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
PHP Fatal error:  Allowed memory size of 4294967296 bytes exhausted (tried to allocate 561594365579040 bytes) in /var/www/cms/docroot/core/modules/datetime/datetime.views.inc on line 43
PHP Stack trace:
PHP   1. {main}() /var/www/cms/docroot/vendor/drush/drush/drush:0
PHP   2. require() /var/www/cms/docroot/vendor/drush/drush/drush:4
PHP   3. Drush\Runtime\Runtime->run($argv = [0 => 'docroot/vendor/drush/drush/drush', 1 => 'cr']) /var/www/cms/docroot/vendor/drush/drush/drush.php:72

The .git messages are interesting -- I wonder why it's trying to perform a git operation in a cache rebuild. That kinda makes me think I should remove the .git line from the .dockerignore, to see if that's what's causing the issue -- but come to think of it, I think you saw that on an image built before the .dockerignore existed.

Also interesting -- I thought I'd attempt a reinstall, and:

The poking around continues...

olivereri commented 2 years ago

I was able replicate the weird behavior where the database is refusing connections. Seems like it something to do with microk8s. On a different machine with a fresh install it's still having pod communication issues. I'm stumped, a lot of troubleshooting guides don't really address this issue.

I think you saw that on an image built before the .dockerignore existed.

Yup, just a few messages up above.

ndouglas commented 2 years ago

Figured it out!

So the message was:

Circular reference detected for service "logger.raven", path: "raven.request_subscriber -> logger.raven -> logger.factory".

Services involved were:

logger.raven: class: Drupal\raven\Logger\Raven arguments:

'@config.factory'

'@logger.log_message_parser'

'@module_handler'

'%kernel.environment%'

'@?current_user'

'@?request_stack'

'@?settings' tags:

{ name: logger }

raven.request_subscriber: class: Drupal\raven\EventSubscriber\RequestSubscriber calls:

[setContainer, ['@service_container']] arguments:

'@config.factory'

'@logger.raven'

'@datetime.time' tags:

{ name: event_subscriber }

So the raven.request_subscriber service injects logger.raven (in order to log), which injects logger.factory (in order to be created). I was stuck here for a while because I couldn't figure out what happened next. I thought there must be something going on with the request subscriber, or with the datetime.time service (since we were seeing the memory blow up with the datetime view). I was looking everywhere but in the right place.

Turns out that we alter Raven's options:

/**
 * Implements hook_raven_options_alter().
 */
function va_gov_backend_raven_options_alter(array &$options) {
  try {
    $repository = Drupal::getContainer()->get('va_gov.consumers.git.repository.factory')->getAppRepository();
    $options['release'] = !empty($repository->getLastCommit()) ? $repository->getLastCommit()->getHash() : NULL;
  }
  catch (RuntimeExceptionAlias $e) {
    watchdog_exception('git', $e, 'Error getting HEAD commit');
  }

}

Narrator: "It was at this moment, he knew..."

So I screwed this up by adding .git to the .dockerignore file. When the Raven logger options are being built, we attempt to get info about the Git repo. We fail because it's not a git repository, so watchdog_exception() is called. That then attempts to load the logger, and there's the circular dependency.

So I removed .git from the .dockerignore, rebuilt, and restarted, and it seems to work normally. So far.

olivereri commented 2 years ago

I was hoping to get further along with this by using VSP EKS. The app manifest is committed to the repo and the resource are deployed however the site isn't accessible at eks-dev.cms.va.gov all that is returned is Service not found. I'm unsure if that's traefik or Drupal. Through curl I can see DNS is resolving correctly.

olivereri commented 2 years ago

Welp, I figured it out. traefik.vfs.va.gov doesn't point to some pod in EKS but an AWS ELB. I added a rule to route traffic correctly in the LB and got this:

Maybe we'll need to commit the containers you've created to DSVA ECR and modify the manifest to pull from there.

Great, so we're in, and it's kind of running. Not sure what we can do about this. For some reason ArgoCD won't let me delete pods in an attempt to recreate them.

ndouglas commented 2 years ago

That's success!

Yeah, def. gotta rebuild the image. I'll see what I can do tomorrow AM. Hoping that by scrum we'll have a bright shiny new Drupal. Just without the image assets, etc.

ndouglas commented 2 years ago

Bummer, I wish I'd caught your message a couple hours ago -- I know Traefik's messages very well at this point and coulda pointed you in the right direction 😭 . Either way, that's awesome work 😃

ndouglas commented 2 years ago

https://console.amazonaws-us-gov.com/ecr/repositories/dsva/cms-drupal/image/sha256:35fcdb46b73b5ba07571143ecb50fa794b5af2cc4441502439176261846a176f/scan-results/?region=us-gov-west-1 huh, neat

ndouglas commented 2 years ago

ndouglas commented 2 years ago

Seems to work, no errors in status report, but:

can't send email
Sentry is linked to cms-prod

department-of-veterans-affairs / va.gov-cms