Explore Lagoon Developer Experience

TheBoatyMcBoatFace commented 1 year ago

Overview

Lagoon is a powerful devops tool that will greatly streamline developer interaction with the CMS. In order to make the most of this tool and streamline our development process, it is important that we have a clear understanding of how to set up and use Lagoon for local development.

As a CMS engineer, I need a clear understanding of how my changes go from my local machine, through approval, and to the production website. How will implementing Lagoon impact my current workflows and what do I need to do in order incorporate those changes?

tl;dr: Change Management for local development

Proposed Tasks

Research and understand the current process for setting up Lagoon for local development.
Create a detailed document outlining the steps and requirements for setting up Lagoon locally.
Document impact of Lagoon to CMS engineer workflow(s)

Why this Matters

By documenting and exploring the Lagoon local dev setup, we can better understand its capabilities and limitations and make informed decisions about how to incorporate it into our development process. This will help us streamline our workflows and improve the efficiency of our development teams.

Acceptance Criteria

[x] Explore Lagoon impact to CMS engineers
[x] Explore Lagoon's impact on Windows, Linux, MacOS (Intel & M1)
[x] Convey a list of improvements & changes for Lagoon

Feedback from external teams sought in Sprint 75.

TheBoatyMcBoatFace commented 1 year ago

Repointed during discovery & after acceptance criteria modifications. Good per Dotti

TheBoatyMcBoatFace commented 1 year ago

Shifting to Sprint 75 due to holiday scheduling conflicts.

EWashb commented 1 year ago

@ElijahLynn has this been completed? If not, I'll bring into sprint 76

ElijahLynn commented 1 year ago

@ElijahLynn has this been completed? If not, I'll bring into sprint 76

No, this got derailed before our onsite and with the team structure shift.

olivereri commented 1 year ago

Notes on working with Pygmy while Lagoonizing CMS-Test

$> pygmy up
[ ] amazeeio-haproxy is not able to start on port 80: listen tcp :80: bind: permission denied
[ ] amazeeio-haproxy is not able to start on port 443: listen tcp :443: bind: permission denied
Please address the above issues before you attempt to start Pygmy again.

Googling around landed on Amazee docs. Had a thought to try and manually start haproxy. https://github.com/amazeeio/docker-haproxy

$> docker run -d -p 80:80 -p 443:443 --volume=/var/run/docker.sock:/tmp/docker.sock --name=amazeeio-haproxy amazeeio/haproxy

Sucess!

$> pygmy up
Successfully pulled pygmystack/ssh-agent
Successfully started amazeeio-ssh-agent
Successfully pulled pygmystack/dnsmasq
Successfully started amazeeio-dnsmasq
Successfully pulled pygmystack/haproxy
Already Running amazeeio-haproxy
Successfully pulled pygmystack/mailhog
Successfully started amazeeio-mailhog
Successfully created network amazeeio-network
Successfully connected amazeeio-ssh-agent to amazeeio-network
Successfully connected amazeeio-haproxy to amazeeio-network
Successfully connected amazeeio-mailhog to amazeeio-network
Successfully configured resolvr Linux Resolver
Validation success for SSH key /home/agile6/.ssh/id_rsa
Successfully added SSH key /home/agile6/.ssh/id_rsa to agent
 ! http://docker.amazee.io/stats (amazeeio-haproxy)
 ! http://mailhog.docker.amazee.io (amazeeio-mailhog)

Ran pygmy down and docker stop [haproxy container ID] then pygmy up again and get the same HAproxy error. Still need to figure out why pygmy doesn't have permissions to bind ports.

olivereri commented 1 year ago

Lagoon support for DDev rough notes: https://docs.google.com/document/d/1GVFPkq48S3O_eIT2TfBOk-j4zvPDbSAPEBhM2yRdApk/edit

EWashb commented 1 year ago

@ElijahLynn @olivereri where are you at on this one?

olivereri commented 1 year ago

@ElijahLynn @olivereri where are you at on this one?

Not much more progress than what is documented above. I feel like this one is blocked by https://github.com/department-of-veterans-affairs/va.gov-cms/issues/12291 because we need a working PoC to really work through DevEx.

dawnpruitt commented 1 year ago

@olivereri @ElijahLynn just checking in on this one to see if it is still blocked and if we anticipate this being completed by the end of sprint 77. Thanks!

ElijahLynn commented 1 year ago

Sorry, I didn't know this was in Sprint 77. I have #7667 and #11051 that I was planning on doing next. Should this story be in this sprint?

EWashb commented 1 year ago

Sorry @ElijahLynn this was confusing. We just explained to Dave that this was blocked by what we are working on now. It should have never been prioritized before the demo environment to begin with. He's fine with it. Don't worry about this one. Just focus on #12291 for this sprint

ElijahLynn commented 1 year ago

Okay, thanks.

olivereri commented 1 year ago

Initial Assumption: There should be no change to Drupal Engineer/Developer experience (DevEx).

Reasoning: Tighter DDEV and Lagoon integration is a nice to have, not a technical blocker. Developers can still use DDEV for local development, create PRs, and merge them even if Lagoon is our new hosting platform. By not using Pygmy, Lagoon's preferred local development option, developers won't be testing locally a mirror of Prod.

Impetus: The "nice to have" portion is a desire to use a single container image as a source of truth. Not having multiple container configurations that must be tracked and accounted for when, for example, upgrading PHP versions. With the adoption of Lagoon there were will be three (3) configurations to track: Tugboat, Pygmy, DDEV.

olivereri commented 1 year ago

Introduction

The initial assumption is correct. This will have no impact for CMS Engineers on WIndows, Linux, or Mac (Intel and M1), unless we want it to. The Usage of Lagoon does not obviate the use of existing local development tools, i.e. DDEV.

It is actually preferable to continue using DDEV for local development rather than Lagoon's preferred tool for local development, Pygmy. The main advantage to using Pygmy over DDEV for local development and testing is that what is run locally is exactly what will be deployed to production. If it runs locally, it also runs on production. It use the same container images, the same services, the same PHP configuration. Beyond that the advantages are trivial.

What is Pygmy?

Pygmy is a Docker based Drupal Development environment that simplifies local development environments for web applications. Lagoon, on the other hand, is a container-based hosting platform for web applications. While they are both used in the context of web development, they serve different purposes.

That being said, Pygmy can be used in conjunction with Lagoon to provide a local development environment that closely matches the production environment in Lagoon. This can help to minimize differences between the local and production environments and ensure that the application works as expected when deployed to Lagoon.

To use Pygmy with Lagoon, you can follow these general steps:

Install Pygmy on your local machine.
Configure Pygmy to work with Lagoon by specifying the URL of the Lagoon API endpoint and the credentials needed to authenticate with the API.
Start the Pygmy containers, which will create a local development environment that closely matches the Lagoon environment.
Clone your web application code to your local machine and configure it to use the Pygmy environment.
Test your application locally to ensure that it works as expected.
When ready, deploy your application to Lagoon using the Lagoon CLI or the Lagoon web interface.
Overall, using Pygmy with Lagoon can help to streamline the development and deployment process for web applications, while also ensuring that the application works as expected in both the local and production environments.

Pygmy provides several things that Docker does not:

An HTTP reverse proxy for nice URLs and HTTPS offloading.
A DNS system so we don't have to remember IP addresses.
SSH agents to use SSH keys within containers.
A system that receives and displays mail locally.

Advantages of DDEV over Pygmy

DDEV provides several features that Pygmy does not have, which can be useful for Drupal development. Here are some examples:

Drupal-specific integration: DDEV has a dedicated Drupal integration that streamlines the setup process for Drupal projects. For example, the DDEV Drupal integration automatically sets up a database, imports a default database dump, and creates a default settings file, saving developers time and effort.
Integrated Xdebug support: DDEV comes with Xdebug, a powerful debugging tool that allows developers to easily debug PHP code in real-time. Pygmy does not have Xdebug integrated, so developers would need to set it up manually.
Comprehensive CLI tools: DDEV provides a comprehensive set of command-line tools that make it easy to manage Drupal development environments. For example, developers can use DDEV commands to import/export databases, update Drupal core and contributed modules, and run automated tests.
Support for multiple environments: DDEV supports creating multiple environments, allowing developers to easily switch between different configurations for different projects. This can be useful for managing Drupal projects that have different requirements, such as different PHP versions or different sets of contributed modules.
Integration with development tools: DDEV integrates with popular development tools like VS Code, PhpStorm, and Sublime Text, providing a seamless development experience. Pygmy is more focused on providing a simple command-line interface for managing local Docker environments.

Overall, while Pygmy is a powerful tool for creating Docker-based local development environments, DDEV offers a more comprehensive set of features that make it a better fit for many Drupal development projects. Developers working on Drupal projects may find that the Drupal-specific integration, Xdebug support, and comprehensive CLI tools of DDEV make it the better choice for their needs.

Other Areas of Developer Experience

Using Lagoon to deploy and host the CMS may have a few more developer experience advantages over the current hosting system. Currently Jenkins is used as the deployment orchestration tool to run all jobs or tasks required to deploy and generally run the CMS.

Finding what job runs deploys, builds and run-time tasks can be daunting. Jenkins is shared with other projects and the UI can get very noisy. It also is an issue inspecting the code for the jobs and Ansible playbooks since they exist in a separate repository.

With Lagoon this will be all more centralized for the UI and codebase. It will become far easier for developers to track their changes from Pull-request to deployment than it currently is.

Improvements and Changes for Lagoon

Currently we are recommending a partnership between Lagoon and DDEV to provide closer integration for local development. This would be like the support that Lando already has for Lagoon:

Closely mimics Lagoon's stack, versions locally
Allows you to easily pull your Lagoon site down locally
Allows you to easily push your changes back to Lagoon

With that integration it would be possible for DDEV to be more representative locally of what will run on production while taking advantage of all the development features DDEV provides. Additionally, it would result in less work for DevOps engineers to maintain separate PHP configurations for each environment and carefully keep thme in sync.

With regard to the Lagoon UI some developer experience imporvments are:

Live logs during deployments. Currently updates are only available on page refreshes.
Container logs aren't avaialble in UI. A different logging system is required to view them.
An automatic re-login on page loads occurs and is a nuisance.

department-of-veterans-affairs / va.gov-cms