search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
283 stars 204 forks source link

Install and Configure Google Lighthouse Plugin in Platform Console #43452

Closed pjhill closed 2 years ago

pjhill commented 2 years ago

Description

The Backstage marketplace has a Google Lighthouse plugin which we will be able to install and configure for use in the Platform Console.

Tasks

Acceptance Criteria

rbeckwith-oddball commented 2 years ago

Are we intending to create any plugins or components ourselves, or is this only going to be limited to installing the Lighthouse plugin?

pjhill commented 2 years ago

To summarize the discussion from stand-up, we don't expect to be creating a custom plugin ourselves at this time. The idea for this ticket is to pluck the plugin from the Backstage marketplace and get it installed in Platform Console.

rbeckwith-oddball commented 2 years ago
rbeckwith-oddball commented 2 years ago

It looks like there isn't a way of installing this without setting up my machine for local development. Following the setup instructions from the document. Should ideally have my environment setup by Monday with the potential for a PR later on Monday or early Tuesday.

rbeckwith-oddball commented 2 years ago

Links used for setting up local environment:

rbeckwith-oddball commented 2 years ago

Got keycloak and Postgres installed and setup. It was a bit of work since I had versions installed specific to tud. Will be attempting to install the plug-in manually around lunch today.

rbeckwith-oddball commented 2 years ago

The platform console UI has been setup. Ran into some hiccups with backstage persistent storage, but should be properly configured now. Have to follow the steps / procedures from here to finish up.

rbeckwith-oddball commented 2 years ago

The Lighthouse plugin was one of the more complicated plugins I encountered. Using the details from previous tests in BigQuerey for comparison, it looks like there might already be a Lighthouse audit service we may be using. In which case, some of the settings will need to be modified to use that service instead of installing another one via docker on the instance actually running the platform-console.

We'll need some further details before I consider it safe to submit a PR for this branch - Mainly if we want lighthouse to be installed as a service on the machine that runs platform-console, or do we already have a lighthouse audit service we should use instead? This would only be a minor tweak to the settings in the .yaml file.

rbeckwith-oddball commented 2 years ago

The creation of the catalog file isn't quite working the way the documents lead me to believe, but this should be a quick fix / implementation when finding out what the deal is. I'll likely check-in on Monday's meeting just to make sure everything is clear.

ericvoshall commented 2 years ago

@rbeckwith-oddball Hey Ryan, any questions we can answer or anything our team can help out with on this?

pjhill commented 2 years ago

Putting this back into In Progress for now since there are a few more tasks to complete before the ticket is wrapped.

rbeckwith-oddball commented 2 years ago

@ericvoshall - sorry for the delayed response, I was out last week. The Google Lighthouse plugin requires the Lighthouse Audit Plugin to run. You can see the changes to the Docker file here.

Is there something specific to do in order to get the the service steup as a container for production or staging, or will something pick up the changes in the docker-compose.yml file?

You can also see that it needs postgres auth stuff to setup the audit tables as well as save audit results.

ericvoshall commented 2 years ago

Our current setup deploys only a single Docker image for the Console UI that we build. I would recommend exploring the instructions here to have the audit service run as a nested Express app instead of as a container.

All of the configuration values required to connect to Postgres (user, host, port, etc) should already be accessible through environment variables (or ideally through Backstage's config object).

rbeckwith-oddball commented 2 years ago

I was able to get the lighthouse audit service to run as an express app, but the lighthouse plugin itself fails to recognize the baseUrl setting.lighthouse_update1.png


 30   static fromConfig(config) {
 31     return new LighthouseRestApi(config.getString("lighthouse.baseUrl"));
 32   }

to become


 30   static fromConfig(config) {
 31     try {
 32       return new LighthouseRestApi(config.getString("lighthouse.baseUrl"));
 33     } catch (err) {
 34       return new LighthouseRestApi('http://127.0.0.1:4008/lighthouse');
 35     }
 36   }

lighthouse_update2.png Unless something was changed that alters the scope of Config objects and not being able to see certain variables?

Problem

Node modules aren't included in the git commits, so I'm not sure of what to do in order to resolve this.

ericvoshall commented 2 years ago

@rbeckwith-oddball Is your branch pushed up? I can take a look and see if I can figure out what's going on.

rbeckwith-oddball commented 2 years ago

@ericvoshall - the git commit check informed me that the yarn dependency wasn't properly linked in the backend. yarn add... seems to have worked. The code is committed as rb/qas/backstage-lighthouse-plugin-express if you want to look over it anyhow.

ericvoshall commented 2 years ago

@rbeckwith-oddball I just pushed a commit up. There were two main changes I made:

  1. I removed the config values other than lighthouse.baseUrl. I was getting an error locally including those (and they're accessible through the backend.database.connection config path anyway).
  2. I included some basic setup to ensure that a database specific to the Lighthouse audit runs was being created on startup.

These changes have this working locally for me:

  1. Update local catalog-info.yaml file to include lighthouse.com/website-url: https://platform-console-ui-dev.vfs.va.gov annotation.
  2. Go to http://127.0.0.1:3000/lighthouse
  3. Create a new audit run for https://platform-console-ui-dev.vfs.va.gov
  4. Go to entity page for Platform Console UI
  5. Lighthouse audit shows

image

rbeckwith-oddball commented 2 years ago

Excellent @ericvoshall! Final question: what should the base URL for production and / or utility be set to? I can see that app-config.production.yaml would be the location to update the production value, but I'm not sure where we would update the utility version for the baseUrl.

ericvoshall commented 2 years ago

The app-config.yaml and app-config.production.yaml names are a little misleading - the two files are merged for all deployed environments (dev, staging, utility). We do not currently deploy to the Production environment.

The value shouldn't need to change based on environment, so I think just including it as it is in the app-config.yaml as http://127.0.0.1:4008/lighthouse should be enough.

rbeckwith-oddball commented 2 years ago

The PR has been submitted https://github.com/department-of-veterans-affairs/platform-console-ui/pull/849

rbeckwith-oddball commented 2 years ago

The automated tests are having some issues that are being investigated now.

rbeckwith-oddball commented 2 years ago

The automated tests have now passed after a few false positives were skipped (confirmed as false by platform-console team). Eric has merged the code and it should be in the dev environment within the next 10-15 minutes.

rbeckwith-oddball commented 2 years ago

It merged, new tweak had to be pushed since the lighthouse baseUrl apparently isn't used internally. Now the platform-console team needs to do something with their manifest files to get the port open.

rbeckwith-oddball commented 2 years ago

They were able to find a way to proxy the request instead of having to open an external port. The issue being encountered now is:

{
  "auditId": "14a3893e-dd77-4e2f-9299-e103984e812e",
  "level": "error",
  "message": "failed to launch puppeteer browser.\nError: Failed to launch the browser process! spawn /home/node/app/node_modules/puppeteer/.local-chromium/linux-982053/chrome-linux/chrome ENOENT\n\n\nTROUBLESHOOTING: https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md\n",
  "service": "lighthouse-audit-service",
  "url": "https://platform-console-ui-dev.vfs.va.gov"
}

But I was able to verify that the file actually exists, so they are investigating further.


~/app/node_modules/puppeteer/.local-chromium/linux-982053/chrome-linux $ stat /home/node/app/node_modules/puppeteer/.local-chromium/linux-982053/chrome-linux/chrome
  File: /home/node/app/node_modules/puppeteer/.local-chromium/linux-982053/chrome-linux/chrome
  Size: 325132000       Blocks: 635032     IO Block: 4096   regular file
Device: 10000ah/1048586d        Inode: 2770546     Links: 1
Access: (0755/-rwxr-xr-x)  Uid: ( 1000/    node)   Gid: ( 1000/    node)
Access: 2022-08-18 13:48:15.000000000 +0000
Modify: 2022-08-18 13:48:15.000000000 +0000
Change: 2022-08-18 13:59:56.113871298 +0000
rbeckwith-oddball commented 2 years ago

I realized at this point, the plug-in is a service / system-level installation. I realized I was writing documentation for installation of the plug-in, but now that the code has been merged, there shouldn't be any additional steps needed to get the plug-in running locally or in development. So, nobody should technically have to go through this again. The documentation made more sense to me when I thought the plug-in was going to be user level. I can still publish the system install, but didn't know if it seemed unnecessary to anyone else.

pjhill commented 2 years ago

Recording the process to get it merged is still important for future plugin installation. This was partially intended to trailblaze for subsequent installation of plugins from the Backstage plugin marketplace if anything relevant pops up.

Is there any documentation that would be helpful for ongoing maintenance of the plugin? Updating it to a new version? Altering the configuration?

rbeckwith-oddball commented 2 years ago

The most up-to-date / relevant documentation can be found at Google Lighthouse Plugin - Confluence.

As it currently stands, the plugin is configured and enabled, but doesn't function properly for self-signed certificates. Further investigation and discussions with Eric lead us to believe that there isn't a method of working around this issue at the moment, but installing the VA certs into he container may rectify the issue (which is something the platform console is doing). lighthouse_issue1.png

{"id":"8ad8b92c-e532-4a07-8167-6bcb4279fd4a","url":"https://platform-console-ui-dev.vfs.va.gov","timeCreated":"2022-08-23T18:26:29.765Z","status":"RUNNING"}

rbeckwith-oddball commented 2 years ago

Update

Installing the certificates in the local containers has rectified the issue with the self-signed certificate errors when running an audit against some of our websites. lighthouse_noerror2.png

pjhill commented 2 years ago

Looking good in dev. We can discuss a plan to promote this to the prod equivalent utility environment in the future.