search

NASA-PDS / devops

Parent repo for PDS DevOps activities
Apache License 2.0
0 stars 0 forks source link

Phase 2: Develop CI/CD for NPM/React projects #69

Closed jordanpadams closed 4 weeks ago

jordanpadams commented 9 months ago

💡 Description

Follow on to https://github.com/NASA-PDS/devops/issues/67

Laying the groundwork for a future roundup requirement, let's create a new GitHub Action in a repo using NPM and test our what we want/need in order to support it.

For now, let's just build out the branch testing, unstable, and stable deliveries:

branch-cicd.yaml - install dependencies, build, verify success:

unstable-cicd.yaml - install dependencies, build, deploy to dev npm (NOTE: requires package.json to already be updated as indicated in README):

stable-cicd.yaml - install dependencies, build, deploy to dev npm (NOTE: requires package.json to already be updated as indicated in README):

Sub-tasks

nutjob4life commented 9 months ago

Quick question:

In this comment @eddiesarevalo suggests using npm publish --tag unstable, but @jordanpadams writes (in the issue description ↑) npm publish --tag beta.

I have no feelings either way. What would you prefer? @anilnatha, do you have a preference?

nutjob4life commented 8 months ago

I believe this pull request will close the following issues

jordanpadams commented 8 months ago

📆 03/2024 status: In work. On schedule

jordanpadams commented 7 months ago

📆 04/2024 status: Updated end date to better reflect the work needed to complete continuous deployment for these applications. No impact on delivery.

jordanpadams commented 6 months ago

📆 05/2024 status: In work. On track for 6/20 delivery.

nutjob4life commented 5 months ago

This epic talks about using GitHub Actions for CI and CD.

CI Part

I've indentified the following repositories as using Node.js (based on presence of the package.json file). These repositories need the Roundup's newest support for Node packaging and artifact hosting plus branch testing—the "CI" part (2 open PRs):

CD Part

These repositories are using Terraform (based on the presence of .tf files)—the "CD" part:

Overlap

Note: there is no overlap.

The Node.js projects don't have Terraform files, and the Terraform projects aren't Node.js.

Question: do you want me to provide the GitHub Actions for "CD" for the Node.js projects even though they don't have any Terraform files, or provide the GitHub Actions for "CD" for the Terraform projects even though they're not Node.js? Or both?

nutjob4life commented 5 months ago

@nutjob4life to query @anilnatha which of the above "CI" repositories is ripest for CD and to provide the Terraform files + the GitHub Action to run them for CD.

nutjob4life commented 5 months ago

@anilnatha consider this your official query: of the repositories under "CI Part" in this comment, which one would you like me to ① create some Terraform files for and ② add GitHub Actions to use those Terraform files to make an automatic dev-tier deployment?

anilnatha commented 5 months ago

@nutjob4life None of the wds-* repos require terraform, and atlas is the repo for the PDS IMG Atlas Product Search tool that is hosted by PDS IMG so don't need to create terraform for it either. I can't speak for the other repos.

anilnatha commented 5 months ago

I would imagine that we need terraform for the s3-browser-* repos. 🤷🏽

@eddiesarevalo ?

jordanpadams commented 5 months ago

@anilnatha to rephrase the question, if we wanted to be able to view and/or test what you and Eddie are currently working on, what repository would benefit from continuous deployment and/or containerization? Note: This may change in the future once we get wordpress in the loop.

anilnatha commented 5 months ago

I think it's in our best interest to focus on flushing out the cloud architecture I initially proposed and building out the portal-wp repo. We don't have to have the wordpress architecture solidified right now, but if we could at least resolve the issues with building out the portals frontend containers and getting those served through CI/CD, it would be a big step forward.

nutjob4life commented 5 months ago

Okay so is the idea to make portal-wp be like a template repo that you duplicate so you can stand-up Wordpress sites easily or more as just a "model", as in, "here's how you'd do it as a fully-fleshed example"?

jordanpadams commented 5 months ago

@nutjob4life this will be the one and only. we don't care about other wordpress sites and how to deploy this. this can be as application specific as needed.

tloubrieu-jpl commented 5 months ago

The build is broken, Sean will try to fix it since @anilnatha is busy and he wants to learn that.

jordanpadams commented 5 months ago

📆 06/2024 status: Delayed 2 additional sprints due to increased scope. No impact on other tasks. Primary task for engineer for this build.

jordanpadams commented 4 months ago

Sprint status: Final updates to dockerfile completed. Awaiting inputs from Anil to fix linter issues.

nutjob4life commented 4 months ago

For everyone tracking this:

@anilnatha and I had a productive meeting wherein I was able to run the portal-wp software on my own Mac Studio for the first time. The path was a bit rocky but this is good progress! The closest I've gotten so far 🎉

Anil, feel free to add your thoughts!

There are a few outstanding hurdles that stand in the way of closing this out for the sprint ending on 2024-08-01, and this'll have to be delayed yet again 😕

Publication of Node packages to npmjs.org

Currently, portal-wp depends on wds-react which in turn depends on wds. Neither wds-react nor wds have been published to the Node registry, npmjs.org. For now, they are "developer" packages that a local developer can use npm link to resolve.

In production, we'll want these to be fully fleshed out and be "delivered" packages that portal-wp will be able to take advantage of.

Build Errors

Prior to deploying a Node-based package, a developer (or delivery automation) would do npm run build to generate the dist directory to contain all of the final HTML, JS, CSS, and etc., assets that comprise the application. This dist directory then can be served at the htdocs of Apache HTTPD, as the web root of Nginx, be sent to an S3 bucket and served by AWS, etc.

However, the npm run build is currently failing in the portal-wp/apps/frontend (17 errors in 5 files). This is a blocker.

API Proxy

The Vite system leveraged by portal-wp uses its built-in support for proxies in order to simplify development. This enables a Node application running in the browser to make calls to a remote API without concern for CORS problems. In portal-wp, Vite sets up a proxy for /api which is transparently routed to the PDS Search API.

It is not clear at this time if Vite's proxy support will work in a deployed version of the portal-wp application. Experimentation and testing is required—but so is getting to a successful npm run build 😉

Deployment Morphology

An unanswered question is what a deployment of portal-wp will look like. As mentioned, a built Node application consists of purely static assets and all dynamic behavior happens in the browser. No server-side processes are necessary.

We (well, @nutjob4life and @anilnatha at least) do not know what the desired deployment shape is:

This is an open question at this time and we look forward to the discussion. Note that the current incarnation of portal-wp does not include WordPress and is strictly a client-side application. If a future version requires WordPress, this will obviously alter the deployment morphology. But we're not at that stage yet!

jordanpadams commented 3 months ago

📆 07/2024 status: Delayed additional sprint due to unforeseen blocks. No impact on other tasks. Primary task for engineer for this build.

nutjob4life commented 3 months ago

@anilnatha could you take a look at this? I'm starting off from a clean state but get an error … I think we're getting closer! Hopefully a quick fix!

@anilnatha could you take a look at this? I'm starting off from a clean state but get an error … I think we're getting closer!
```console
$ cd /tmp
$ rm -rf clean-slate && mkdir clean-slate
$ cd clean-slate
$ git clone --quiet git@github.com:NASA-PDS/wds.git
$ cd wds
$ nvm use
Found '/tmp/clean-slate/wds/.nvmrc' with version <lts/hydrogen>
Now using node v18.20.4 (npm v10.7.0)
$ npm clean-install && npm run build:css && npm run build:icons && npm link

added 191 packages, and audited 192 packages in 590ms

45 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

> @nasapds/wds@0.0.1 build:css
> rm -rf css && sass scss:css

> @nasapds/wds@0.0.1 build:icons
> node scripts/build-icons.js

Generated icons.js file with 69 icons

added 1 package, and audited 3 packages in 371ms

found 0 vulnerabilities
$ cd ..
$ git clone --quiet git@github.com:NASA-PDS/wds-react.git
$ cd wds-react
$ nvm use
Found '/tmp/clean-slate/wds-react/.nvmrc' with version <lts/iron>
Now using node v20.16.0 (npm v10.8.1)
$ npm clean-install && npm link @nasapds/wds
npm warn deprecated @npmcli/move-file@1.1.2: This functionality has been moved to @npmcli/fs
npm warn deprecated @npmcli/move-file@2.0.1: This functionality has been moved to @npmcli/fs

added 1588 packages, and audited 1589 packages in 8s

290 packages are looking for funding
  run `npm fund` for details

8 vulnerabilities (1 moderate, 7 high)

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
npm warn deprecated @npmcli/move-file@1.1.2: This functionality has been moved to @npmcli/fs
npm warn deprecated @npmcli/move-file@2.0.1: This functionality has been moved to @npmcli/fs

added 1 package, and audited 1591 packages in 1s

290 packages are looking for funding
  run `npm fund` for details

8 vulnerabilities (1 moderate, 7 high)

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
$ npm run build-icons 

> @nasapds/wds-react@0.1.0 build-icons
> svgr --jsx-runtime automatic -d ./src/components/Icons/ node_modules/@nasapds/wds/icons/

node_modules/@nasapds/wds/icons/arrow-circle-up.svg -> src/components/Icons/ArrowCircleUp.tsx
node_modules/@nasapds/wds/icons/arrow-circle-down.svg -> src/components/Icons/ArrowCircleDown.tsx
node_modules/@nasapds/wds/icons/arrow-filled-down.svg -> src/components/Icons/ArrowFilledDown.tsx
…
node_modules/@nasapds/wds/icons/view-grid.svg -> src/components/Icons/ViewGrid.tsx
node_modules/@nasapds/wds/icons/view-mosaic.svg -> src/components/Icons/ViewMosaic.tsx
/private/tmp/clean-slate/wds-react/node_modules/@svgr/cli/dist/index.js:435
    throw error;
    ^

TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received an instance of Object
    at Object.extname (node:path:1389:5)
    at /private/tmp/clean-slate/wds-react/icon-index-template.cjs:5:51
    at Array.map (<anonymous>)
    at defaultIndexTemplate (/private/tmp/clean-slate/wds-react/icon-index-template.cjs:4:35)
    at generateIndex (/private/tmp/clean-slate/wds-react/node_modules/@svgr/cli/dist/index.js:216:25)
    at handle (/private/tmp/clean-slate/wds-react/node_modules/@svgr/cli/dist/index.js:253:17)
    at async /private/tmp/clean-slate/wds-react/node_modules/@svgr/cli/dist/index.js:268:7
    at async Promise.all (index 0)
    at async dirCommand (/private/tmp/clean-slate/wds-react/node_modules/@svgr/cli/dist/index.js:264:3)
    at async run (/private/tmp/clean-slate/wds-react/node_modules/@svgr/cli/dist/index.js:431:3) {
  code: 'ERR_INVALID_ARG_TYPE'
}

Node.js v20.16.0
$ echo 🤔
🤔
anilnatha commented 3 months ago

@nutjob4life The icons do still build when the npm run build-icons command is issued. The issue you encountered is something that has been plaguing us since Eddie first implemented the icon build. We haven't been able to resolve the issue yet. :( If you look in the distribution folder, you'll find that the icons do in fact get rendered correctly [cjs|esm]/components/icons/

CC: @eddiesarevalo

anilnatha commented 3 months ago

@nutjob4life I created a ticket to track the issue you reported, but it shouldn't be a blocker. Let me know if you are in fact blocked though so I can prioritize the bug fix. Reference: https://github.com/NASA-PDS/wds-react/issues/52

anilnatha commented 3 months ago

@nutjob4life I fixed the build-icons issue, PR to get that reviewed and merged is here.

nutjob4life commented 3 months ago

@anilnatha getting closer! I'm not up to the docker compose up step:

 => => resolve docker.io/library/wordpress:6.3.1-php8.2-apache@sha256:34c77a88700b856ad2271e386de6214fba4088dfc5b94  0.0s
 => => sha256:61c2e8cf23ab24df88a251a33d11d42677dca7e4eba0a028c1838d6f39c44843 4.71kB / 4.71kB                       0.0s
 => => sha256:34c77a88700b856ad2271e386de6214fba4088dfc5b944067aafebad8a8f86b1 1.86kB / 1.86kB                       0.0s
 => => sha256:53be2feed7f7f8b2451fe62d7bf250b0d7455eed5ac436afe2f57e3b23020e00 18.37kB / 18.37kB                     0.0s
 => [backend internal] load build context                                                                            0.0s
 => => transferring context: 2B                                                                                      0.0s
 => CACHED [backend 2/6] WORKDIR /usr/src/wordpress/wp-content                                                       0.0s
 => ERROR [backend 3/6] COPY ./data/wp-content/themes/ themes/                                                       0.0s
 => ERROR [backend 4/6] COPY ./data/wp-content/plugins/ plugins/                                                     0.0s
------
 > [backend 3/6] COPY ./data/wp-content/themes/ themes/:
------
------
 > [backend 4/6] COPY ./data/wp-content/plugins/ plugins/:
------
failed to solve: failed to compute cache key: failed to calculate checksum of ref 68b7b2b3-558e-4f9e-9818-1541749fbbb3::7byfdhiihsb66bojnnpvq6mmx: failed to walk /var/lib/docker/tmp/buildkit-mount3244437086/data/wp-content: lstat /var/lib/docker/tmp/buildkit-mount3244437086/data/wp-content: no such file or directory

What do you think?

anilnatha commented 3 months ago

@nutjob4life With where we are with the MVP I think we should skip trying to get the WordPress set up and focus on just getting the frontend application up and running. We will be starting on the wordpress set up soon, but it's not needed for the MVP.

Should I comment out the wordpress configuration(s)

nutjob4life commented 3 months ago

@anilnatha I was just following the instructions in the README, I swear! 😇

nutjob4life commented 3 months ago

@jordanpadams @tloubrieu-jpl I know we're trying to reduce the S3 buckets in use, but in order to turn this from a "technology demonstration" to an actual hosted app, I'll need a bucket. What's the process for procuring one?

jordanpadams commented 3 months ago

@nutjob4life I have no problem with all the buckets we need, just not stale/unused/undocumented buckets :-)

@viviant100 @sjoshi-jpl how should we put requests in for this? If this is in MCP-Dev, is there an FAQ somewhere we can direct users when they ask "How can I create an S3 bucket?"

nutjob4life commented 3 months ago

@jordanpadams copy that! 😁

jordanpadams commented 3 months ago

@sjoshi-jpl @viviant100 @tloubrieu-jpl ping! can we get a bucket created ASAP in Dev? This is both for @nutjob4life testing, as well as @eddiesarevalo to deploy manually for testing

nutjob4life commented 3 months ago

@viviant100 is fast 🏎️💨

The bucket was just created, s3://pds-portal-demo … thanks much!

nutjob4life commented 3 months ago

When running terraform apply:

arn:aws:sts::441083951559:assumed-role/mcp-tenantDeveloper/skelly is not authorized to perform: s3:PutBucketPolicy on resource: "arn:aws:s3:::pds-portal-demo" because public policies are blocked by the BlockPublicPolicy block public access setting

This is using MCP credentials from "Kion":

jordanpadams commented 3 months ago

📆 08/2024 status: Work continues. On schedule to have a deployed NPM project by end of sprint.

tloubrieu-jpl commented 2 months ago

@sjoshi-jpl is helping on that.

Developers can not PUT to s3 buckets. We will need support from SAs on that.

nutjob4life commented 2 months ago

Working "by hand", finally have first success using terraform apply to get files to appear in S3!

In order to still make a "proof of concept", I'll work next to see if I can hook up CloudFront. Meanwhile work on OIDC continues in parallel.

nutjob4life commented 2 months ago

CloudFront update:

I'm not exactly sure what's wrong. In CloudFront instance E18VQ4K51WT0LV, the origin s3_webmod_bkt is connected to pds-portal-demo.s3.us-west-2.amazonaws.com and I've added origin path /portal.

But visiting https://pds-sit.mcp.nasa.gov/portal gives

<Error>
  <Code>AccessDenied</Code>
  <Message>Access Denied</Message>
  <RequestId>2RZ44QYTN6MSYYXB</RequestId>
  <HostId>
    tqqqiPJRIWD96jInhV7IcHLzl0KpPy3MD+GjnUMoRh2XU9zVHb/ELh2KA3X0bFCGI1MzYkMtYe8=
  </HostId>
</Error>

Gonna need some AWS experts to chime in 🙏 @viviant100 @sjoshi-jpl @tloubrieu-jpl

nutjob4life commented 2 months ago

Thanks to @viviant100 in this thread I can get closer—but thanks to JPL Cyber Security I can't actually access VPN (or even public .jpl.nasa.gov sites), so will put this on hold until they unblock me.

nutjob4life commented 2 months ago

Okay, once again kudos to @viviant100 for getting things configured. I'm happy to report that visiting https://pds-sit.mcp.nasa.gov/portal/ (note you must use the trailing /) finally gives the expected portal-wp web app. Of course, this is still done using the hand-run terraform apply, and it's using an older version of the app.

Next steps are to ① use a more recent update to the develop tree of portal-wp; ② get the OIDC authentication working so GitHub Actions can do this; ③ consider production deployment scenarios; ④ foresee the unforeseen issues.

Screenshot 2024-09-13 at 9 17 28 AM
nutjob4life commented 2 months ago

@anilnatha @eddiesarevalo I've updated my local wds, wds-react, and portal-wp directories from GitHub with the latest develop branch commits on each.

In wds, I did nvm use && npm clean-install && npm run build:css && npm run build:icons && npm link. So far so good.

In wds-react, I did nvm use && npm clean-install && npm link && npm link @nasapds/wds && npm run build-icons. Also works.

In portal-wp/apps/frontend, though, is where I run into some trouble. After doing nvm use && npm clean-install && npm link && npm link @nasapds/wds && npm link @nasapds/wds-react && npm run build, the run build steps fails with:

> pds-portal-frontend@0.0.0 build
> tsc && vite build

src/pages/search/index.tsx:35:3 - error TS2305: Module '"@nasapds/wds-react"' has no exported member 'Button'.

35   Button,
     ~~~~~~

src/pages/search/index.tsx:36:3 - error TS2305: Module '"@nasapds/wds-react"' has no exported member 'FeaturedLink'.

36   FeaturedLink,
     ~~~~~~~~~~~~

src/pages/search/index.tsx:37:3 - error TS2305: Module '"@nasapds/wds-react"' has no exported member 'FeaturedLinkDetails'.

37   FeaturedLinkDetails,
     ~~~~~~~~~~~~~~~~~~~

src/pages/search/index.tsx:38:3 - error TS2305: Module '"@nasapds/wds-react"' has no exported member 'FeaturedLinkDetailsVariant'.

38   FeaturedLinkDetailsVariant,
     ~~~~~~~~~~~~~~~~~~~~~~~~~~

src/pages/search/index.tsx:700:21 - error TS2322: Type '"search"' is not assignable to type '"filled" | "outlined" | "standard" | undefined'.

700                     variant="search"
                        ~~~~~~~

  ../../../wds-react/node_modules/@mui/material/TextField/TextField.d.ts:201:3
    201   variant: 'filled';
          ~~~~~~~
    The expected type comes from property 'variant' which is declared here on type 'IntrinsicAttributes & TextFieldProps'

Found 5 errors in the same file, starting at: src/pages/search/index.tsx:35

Ideas? Thanks in advance!

nutjob4life commented 2 months ago

Thanks @eddiesarevalo for your help 🎉

The next issue now is: when you visit https://pds-sit.mcp.nasa.gov/portal/ the line <script type="module" crossorigin src="/assets/index-Cqbnc77p.js"></script> in index.html tries to load https://pds-sit.mcp.nasa.gov/assets/index-Cqbnc77p.js instead of https://pds-sit.mcp.nasa.gov/portal/assets/index-Cqbnc77p.js with /portal/ in it

This was working before, so I think something might've changed

tloubrieu-jpl commented 2 months ago

Waiting for @sjoshi-jpl regarding the status of the ODC authentication. Will see that next week.

jordanpadams commented 1 month ago

📆 09/2024 status: delayed several sprints working out OIDC privileges. operations task. no impact on delivery.

jordanpadams commented 4 weeks ago

Calling this phase completed. Moving remaining sub-tasks to B15.1 to work with Drupal CICD