Develop Unstable GitHub Action CI

[jordanpadams]

💡 Description

Merge to develop branch and run unstable build

[nutjob4life]

Based on this answer, we'll use the Roundup Action for the CI part of things, and a future action for CD.

Roundup Action supports Maven- and Python-based repositories for unit testing, documentation generation, and publication of artifacts to Sonatype (for Maven) and PyPI (for Python). It's object-oriented, meaning that each step it executes is generic and doesn't know what the underlying repository is, be that Maven or Python. So it provides a framework for plugging in additional kinds of repositories.

Enter Node.js.

Since I'm not a Node.js developer, though, I'll need some help, so I'm roping in @anilnatha to provide some insight.

Roundup Action "Steps"

The Roundup Action executes a series of steps which are configurable and depend on the "assembly" being run. An assembly defines the steps and the order in which they're run. There's the NoOpAssembly which has zero steps. There's the EnvironmentalAssembly which takes its steps from an environment variable. And there's the PDSAssembly which is what we use and has the following steps, in order:

• preparation (StepName.preparation)
• unit test (StepName.unitTest)
• integration test (StepName.integrationTest)
• docs (StepName.docs)
• version bump (StepName.versionBump)
• build (StepName.build)
• artifact publication (StepName.artifactPublication)
• requirements (StepName.requirements)
• changelog (StepName.changeLog)
• GitHub release (StepName.githubRelease)
• doc publication (StepName.docPublication)
• version commit (StepName.versionCommit)
• cleanup (StepName.cleanup)

An assembly can be either "stable" or "unstable". The StablePDSAssembly is stable; UnstablePDSSAssembly is unstable. You can see in your repository's stable-cicd.yaml and unstable-cicd.yaml how this flag is set. A stable assembly is for an official, stable release of the software and releases an official artifact to the correct artifactory, publishes documentation, updates the changelog and issues, and gets the next development version ready for work. An unstable release creates a -SNAPSHOT release (for Maven) or a test.pypi.org release (for Python) and deletes prior unstable releases.

An assembly is associated with a Context that maps the step enumerated StepNames to the Step classes that implement each step and also provide environment variables and command-line arguments.

The Step Classes

Class Step is abstract and expect concrete subclasses to implement the execute method to do the work when it's run. For example, the Maven class _UnitTestStep's execute method does mvn test while the Python class _UnitTestStep's execute tries to run tox -e py39 but if that fails falls back to python setup.py test.

As another example, the Maven class _BuildStep does mvn compile and the Python class _BuildStep does python setup.py bdist_wheel.

Some steps are common between Maven and Python. The ChangeLogStep is one such step. It uses the GitHub Changelog Generator to automatically create our CHANGELOG.md file. The RequirementsStep uses our own lasso-requirements to create our requirements report. And so on.

Node.js Classes

Our task therefore is to define for Node.js repositories what it will mean to do the following.

Preparation

For Maven repositories, the preparation step involves creating the settings.xml file used by Maven and also generating the GPG key in order to sign artifacts. For Python, this creates a Python venv and installs the package being rounded up into it.

❓ Does NPM use signed artifacts? What preparation steps are needed for Node.js repositories? Is it npm install . or something more involved?

Unit Test

As mentioned above, for Maven this is mvn test and for Python it's tox -e py39 etc.

❓ Is this just npm test?

Integration Test

Neither of the current Maven and Python Roundups do anything for the integration test step so we'll skip it for Node.js as well.

Documentation Generation

In Maven repositories, we automatically generate documentation with mvn package site site:stage and with Python we do sphinx-build -a -b html docs/source docs/build.

❓ Does Node.js have automatically-generated documentation and if so, how?

Version Bumping

At this point in the roundup, we're ready to increment the version number of the project. This only happens for stable builds, so unstable builds make this a no-op.

In Maven repositories, we use git describe --tags --abbrev=0 --match release/* in order to find out if we have a release/X.Y.Z kind of push. We then add this version label to an open bugs that remain in the project. Maven roundups then run mvn -DgenerateBackupPoms=false -DnewVersion=X.Y.Z versions:set which updates the pom.xml with the new version number.

In Python repositories, we use the same git command to get the release/X.Y.Z tag and add the version label to open bugs. We then find the VERSION.txt file and set its contents to X.Y.Z.

❓ Does npm provide a command to update the version key in package.json? If not, we'll update package.json directly.

Build Step

As mentioned above, Maven does mvn compile and Python does python setup.py bdist_wheel. These generate the corresponding artifacts target/whatever-X.Y.Z.jar (among others) and dist/whatever-X.Y.Z.whl.

❓ What's the equivalent for Node.js?

Artifact Publication

For Maven this means running mvn --activate-profiles release clean package site deploy to create the source, documentation, and binary jar files to Sonatype for a stable release, and mvn clean site deploy for an unstable release. For Python, it's twine upload --repository-url URL with URL = https://upload.pypi.org/legacy/ for a stable release or https://test.pypi.org/legacy/ for an unstable release.

❓ What will this mean for Node.js? Are there separate artifactories for Node.js for stable and unstable releases is there just one and there are somehow marked differently?

GitHub Release

In addition to publishing artifacts at the language-specific repository (Sonatype for Maven, PyPI for Python, https://www.npmjs.com/ for Node.js), we also push artifacts to the release page on GitHub.

In Maven repositories, we do the following:

• Delete all release tags that have SNAPSHOT in their name
• For unstable releases, use lasso-releasers to publish a new SNAPSHOT release tag, then clean up any leftover release/* tags
• For stable releases, use git tag --annotate to tag the release as X.Y.Z (from release/X.Y.Z) and then use lasso-releasers to publish a new official release tag

In Python repositories, it's similar:

• Delete all release tags with dev in their name
• For unstable releases, use lass-releasers to publish a new dev release, then clean up any leftover release/* tags
• For stable releases, use git tag --annotate to tag the release as X.Y.Z (from release/X.Y.Z) and then use lasso-releasers to publish a new official release tag

❓ For Node.js, we will have to update lasso-releasers with its own support for publishing artifacts to GitHub. Perhaps @anilnatha can work with @tloubrieu-jpl on this?

Documentation Publication

Documentation publication is a generic step but Maven and Python subclass this to provide the location of the documentation directory, since the two kinds of repositories end up depositing their files in different locations.

❓ As for "Documentation Generation" above, does Node.js have automatically-generated documentation?

Version Commit

The penultimate step in the PDS assembly is to commit any changed version files. This only happens in stable assemblies, so for unstable roundups, this is a no-op.

In Maven, this just means committing the pom.xml file (and any pom.xml files in subdirectories for multi-module repositories). In Python, this is just committing the VERSION.txt file.

❓ In Node.js, is this as simple as committing the package.json?

Cleanup

Cleanup is the final step in the PDS assembly and only happens for stable roundups.

In Maven-land, this means finding the newly-committed version in pom.xml and adding one to the minor version number and resetting the micro number to zero, so if we just released stable version 1.2.3, the next version is 1.3.0. This new version is written to the pom.xml (and any pom.xml files in subdirectories for multi-module repositories) along with -SNAPSHOT so the next cycle of development can begin. These pom.xml files are then committed.

In Python-land, we do the same thing but for the one VERSION.txt file.

❓ Can we just update version in package.json as well and then commit it?

Final Questions

What's the difference between npm, npx, and yarn—and why are there three?

[eddiesarevalo]

@nutjob4life ### I'll include my answers inline below:

Based on this answer, we'll use the Roundup Action for the CI part of things, and a future action for CD.

Roundup Action supports Maven- and Python-based repositories for unit testing, documentation generation, and publication of artifacts to Sonatype (for Maven) and PyPI (for Python). It's object-oriented, meaning that each step it executes is generic and doesn't know what the underlying repository is, be that Maven or Python. So it provides a framework for plugging in additional kinds of repositories.

Enter Node.js.

Since I'm not a Node.js developer, though, I'll need some help, so I'm roping in @anilnatha to provide some insight.

Roundup Action "Steps"

The Roundup Action executes a series of steps which are configurable and depend on the "assembly" being run. An assembly defines the steps and the order in which they're run. There's the NoOpAssembly which has zero steps. There's the EnvironmentalAssembly which takes its steps from an environment variable. And there's the PDSAssembly which is what we use and has the following steps, in order:

• preparation (StepName.preparation)
• unit test (StepName.unitTest)
• integration test (StepName.integrationTest)
• docs (StepName.docs)
• version bump (StepName.versionBump)
• build (StepName.build)
• artifact publication (StepName.artifactPublication)
• requirements (StepName.requirements)
• changelog (StepName.changeLog)
• GitHub release (StepName.githubRelease)
• doc publication (StepName.docPublication)
• version commit (StepName.versionCommit)
• cleanup (StepName.cleanup)

An assembly can be either "stable" or "unstable". The StablePDSAssembly is stable; UnstablePDSSAssembly is unstable. You can see in your repository's stable-cicd.yaml and unstable-cicd.yaml how this flag is set. A stable assembly is for an official, stable release of the software and releases an official artifact to the correct artifactory, publishes documentation, updates the changelog and issues, and gets the next development version ready for work. An unstable release creates a -SNAPSHOT release (for Maven) or a test.pypi.org release (for Python) and deletes prior unstable releases.

An assembly is associated with a Context that maps the step enumerated StepNames to the Step classes that implement each step and also provide environment variables and command-line arguments.

The Step Classes

Class Step is abstract and expect concrete subclasses to implement the execute method to do the work when it's run. For example, the Maven class _UnitTestStep's execute method does mvn test while the Python class _UnitTestStep's execute tries to run tox -e py39 but if that fails falls back to python setup.py test.

As another example, the Maven class _BuildStep does mvn compile and the Python class _BuildStep does python setup.py bdist_wheel.

Some steps are common between Maven and Python. The ChangeLogStep is one such step. It uses the GitHub Changelog Generator to automatically create our CHANGELOG.md file. The RequirementsStep uses our own lasso-requirements to create our requirements report. And so on.

Node.js Classes

Our task therefore is to define for Node.js repositories what it will mean to do the following.

Preparation

For Maven repositories, the preparation step involves creating the settings.xml file used by Maven and also generating the GPG key in order to sign artifacts. For Python, this creates a Python venv and installs the package being rounded up into it.

❓ Does NPM use signed artifacts? What preparation steps are needed for Node.js repositories? Is it npm install . or something more involved?

I’m not completely familiar with Maven repositories and artifacts but for node.js usually just running npm install is enough. Of course it depends on the project. Npm install installs all the dependencies and creates the npm-package.lock file which contains the exact versioned dependency tree of what was installed.

You can also add your own scripts to the package.json if you need to do something else.

As for integrity checks, since they’re coming from the same source npmjs.com we’re not so worried about MIM attacks as with JARs downloaded from different mirrors. Although npm might do some key verification under the hood.

Unit Test

As mentioned above, for Maven this is mvn test and for Python it's tox -e py39 etc.

❓ Is this just npm test?

There are various testing frameworks for different projects in npm. They are usually hooked into “npm test” although that command might change depending on what is included inside the package.json scripts object.

One popular example for testing react projects is Jest https://jestjs.io/docs/getting-started The script is added in the package.json file

{
  "scripts": {
    "test": "jest"
  }
}

So running npm test would run “jest”

Integration Test

Neither of the current Maven and Python Roundups do anything for the integration test step so we'll skip it for Node.js as well.

Documentation Generation

In Maven repositories, we automatically generate documentation with mvn package site site:stage and with Python we do sphinx-build -a -b html docs/source docs/build.

❓ Does Node.js have automatically-generated documentation and if so, how?

In the same way as for testing there are various documentation creation frameworks in npm. They can have various ways to run the documentation build. Usually if it exists it would be in the scripts section the package.json file.

One popular example for react is jsdoc https://jsdoc.app The script is added in the package.json file

"scripts": {
  ...
    "docs": "jsdoc -c jsdoc.conf.json"
  ...
}

So to run it would be the command npm docs although you can always change the word “docs” to whatever you wish.

Version Bumping

At this point in the roundup, we're ready to increment the version number of the project. This only happens for stable builds, so unstable builds make this a no-op.

In Maven repositories, we use git describe --tags --abbrev=0 --match release/* in order to find out if we have a release/X.Y.Z kind of push. We then add this version label to an open bugs that remain in the project. Maven roundups then run mvn -DgenerateBackupPoms=false -DnewVersion=X.Y.Z versions:set which updates the pom.xml with the new version number.

In Python repositories, we use the same git command to get the release/X.Y.Z tag and add the version label to open bugs. We then find the VERSION.txt file and set its contents to X.Y.Z.

❓ Does npm provide a command to update the version key in package.json? If not, we'll update package.json directly.

There are some libraries that can handle automatically updating the version on the package.json but I haven’t used them so I can’t vouch for their reliability. I usually set it manually. Npm packages use semantic versioning spec otherwise known as “semver” https://docs.npmjs.com/about-semantic-versioning

Build Step

As mentioned above, Maven does mvn compile and Python does python setup.py bdist_wheel. These generate the corresponding artifacts target/whatever-X.Y.Z.jar (among others) and dist/whatever-X.Y.Z.whl.

❓ What's the equivalent for Node.js?

This probably correlates to npm build It will spit out the distribution files that are to be uploaded to the server as you would do with jars.

Artifact Publication

For Maven this means running mvn --activate-profiles release clean package site deploy to create the source, documentation, and binary jar files to Sonatype for a stable release, and mvn clean site deploy for an unstable release. For Python, it's twine upload --repository-url URL with URL = https://upload.pypi.org/legacy/ for a stable release or https://test.pypi.org/legacy/ for an unstable release.

❓ What will this mean for Node.js? Are there separate artifactories for Node.js for stable and unstable releases is there just one and there are somehow marked differently?

On the npm js website you can deploy various versions with different tags with only the official one being automatically downloaded with npm install

For example if we want unstable tag to differentiate from stable versions then when updating the version in the /package.json file you can use the X.X.X semver syntax to set a stable version or X.X.X-unstable.X semver syntax for an unstable version.

Use npm publish to publish a stable version or use npm publish --tag unstable to publish an unstable version.

If you want to test an unstable version make sure to install the unstable version instead of the latest version. For example use npm install @nasapds/<unstable_project>@unstable for the latest unstable version or npm install @nasapds/<unstable_project>@X.X.X-unstable.X for a specific version.

An example is at https://www.npmjs.com/package/@nasapds/pds-wds-react?activeTab=versions we have normal stable versions and beta versions. All in the version history but with different tags.

GitHub Release

In addition to publishing artifacts at the language-specific repository (Sonatype for Maven, PyPI for Python, https://www.npmjs.com/ for Node.js), we also push artifacts to the release page on GitHub.

In Maven repositories, we do the following:

• Delete all release tags that have SNAPSHOT in their name
• For unstable releases, use lasso-releasers to publish a new SNAPSHOT release tag, then clean up any leftover release/* tags
• For stable releases, use git tag --annotate to tag the release as X.Y.Z (from release/X.Y.Z) and then use lasso-releasers to publish a new official release tag

In Python repositories, it's similar:

• Delete all release tags with dev in their name
• For unstable releases, use lass-releasers to publish a new dev release, then clean up any leftover release/* tags
• For stable releases, use git tag --annotate to tag the release as X.Y.Z (from release/X.Y.Z) and then use lasso-releasers to publish a new official release tag

❓ For Node.js, we will have to update lasso-releasers with its own support for publishing artifacts to GitHub. Perhaps @anilnatha can work with @tloubrieu-jpl on this?

I’m not sure about this one. I’ll leave it to @tloubrieu-jpl and @anilnatha

Documentation Publication

Documentation publication is a generic step but Maven and Python subclass this to provide the location of the documentation directory, since the two kinds of repositories end up depositing their files in different locations.

❓ As for "Documentation Generation" above, does Node.js have automatically-generated documentation?

As with the third question about documentation the generated docs will be available in a directory in the project. This can then be put anywhere you wish. The location of where it gets placed though is up to the generator and if it has a way to set a custom location.

Version Commit

The penultimate step in the PDS assembly is to commit any changed version files. This only happens in stable assemblies, so for unstable roundups, this is a no-op.

In Maven, this just means committing the pom.xml file (and any pom.xml files in subdirectories for multi-module repositories). In Python, this is just committing the VERSION.txt file.

❓ In Node.js, is this as simple as committing the package.json?

I'm a bit confused on this one. Do you mean how does nodejs know about the commits related to a version change?

Cleanup

Cleanup is the final step in the PDS assembly and only happens for stable roundups.

In Maven-land, this means finding the newly-committed version in pom.xml and adding one to the minor version number and resetting the micro number to zero, so if we just released stable version 1.2.3, the next version is 1.3.0. This new version is written to the pom.xml (and any pom.xml files in subdirectories for multi-module repositories) along with -SNAPSHOT so the next cycle of development can begin. These pom.xml files are then committed.

In Python-land, we do the same thing but for the one VERSION.txt file.

❓ Can we just update version in package.json as well and then commit it?

Yes updating the “version” value in the package.json file should be enough to let npmjs know this is a different version. Remember to use semver! https://docs.npmjs.com/about-semantic-versioning

Final Questions

What's the difference between npm, npx, and yarn—and why are there three?

Npm is is a package manager that handles downloading and managing projects from the online repository npmjs.com. Its installed along with with node.js from http://nodejs.org

Npx does the same as npm but instead of downloading the project it can run it without saving it to your disk. It can even run projects straight off GitHub without having to install. It is also installed with node.js from http://nodejs.org

Yarn is another package manager that does the same thing as npm such as handling downloads and managing projects from the online repository npmjs.com. It is meant to be an improvement over npm but is not as popular as npm. It has to be installed separately from node.js and npm

npm and npx are included in the nodejs download and they fulfill different roles. Yarn is an alternative to npm. nodejs isn't limited to these 3 CLI's but these are three common ones indeed.

[nutjob4life]

As mentioned above, Maven does mvn compile and Python does python setup.py bdist_wheel. These generate the corresponding artifacts target/whatever-X.Y.Z.jar (among others) and dist/whatever-X.Y.Z.whl. ❓ What's the equivalent for Node.js?

This probably correlates to npm build It will spit out the distribution files that are to be uploaded to the server as you would do with jars.

Thanks @eddiesarevalo! One quick question: does npm build rely on some earlier invocation or dependency? I get Unknown command: "build":

$ cat package.json
{
  "name": "mynodetest",
  "version": "0.0.0",
  "description": "Just a test for node.js",
  "main": "index.js",
  "scripts": {
    "test": "jest",
    "docs": "jsdoc -c jsdoc.conf.json"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/NASA-PDS/mynodetest.git"
  },
  "keywords": [
    "testing"
  ],
  "author": "nutjob4life",
  "license": "Apache-2.0",
  "bugs": {
    "url": "https://github.com/NASA-PDS/mynodetest/issues"
  },
  "homepage": "https://github.com/NASA-PDS/mynodetest",
  "dependencies": {},
  "devDependencies": {
    "jest": "29.7",
    "jsdoc": "^4.0.2"
  }
}
$ cat index.js 
console.log('hello, 🌎');
$ node index.js 
hello, 🌎
$ npm build
Unknown command: "build"

To see a list of supported npm commands, run:
  npm help

[nutjob4life]

Node.js support is really what this ticket is all about now!

[tloubrieu-jpl]

@nutjob4life requested help from @anilnatha and @eddiesarevalo .

[tloubrieu-jpl]

Still blocked.

[anilnatha]

@nutjob4life (fyi) I'm trying to ascertain what we need to do/try to get this working, as soon as I have something, I'll report back.

[nutjob4life]

@anilnatha I don't think you need to do anything. You already figured it out!

@tloubrieu-jpl this is no longer blocked.

[anilnatha]

Ahhh, I thought there was something else I needed to look into, was playing catch up with the messages in this ticket. @nutjob4life

[anilnatha]

Also, @nutjob4life , as part of our deployments, we should be running npm clean-install in lieu of npm install if we aren't already.

[tloubrieu-jpl]

This is done, @nutjob4life will create a PR after he develops the stable part of roundup for nodejs.

[nutjob4life]

I believe this pull request will close the following issues

• 63

• 64

• 66

• 69

[jordanpadams]

Changed this task to be just CI for now, and will create new task to focus on CD