`nx release` incorrectly derives next prerelease version based on conventional commits

lorenzodejong commented 8 months ago

Current Behavior

When using the conventional-commit specifier and creating a prerelease the next version always derives it to be a patch change. Even though we created a conventional commit which indicates a minor or major change.

Expected Behavior

The derived version for a prerelease should also take minor and major changes into account when resolving to the next prerelease version.

Essentially, creating a prerelease should already indicate the next version which is going to be released. This specifically helps when implementing CI/CD workflows which allows contributors to test their prereleases from a PR before integrating into the main branch.

GitHub Repo

No response

Steps to Reproduce

Run: npx create-nx-workspace@18.0.7 nx-version-example

✔ Which stack do you want to use? · none ✔ Package-based monorepo, integrated monorepo, or standalone project? · integrated ✔ Do you want Nx Cloud to make your CI fast? · skip

Run: nx g @nx/js:library sample-library-a --directory packages/sample-library-a

✔ Which unit test runner would you like to use? · jest ✔ Which bundler would you like to use to build the library? Choose 'none' to skip build setup. · tsc

Create a commit of all the previous changes
Run nx release --skip-publish --first-release, choose minor for the type of change
You should see the resulting version (and created commit/tag) point to 0.1.0
Create a small change somewhere in the library, for example in packages/sample-library-a/src/lib/sample-library-a.ts
Run: git add . && git commit -m 'feat!: this should become 1.0.0'
Run: nx release version --specifier prerelease --preid beta --dry-run

Observed behavior: the newly derived version is 0.1.1-beta.0. However because we created a breaking change using the feat! conventional commit syntax, i would expect the version to become 1.0.0-beta.0.

Nx Report

Node   : 18.13.0
OS     : darwin-arm64
pnpm   : 8.15.3

nx                 : 18.0.7
@nx/js             : 18.0.7
@nx/jest           : 18.0.7
@nx/linter         : 18.0.7
@nx/eslint         : 18.0.7
@nx/workspace      : 18.0.7
@nx/devkit         : 18.0.7
@nx/eslint-plugin  : 18.0.7
@nrwl/tao          : 18.0.7
typescript         : 5.3.3

Failure Logs

No response

Package Manager Version

No response

Operating System

[X] macOS
[ ] Linux
[ ] Windows
[ ] Other (Please specify)

Additional Information

No response

JamesHenry commented 8 months ago

@lorenzodejong Thanks a lot we'll take a look. One thing I would point out though is that the semver spec defines 0.x as unstable by nature, so I believe most tools would not class a breaking change within 0.x as requiring a bump to 1.0.0 https://semver.org/#spec-item-4 (and it in fact would be incorrect to do so, the bump to 1.0.0 is a significant milestone, and is the point at which your API stabilises, not the point at which you first have a breaking change in your project)

lorenzodejong commented 8 months ago

@JamesHenry that's a good point to consider. However actually this is how nx release currently determines the new version when initiating a "breaking change" in the non-stable (0.x.x) version range.

From the reproduction steps of my OP, if you would replace the command from step 6 to nx release version --dry-run, you would actually see that it currently resolves to version 1.0.0. Based on that consideration i would expect the prerelease for that specific change to follow the same approach, resulting in 1.0.0-beta.0.

However i do agree that any commit indicating a "breaking change" in the non-stable range should not necessarily result in a version bump to 1.0.0. In my mind there's two ways of approaching this:

The nx release configuration allows configuring this explicit behavior, so that a conventional commit doesn't bump a commit to the next major in the non-stable range
The current implementation is retained, however it's up to the maintainers of the repository to not include any breaking conventional commits up until the initial major release of the package(s)

fahslaj commented 7 months ago

Hi @lorenzodejong , the way that Nx Release handles the 'prerelease' identifier is the same as how npm version handles it - which is that it is equivalent to 'prepatch' if the current version is not a already prerelease, otherwise it increments the prerelease version (i.e. 0.1.0-beta.1 -> 0.1.0-beta.2). You can see this in action here in the node-semver package. In order the major version prerelease to be created, you would need to call nx release version --specifier premajor --preid beta --dry-run.

Zordrak commented 7 months ago

I don't think the way npm version handles this is necessarily relevant. If we are using npm version then we have all of the responsibility of determining the next version and constructing any potential prerelease version information. We have to effectively define and grok our own semver-update tokens from commit messages and implement them. Then when it comes to the npm version command we will know what type of version bump we require for our prerelease.

When we are having our version bumping calculation determined for us using conventional-commits, we have removed all of that version-bumping knowledge from our pipeline, depending instead on conventional commits. If we have to put all of the version logic back in so that in a prerelease we can inform nx what type of version bump we want, then we have made the conventional commits automation redundant, as we have to implement it anyway.

From a developer workflow perspective, when working on a new feature and performing testing with prerelease package versions, we need to inform the developer as to what the version of their package would be if they merged it today based upon the commit history that we will automatically determine the version number from. It is inappropriate for us to ask developers to merge changes and hope that they got the version bump they intended for, without any way to inform them prior to the release as to what their expectation should be. Since we want to use conventional commits to automate the decision making, we should not ask pipeline creators to reinvent conventional commit logic just for the purposes of the prerelease.

npm version is the command used to implement the bump, conventional commits is how we know what type of bump.

We need an appropriate mechanism to apply the conventional commits decision making, to the prerelease versioning.

I think the @lorenzodejong 's expectation that providing a pre-id, and no explicit specifier is perfectly aligned to the existing nx behaviour that not providing a release specifier defaults to conventional commit logic, and that the expected behaviour should be to create a prerelease with the same base version as would be created if pre-id were not specified.

fahslaj commented 7 months ago

@Zordrak Thank you for the clear write up. Nx Release's default behavior relies on the user to provide a specifier. If the specifier is not provided, then Nx Release will prompt the user. In this case, I think the existing behavior makes sense, as the user is explicitly picking the specifier.

However, when conventionalCommits is enabled, I definitely see the argument to interpret prerelease as you and @lorenzodejong described. As you mentioned clearly, it would be valuable in this case for Nx Release to pick what type of prerelease is necessary based on past commits.

Zordrak commented 7 months ago

I have created a resolution that should work for me - but the quality of it is a little embarrassing.

The existing code has all of the logic implementation that I need, but does not expose the outputs I need as they are only used as interim variables. Therefore to maintain 1:1 logic with the existing implementation I have effectively had to duplicate the existing logic into new files, only stripping out the logic that isn't needed, leaving quite a complicated and ugly implementation for something that is in essence quite simple.

Existing Behaviour

When using conventional commits, the existing implementation performs these steps:

Determine the "specifier" for each project's version bump via conventional-commits
Use the specifier to calculate a new version for the project
Write the new version to the project's package.json

When a pre-release is desired, the word "prerelease" is taken as an override specifier, preventing step 1, and hard-coding the input to step 2.

Desired Behaviour

The argument "prerelease" (or another word, to prevent collision with existing npm taxonomy) would not override the specifier, but tact as a behaviour modifier so that once the specifier has been determined it can be prefixed with "pre" (pre${specifier}), before being passed into step 2 as an input.

Determine the "specifier" for each project's version bump via conventional-commits
Prefix the specifier with the string pre and used the modified specifier to calculate a new version for the project
Write the new version to the project's package.json

I would be perfectly happy for this to be a separate argument so that the existing behaviour of the "prerelease" specifier remains in line with npm's expectations.

Proof of Concept Implementation

To implement this proof, I have effectively duplicated:

packages/js/src/generators/release-version/release-version.ts (as scripts/release-version.ts)
packages/nx/src/command-line/release/version.ts (as scripts/nx-release.ts)

scripts/release-version.ts

In the duplicated File 1, we only want the logic that is used to determine the conventional-commits derived bump specifier, therefore instead of providing VersionData it now provides SpecifierData. All of the logic that implements change on-disk is removed along with the switch cases that are not implementation via conventional-commits.

If I were modifying the existing implementation in-situ this would be very easy to implement as we would simply provide access to the data already calculated halfway through the function.

scripts/nx-release.ts

The content of File 2 is consumed into a custom release-initiator.

This performs the usual steps of:

versionData = releaseVersion(specifier: string|undefined);
releaseChangelog(versionData);
releasePublish();

However when provided with a preid argument, and without a specifier:

specifier = getSpecifierData();
versionData = releaseVersion(specifier: `pre${specifier}`, preid);
releaseChangelog(versionData);
releasePublish();

This allows for either a traditional release, or a pre-release that invokes the modified File 1 to provide Specifiers for each project which are then used to enact the correct prerelease version bump in each case.

Files

scripts/release-version.ts

import {
  ProjectGraphProjectNode,
  Tree,
  output,
  readJson,
  workspaceRoot,
} from '@nx/devkit';
import {
  resolveLocalPackageDependencies,
} from '@nx/js/src/generators/release-version/utils/resolve-local-package-dependencies';
import { parseRegistryOptions } from '@nx/js/src/utils/npm-config';
import chalk from 'chalk';
import { exec } from 'node:child_process';
import { join } from 'node:path';
import {
  NxReleaseConfig,
} from 'nx/src/command-line/release/config/config';
import {
  ReleaseGroupWithName,
} from 'nx/src/command-line/release/config/filter-release-groups';
import {
  getFirstGitCommit,
  getLatestGitTagForPattern,
} from 'nx/src/command-line/release/utils/git';
import {
  resolveSemverSpecifierFromConventionalCommits,
} from 'nx/src/command-line/release/utils/resolve-semver-specifier';
import { ProjectGraph } from 'nx/src/config/project-graph';
import { interpolate } from 'nx/src/tasks-runner/utils';
import ora from 'ora';
import { prerelease } from 'semver';

export const validReleaseVersionPrefixes = ['auto', '', '~', '^', '='] as const;

export interface ReleaseSpecifierGeneratorSchema {
  projects: ProjectGraphProjectNode[];
  releaseGroup: ReleaseGroupWithName;
  projectGraph: ProjectGraph;
  packageRoot?: string;
  currentVersionResolver?: 'registry' | 'disk' | 'git-tag';
  currentVersionResolverMetadata?: Record<string, unknown>;
  fallbackCurrentVersionResolver?: 'disk';
  firstRelease?: boolean;
  versionPrefix?: typeof validReleaseVersionPrefixes[number];
  conventionalCommitsConfig?: NxReleaseConfig['conventionalCommits'];
}

export type SpecifierData = Record<string, {
  /**
   * newVersion will be null in the case that no changes are detected for the project,
   * e.g. when using conventional commits
   */
  specifier: string | null | undefined;
  currentVersion: string;
  dependentProjects: any[];
}>;

export async function releaseSpecifierGenerator(
  tree: Tree,
  options: ReleaseSpecifierGeneratorSchema
): Promise<SpecifierData> {
  try {
    const specifierData: SpecifierData = {};

    if (!options.conventionalCommitsConfig) {
      throw new Error(
        `The "conventionalCommitsConfig" option is required for the "release-specifier" generator.`
      );
    }

    if (options.firstRelease) {
      // always use disk as a fallback for the first release
      options.fallbackCurrentVersionResolver = 'disk';
    }

    const projects = options.projects;

    const resolvePackageRoot = createResolvePackageRoot(options.packageRoot);

    // Resolve any custom package roots for each project upfront as they will need to be reused during dependency resolution
    const projectNameToPackageRootMap = new Map<string, string>();
    for (const project of projects) {
      projectNameToPackageRootMap.set(
        project.name,
        resolvePackageRoot(project)
      );
    }

    let currentVersion: string | undefined = undefined;
    let currentVersionResolvedFromFallback = false;

    // only used for options.currentVersionResolver === 'git-tag', but
    // must be declared here in order to reuse it for additional projects
    let latestMatchingGitTag:
      | { tag: string; extractedVersion: string }
      | null
      | undefined = undefined;

    // if specifier is undefined, then we haven't resolved it yet
    // if specifier is null, then it has been resolved and no changes are necessary
    let specifier: string | null | undefined = undefined;

    for (const project of projects) {
      const projectName = project.name;
      const packageRoot = projectNameToPackageRootMap.get(projectName);
      if (!packageRoot) {
        throw new Error(
          `The project "${projectName}" does not have a packageRoot available. Please report this issue on https://github.com/nrwl/nx`
        );
      }

      const packageJsonPath = join(packageRoot, 'package.json');

      const color = getColor(projectName);

      const log = (msg: string) => {
        console.log(color.instance.bold(projectName) + ' ' + msg);
      };

      if (!tree.exists(packageJsonPath)) {
        throw new Error(
          `The project "${projectName}" does not have a package.json available at ${packageJsonPath}.

To fix this you will either need to add a package.json file at that location, or configure "release" within your nx.json to exclude "${projectName}" from the current release group, or amend the packageRoot configuration to point to where the package.json should be.`
        );
      }

      output.logSingleLine(
        `Running release version for project: ${color.instance.bold(
          project.name
        )}`
      );

      const packageJson = readJson(tree, packageJsonPath);
      log(
        `🔍 Reading data for package "${packageJson.name}" from ${packageJsonPath}`
      );

      const { name: packageName, version: currentVersionFromDisk } =
        packageJson;

      switch (options.currentVersionResolver) {
        case 'registry': {
          const metadata = options.currentVersionResolverMetadata;
          const registryArg =
            typeof metadata?.registry === 'string'
              ? metadata.registry
              : undefined;
          const tagArg =
            typeof metadata?.tag === 'string' ? metadata.tag : undefined;

          const warnFn = (message: string) => {
            console.log(chalk.keyword('orange')(message));
          };
          const { registry, tag, registryConfigKey } =
            await parseRegistryOptions(
              workspaceRoot,
              {
                packageRoot: join(workspaceRoot, packageRoot),
                packageJson,
              },
              {
                registry: registryArg,
                tag: tagArg,
              },
              warnFn
            );

          /**
           * If the currentVersionResolver is set to registry, and the projects are not independent, we only want to make the request once for the whole batch of projects.
           * For independent projects, we need to make a request for each project individually as they will most likely have different versions.
           */
          if (
            !currentVersion ||
            options.releaseGroup.projectsRelationship === 'independent'
          ) {
            const spinner = ora(
              `${Array.from(new Array(projectName.length + 3)).join(
                ' '
              )}Resolving the current version for tag "${tag}" on ${registry}`
            );
            spinner.color =
              color.spinnerColor as typeof colors[number]['spinnerColor'];
            spinner.start();

            try {
              // Must be non-blocking async to allow spinner to render
              currentVersion = await new Promise<string>((resolve, reject) => {
                exec(
                  `npm view ${packageName} version --"${registryConfigKey}=${registry}" --tag=${tag}`,
                  (error, stdout, stderr) => {
                    if (error) {
                      return reject(error);
                    }
                    if (stderr) {
                      return reject(stderr);
                    }
                    return resolve(stdout.trim());
                  }
                );
              });

              spinner.stop();

              log(
                `📄 Resolved the current version as ${currentVersion} for tag "${tag}" from registry ${registry}`
              );
            } catch (e) {
              spinner.stop();

              if (options.fallbackCurrentVersionResolver === 'disk') {
                log(
                  `📄 Unable to resolve the current version from the registry ${registry}. Falling back to the version on disk of ${currentVersionFromDisk}`
                );
                currentVersion = currentVersionFromDisk;
                currentVersionResolvedFromFallback = true;
              } else {
                throw new Error(
                  `Unable to resolve the current version from the registry ${registry}. Please ensure that the package exists in the registry in order to use the "registry" currentVersionResolver. Alternatively, you can use the --first-release option or set "release.version.generatorOptions.fallbackCurrentVersionResolver" to "disk" in order to fallback to the version on disk when the registry lookup fails.`
                );
              }
            }
          } else {
            if (currentVersionResolvedFromFallback) {
              log(
                `📄 Using the current version ${currentVersion} already resolved from disk fallback.`
              );
            } else {
              log(
                `📄 Using the current version ${currentVersion} already resolved from the registry ${registry}`
              );
            }
          }
          break;
        }
        case 'disk':
          currentVersion = currentVersionFromDisk;
          log(
            `📄 Resolved the current version as ${currentVersion} from ${packageJsonPath}`
          );
          break;
        case 'git-tag': {
          if (
            !currentVersion ||
            // We always need to independently resolve the current version from git tag per project if the projects are independent
            options.releaseGroup.projectsRelationship === 'independent'
          ) {
            const releaseTagPattern = options.releaseGroup.releaseTagPattern;
            latestMatchingGitTag = await getLatestGitTagForPattern(
              releaseTagPattern,
              {
                projectName: project.name,
              }
            );
            if (!latestMatchingGitTag) {
              if (options.fallbackCurrentVersionResolver === 'disk') {
                log(
                  `📄 Unable to resolve the current version from git tag using pattern "${releaseTagPattern}". Falling back to the version on disk of ${currentVersionFromDisk}`
                );
                currentVersion = currentVersionFromDisk;
                currentVersionResolvedFromFallback = true;
              } else {
                throw new Error(
                  `No git tags matching pattern "${releaseTagPattern}" for project "${project.name}" were found. You will need to create an initial matching tag to use as a base for determining the next version. Alternatively, you can use the --first-release option or set "release.version.generatorOptions.fallbackCurrentVersionResolver" to "disk" in order to fallback to the version on disk when no matching git tags are found.`
                );
              }
            } else {
              currentVersion = latestMatchingGitTag.extractedVersion;
              log(
                `📄 Resolved the current version as ${currentVersion} from git tag "${latestMatchingGitTag.tag}".`
              );
            }
          } else {
            if (currentVersionResolvedFromFallback) {
              log(
                `📄 Using the current version ${currentVersion} already resolved from disk fallback.`
              );
            } else {
              log(
                // In this code path we know that latestMatchingGitTag is defined, because we are not relying on the fallbackCurrentVersionResolver, so we can safely use the non-null assertion operator
                `📄 Using the current version ${currentVersion} already resolved from git tag "${
                  latestMatchingGitTag!.tag
                }".`
              );
            }
          }
          break;
        }
        default:
          throw new Error(
            `Invalid value for options.currentVersionResolver: ${options.currentVersionResolver}`
          );
      }

      /**
       * If we are versioning independently then we always need to determine the specifier for each project individually, except
       * for the case where the user has provided an explicit specifier on the command.
       *
       * Otherwise, if versioning the projects together we only need to perform this logic if the specifier is still unset from
       * previous iterations of the loop.
       *
       * NOTE: In the case that we have previously determined via conventional commits that no changes are necessary, the specifier
       * will be explicitly set to `null`, so that is why we only check for `undefined` explicitly here.
       */
      if (specifier === undefined || options.releaseGroup.projectsRelationship === 'independent') {

        const specifierSource = 'conventional-commits';
        switch (specifierSource) {
          case 'conventional-commits': {
            if (options.currentVersionResolver !== 'git-tag') {
              throw new Error(
                `Invalid currentVersionResolver "${options.currentVersionResolver}" provided for release group "${options.releaseGroup.name}". Must be "git-tag" when "specifierSource" is "conventional-commits"`
              );
            }

            const affectedProjects =
              options.releaseGroup.projectsRelationship === 'independent'
                ? [projectName]
                : projects.map((p) => p.name);

            // latestMatchingGitTag will be undefined if the current version was resolved from the disk fallback.
            // In this case, we want to use the first commit as the ref to be consistent with the changelog command.
            const previousVersionRef = latestMatchingGitTag
              ? latestMatchingGitTag.tag
              : options.fallbackCurrentVersionResolver === 'disk'
              ? await getFirstGitCommit()
              : undefined;

            if (!previousVersionRef) {
              // This should never happen since the checks above should catch if the current version couldn't be resolved
              throw new Error(
                `Unable to determine previous version ref for the projects ${affectedProjects.join(
                  ', '
                )}. This is likely a bug in Nx.`
              );
            }

            specifier = await resolveSemverSpecifierFromConventionalCommits(
              previousVersionRef,
              options.projectGraph,
              affectedProjects,
              options.conventionalCommitsConfig
            );

            if (!specifier) {
              log(
                `🚫 No changes were detected using git history and the conventional commits standard.`
              );
              break;
            }

            // TODO: reevaluate this logic/workflow for independent projects
            //
            // Always assume that if the current version is a prerelease, then the next version should be a prerelease.
            // Users must manually graduate from a prerelease to a release by providing an explicit specifier.
            if (prerelease(currentVersion ?? '')) {
              specifier = 'prerelease';
              log(
                `📄 Resolved the specifier as "${specifier}" since the current version is a prerelease.`
              );
            } else {
              log(
                `📄 Resolved the specifier as "${specifier}" using git history and the conventional commits standard.`
              );
            }
            break;
          }
        }
      }

      // Resolve any local package dependencies for this project (before applying the new version or updating the versionData)
      const localPackageDependencies = resolveLocalPackageDependencies(
        tree,
        options.projectGraph,
        projects,
        projectNameToPackageRootMap,
        resolvePackageRoot,
        // includeAll when the release group is independent, as we may be filtering to a specific subset of projects, but we still want to update their dependents
        options.releaseGroup.projectsRelationship === 'independent'
      );

      const dependentProjects = Object.values(localPackageDependencies)
        .flat()
        .filter((localPackageDependency) => {
          return localPackageDependency.target === project.name;
        });

      if (!currentVersion) {
        throw new Error(
          `The current version for project "${project.name}" could not be resolved. Please report this on https://github.com/nrwl/nx`
        );
      }

      specifierData[projectName] = {
        specifier,
        currentVersion,
        dependentProjects,
      };

      if (!specifier) {
        log(
          `🚫 Skipping specifier generation for "${packageJson.name}" as no changes were detected.`
        );
        continue;
      }
    }

    return specifierData;
  } catch (e: any) {
    if (process.env.NX_VERBOSE_LOGGING === 'true') {
      output.error({
        title: e.message,
      });
      // Dump the full stack trace in verbose mode
      console.error(e);
    } else {
      output.error({
        title: e.message,
      });
    }
    process.exit(1);
  }
}

export default releaseSpecifierGenerator;

function createResolvePackageRoot(customPackageRoot?: string) {
  return (projectNode: ProjectGraphProjectNode): string => {
    // Default to the project root if no custom packageRoot
    if (!customPackageRoot) {
      return projectNode.data.root;
    }
    return interpolate(customPackageRoot, {
      workspaceRoot: '',
      projectRoot: projectNode.data.root,
      projectName: projectNode.name,
    });
  };
}

const colors = [
  { instance: chalk.green, spinnerColor: 'green' },
  { instance: chalk.greenBright, spinnerColor: 'green' },
  { instance: chalk.red, spinnerColor: 'red' },
  { instance: chalk.redBright, spinnerColor: 'red' },
  { instance: chalk.cyan, spinnerColor: 'cyan' },
  { instance: chalk.cyanBright, spinnerColor: 'cyan' },
  { instance: chalk.yellow, spinnerColor: 'yellow' },
  { instance: chalk.yellowBright, spinnerColor: 'yellow' },
  { instance: chalk.magenta, spinnerColor: 'magenta' },
  { instance: chalk.magentaBright, spinnerColor: 'magenta' },
] as const;

function getColor(projectName: string) {
  let code = 0;
  for (let i = 0; i < projectName.length; ++i) {
    code += projectName.charCodeAt(i);
  }
  const colorIndex = code % colors.length;

  return colors[colorIndex];
}

scripts/nx-release.ts

import { releaseChangelog, releasePublish, releaseVersion } from 'nx/release';
import {
  createNxReleaseConfig,
  handleNxReleaseConfigError,
} from 'nx/src/command-line/release/config/config';
import {
  ReleaseGroupWithName,
  filterReleaseGroups,
} from 'nx/src/command-line/release/config/filter-release-groups';
import { batchProjectsByGeneratorConfig } from 'nx/src/command-line/release/utils/batch-projects-by-generator-config';
import { VersionData } from 'nx/src/command-line/release/version';
import { readNxJson } from 'nx/src/config/nx-json';
import { FsTree } from 'nx/src/generators/tree';
import { createProjectFileMapUsingProjectGraph } from 'nx/src/project-graph/file-map-utils';
import {
  createProjectGraphAsync,
} from 'nx/src/project-graph/project-graph';
import { workspaceRoot } from 'nx/src/utils/workspace-root';
import * as yargs from 'yargs';
import { releaseSpecifierGenerator, type ReleaseSpecifierGeneratorSchema, type SpecifierData } from './release-version';

type Args = {
  [x: string]: unknown;
  specifier: string | undefined;
  preid: string | undefined;
  dryRun: boolean;
  verbose: boolean;
  _: (string | number)[];
  $0: string;
}

(async () => {
  const args: Args = await yargs
  .version(false) // don't use the default meaning of version in yargs
  .option('specifier', {
    description:
      'Explicit version specifier to use, if overriding conventional commits',
    type: 'string',
  })
  .option('preid', {
    description: 'Preid to use for prerelease versions',
    type: 'string',
  })
  .option('dryRun', {
    alias: 'd',
    description:
      'Whether or not to perform a dry-run of the release process, defaults to true',
    type: 'boolean',
    default: true,
  })
  .option('verbose', {
    description:
      'Whether or not to enable verbose logging, defaults to false',
    type: 'boolean',
    default: true,
  })
  .parseAsync();

  const { specifier, preid, dryRun, verbose } = args;

  const versionData: VersionData = {};

  if (!preid) {
    const { projectsVersionData } = await releaseVersion({ specifier, dryRun, verbose });
    for (const [project, data] of Object.entries(projectsVersionData)) {
      versionData[project] = data;
    }
  } else {
    if (specifier) {
      throw new Error('Cannot specify both a specifier and a preid');
    }

    const specifierData: SpecifierData = await getSpecifierData(args);

    for await (const [project, data] of Object.entries(specifierData)) {
      // check if data.specifier matches patch, minor, or major
      if (['patch', 'minor', 'major'].includes(data.specifier as string)) {
        const preSpecifier = `pre${data.specifier}`;
        const { projectsVersionData } = await releaseVersion({ specifier: preSpecifier, preid, projects: [project], dryRun, verbose });
        versionData[project] = projectsVersionData[project];
      } else {
        versionData[project] = {
          currentVersion: data.currentVersion,
          dependentProjects: data.dependentProjects,
          newVersion: null,
        };
      }
    }
  }

  const changelog = await releaseChangelog({ versionData, dryRun, verbose });
  const publishStatus = await releasePublish({ dryRun, verbose });
  const output = {
    versionData,
    changelog,
    publishStatus,
  };

  console.log(JSON.stringify(output, null, 2));
  process.exit(publishStatus);
})();

async function getSpecifierData(args: Args): Promise<SpecifierData> {
  const specifierData: SpecifierData = {};

  const projectGraph = await createProjectGraphAsync({ exitOnError: true });
  const nxJson = readNxJson();

  // Apply default configuration to any optional user configuration
  const { error: configError, nxReleaseConfig } = await createNxReleaseConfig(
    projectGraph,
    await createProjectFileMapUsingProjectGraph(projectGraph),
    nxJson.release
  );
  if (configError) {
    return await handleNxReleaseConfigError(configError);
  }

  if (!nxReleaseConfig) {
    throw new Error('No release configuration found');
  }

  const {
    error: filterError,
    releaseGroups,
    releaseGroupToFilteredProjects,
  } : {
    error: null | { title: string; bodyLines?: string[] };
    releaseGroups: ReleaseGroupWithName[];
    releaseGroupToFilteredProjects: Map<ReleaseGroupWithName, Set<string>>;
  } = filterReleaseGroups(
    projectGraph,
    nxReleaseConfig,
    args.projects as string[],
    args.groups as string[],
  );
  if (filterError) {
    console.error(filterError);
    process.exit(1);
  }

  const tree = new FsTree(workspaceRoot, args.verbose);

  for (const releaseGroup of releaseGroups) {
    const releaseGroupName = releaseGroup.name;
    const releaseGroupProjectNames = Array.from(
      releaseGroupToFilteredProjects.get(releaseGroup) || []
    );
    const projectBatches = batchProjectsByGeneratorConfig(
      projectGraph,
      releaseGroup,
      // Only batch based on the filtered projects within the release group
      releaseGroupProjectNames
    );

    for (const [
      generatorConfigString,
      projectNames,
    ] of projectBatches.entries()) {
      const [generatorName, generatorOptions] = JSON.parse(
        generatorConfigString
      );

      const options: ReleaseSpecifierGeneratorSchema = {
        projects: projectNames.map((name) => projectGraph.nodes[name]),
        releaseGroup,
        projectGraph,
        packageRoot: undefined,
        currentVersionResolver: generatorOptions.currentVersionResolver,
        conventionalCommitsConfig: nxReleaseConfig.conventionalCommits,
      };

      const leData = await releaseSpecifierGenerator(tree, options);

      for (const [key, value] of Object.entries(leData)) {
        if(specifierData[key]) {
          throw new Error(`Specifier data key "${key}" already exists in specifier data. This is likely a bug.`);
        }
        specifierData[key] = value;
      }
    }
  }

  return specifierData;
}

Zordrak commented 7 months ago

Still a small amount of work to do I think because a release that isn't a prerelease does not override the currentRelease if no changes have been made. So I will need to add something to forcefully allow the promotion of a prerelease to a release.. and possibly handle cleaning up pre-releases from the changelog?

Zordrak commented 7 months ago

I'm beginning to think this is a wider problem that nx just doesnt account for the nature of prerelease workflows where a prerelease has to be published so that it can be integration-tested.

Even having tried to fix everything I can, I'm still left with the problem that:

conventional-commits must use the git-tag specifier source, so you cannot do a "treat published packages with dist-tag: latest as the original version"
If you publish a pre-release and tag it, it's treated like a production release and there's no way to do a "release" that auto bumps anything thats currently a prerelease. So when your feature branch is merged to master, master just thinks your prerelease version is the current latest release version and doesnt need bumping.
If you publish a pre-release (x.x.x-0) and don't tag it, so that when you merge to master you can perform a new release from the previous production release, you cant do subsequent pre-releases because
- you cant overwrite a published prerelease version, so once -0 is used it's used.
- it cant determine the pre-existing prerelease version (-0) so as to increment it (to -1)
- the semver package, and therefore nx, doesnt allow for you to add buildinfo to the prerelease version so as to at least make it unique and independently publishable (i.e. -0+commithash)

@fahslaj Do you have any pre-existing use cases that publish prerelease versions to remote registries, while having a joined up development process? As far as I can tell, nx just isn't designed to support it, and what I'm actually doing is self-defeating because I'm trying to get nx to do something it's not designed to do.

All I'm trying to achieve, and I would imagine would be the original goal of @lorenzodejong is:

master is released. dist-tag: latest == v1.0.0
developer branches master - branch/dist-tag == "foo". Creates PR. Workflow tests, builds and publishes prerelease (based on conventional-commits) e.g. v1.1.0-foo.0 or v1.1.0-foo.0+commithash
developer updates PR. Workflow tests, builds and published new preprelease, e.g. v1.1.0-foo.1 or v.1.1.0-foo.0+commithash
Prereleases can be consumed by downstream packages in integration testing externally to the process
Successful merge (whether squash or merge) of branch to master kicks off release workflow that, based on conventional-commits, updates the package versions, writes new changelogs and publishes the packages, e.g. v1.1.0

fahslaj commented 7 months ago

Hi @Zordrak I appreciate your passion and hard work digging into this potential feature. You’re right that Nx Release currently does not support a fully automated prerelease workflow. Currently, it’s up to the user to specify manually when a prerelease should be cut and when a prerelease version should be graduated to a production release version. Handling of prereleases is something that we’re looking to improve holistically, and the observations and perspective of you and @lorenzodejong are greatly appreciated. Unfortunately, I do not have an easy answer for you to get the behavior your desire right now without rewriting the version generator.

As a side note, we’re aware that the sheer amount of functionality within the version generator makes it difficult replace. We’ve got plans to extract common functionality into utility functions exported from Nx itself to aid users in creating custom version generators - we’re just not quite there yet.

Zordrak commented 7 months ago

All good news, thank you very much for the response. For now I have decided that it's simply not a maintainable solution, and so am going to require developers to manually push their own pre-release builds, and focus the automation on the releases.

I look forward to the future improvements.

pachuka commented 6 months ago

Awesome discussion! Ultimately I think the behavior described here https://github.com/nrwl/nx/issues/22150#issuecomment-2047603955 is what lerna already does for prerelease versioning which if I recall correctly determines it via this describe-ref utility -> https://github.com/lerna/lerna/blob/c78d8ffdca482f94b8a0f1267566610db4484bb8/libs/core/src/lib/describe-ref.ts.

The output of that is used in the publish command there https://github.com/lerna/lerna/blob/c78d8ffdca482f94b8a0f1267566610db4484bb8/libs/commands/publish/src/index.ts#L586 where it generates a version based on: ${nextVersion}-${preid}.${Math.max(0, refCount - 1)}+${sha}

I think should be able to do something pretty much identical to that for prerelease (or have it be opt-in behavior).

Based on my testing of nx release the prerelease tagging is the main thing preventing me from being able to drop lerna completely and switching to pure nx tooling. I typically use it in a scenario where having non-mainline branch publishes are useful but I don't want those tagged/committed back/marked as github releases, but I do want the npm package to get published for consumption for the various PR testing and E2E testing scenarios.

Not sure if that helps any, but I've used that describe-ref logic + the logic in the publish script outside of the lerna ecosystem when I wanted a separate release process to do something very similar for prereleases.

Thanks!

nrwl / nx