search

microsoft / rushstack

Monorepo for tools developed by the Rush Stack community
https://rushstack.io/
Other
5.94k stars 598 forks source link

[rush] Document how to publish to private self-hosted GitLab and Sonatype Nexus registry with uncommon port numbers #1522

Open renoirb opened 5 years ago

renoirb commented 5 years ago

Documentation is lacking. (Bug?)

In a corporate setting, we have our own self-hosted Sonatype Nexus registry and our own self-hosted GitLab. Here are the settings I can use (anonymized, of course)

With Yarn/npm i could publish with an ~/.npmrc that looks like this;

; _auth being the echo -n 'john.doe@foo.bar.example.org:supersekret' | openssl base64
registry=https://nexus.foo.bar.example.org:50043/repository/npm-hosted
_auth=foooooo

My project's common/config/rush/.npmrc Would contain only the registry lines, nothing about authentication.

What should I put in rush.json for my situation? git+https://...? Docs are too vague in the version bump, pack, publish flow.

The docs is unclear about the appropriate steps, assuming on master branch (of course). Tests, my fallback place to learn about things, has only irrelevant strings. I'm assuming it's all normal npm,pnpm,yarn process automated by rush.

I'm aware as maintainer, when I release, I should run commands manually.

Scenario:

(Recommendations for doing this CI side, assuming CI can push to Nexus. I can help document if I can make it work).

On related note. Here is my GitLab CI YAML. My project is a fork from this public experiment. https://github.com/renoirb/experiments-201908-rush-typescript-just-bili-monorepo/blob/master/.gitlab-ci.yml

octogonz commented 5 years ago

(Related conversation on Gitter.)

Did you try using the --npm-auth-token parameter for rush publish, like in common/config/azure-pipelines/templates/buildAndPublish.yaml?

Generally we wouldn't want to store a sensitive publishing token in a file on disk such as ~/.npmrc. Instead it gets passed via an environment variable.

octogonz commented 5 years ago

If there's specific issues that aren't covered well in the Publishing packages docs, let me know and we can try to improve them. From the issue description, it wasn't completely clear to me which part you are having trouble with.

octogonz commented 5 years ago

@deanbot appeared to have a related issue. In his case, the problem was that rush.json's repository.url did not exactly match the git remote get-url origin strings.

To address that part of this problem, I think we should do a quick PR that:

  1. Relaxes the getRemoteMasterBranch() normalization (normalizedRemoteUrl) to be a looser match, and
  2. Improves the rush.json comments for repository.url to better explain how you're supposed to specify this URL
octogonz commented 5 years ago

@deanbot 's repro involved URLs of the form:

https://my-tenant.visualstudio.com/my-project/_git/my-repo

vs

my-tenant@vs-ssh.visualstudio.com:v3/my-tenant/my-project/my-repo

According to the Git docs, [user@]server:project.git is a shorthand for ssh://[user@]server/project.git

octogonz commented 5 years ago

Maybe we can use the normalize-git-url package to sort this out.

renoirb commented 5 years ago

Yes, I tried with the --npm-auth-token. I'll document better everything I've tried when I get back to ny workstation. But errors came up because of trying to get history using git. Because it's not over ssh, with a different port.

renoirb commented 5 years ago

Update:

  1. rush version --version-policy libraries --bump works, now that common/config/rush/version-policies.json has versions matching each packages. Notice thought that I have to commit and tag manually. See
  2. rush publish ... is still not working. I mean, it worked once. Can't tell when. And cannot re-publish anything anymore.

Nexus registry is ONLY through TLS (https://nexus.foo.example.org:50043/repository/npm-hosted/). We can't work with opening non TLS port.

Here are a few error logs I've got

export NEXUS_AUTH_TOKEN=secrettokenstring
rush publish \
        --publish \
        --apply\
        --publish \
        --version-policy libraries \
        --include-all \
        --set-access-level public \
        --npm-auth-token ${NEXUS_AUTH_TOKEN} \
        -r 'https://nexus.foo.example.org:50043/repository/npm-hosted/'
0 info it worked if it ends with ok
1 verbose cli [ '/usr/bin/node',
1 verbose cli   '/usr/bin/npm',
1 verbose cli   'publish',
1 verbose cli   '--registry https://nexus.foo.example.org:50043/repository/npm-hosted/:_authToken=secrettokenstring',
1 verbose cli   '--access',
1 verbose cli   'public' ]
2 info using npm@6.9.0
3 info using node@v10.16.3
4 verbose npm-session bd3f917a6ad23924
5 verbose publish [ '.' ]
6 info lifecycle @frontend-bindings/data-model@1.0.1~prepublish: @frontend-bindings/data-model@1.0.1
7 info lifecycle @frontend-bindings/data-model@1.0.1~prepare: @frontend-bindings/data-model@1.0.1
8 info lifecycle @frontend-bindings/data-model@1.0.1~prepublishOnly: @frontend-bindings/data-model@1.0.1
9 verbose lifecycle @frontend-bindings/data-model@1.0.1~prepublishOnly: unsafe-perm in lifecycle true
10 verbose lifecycle @frontend-bindings/data-model@1.0.1~prepublishOnly: PATH: /usr/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/data/projects/frontend-bindings/frontend-bindings/packages/data-model/node_modules/.bin:/data/projects/frontend-bindings/frontend-bindings/packages/data-model/node_modules/.bin:/usr/bin:/data/projects/frontend-bindings/frontend-bindings/common/temp/node_modules/.bin:/data/projects/frontend-bindings/frontend-bindings/node_modules/.bin:/data/.npm/bin:/home/vagrant/bin:/home/vagrant/node_modules/.bin:/data/.npm/bin:/home/vagrant/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
11 verbose lifecycle @frontend-bindings/data-model@1.0.1~prepublishOnly: CWD: .
12 silly lifecycle @frontend-bindings/data-model@1.0.1~prepublishOnly: Args: [ '-c', 'rushx build' ]
13 silly lifecycle @frontend-bindings/data-model@1.0.1~prepublishOnly: Returned: code: 0  signal: null
14 info lifecycle @frontend-bindings/data-model@1.0.1~prepack: @frontend-bindings/data-model@1.0.1
15 info lifecycle @frontend-bindings/data-model@1.0.1~postpack: @frontend-bindings/data-model@1.0.1
16 notice
17 notice 📦  @frontend-bindings/data-model@1.0.1
18 notice === Tarball Contents ===
19 notice 2.4kB  package.json
19 notice 1.3kB  CHANGELOG.json
19 notice 268B   CHANGELOG.md
19 notice 36.3kB dist/browser.cjs.js
...
20 notice === Tarball Details ===
21 notice name:          @frontend-bindings/data-model
21 notice version:       1.0.1
21 notice package size:  18.5 kB
21 notice unpacked size: 100.7 kB
21 notice shasum:      ...
21 notice integrity:     ...
21 notice total files:   24
22 notice
23 http fetch PUT 401 https://nexus.foo.example.org:50043/repository/npm-hosted/@frontend-bindings%2fdata-model 355ms
24 verbose stack Error: Unable to authenticate, need: BASIC realm="Sonatype Nexus Repository Manager"
24 verbose stack     at res.buffer.catch.then.body (/usr/lib/node_modules/npm/node_modules/npm-registry-fetch/check-response.js:94:17)
24 verbose stack     at process._tickCallback (internal/process/next_tick.js:68:7)
25 verbose statusCode 401
26 verbose pkgid @frontend-bindings/data-model@1.0.1
27 verbose cwd /data/projects/frontend-bindings/frontend-bindings/packages/data-model
28 verbose Linux 4.15.0-62-generic
29 verbose argv "/usr/bin/node" "/usr/bin/npm" "publish" "--registry https://nexus.foo.example.org:50043/repository/npm-hosted/:_authToken=secrettokenstring" "--access" "public"
30 verbose node v10.16.3
31 verbose npm  v6.9.0
32 error code E401
33 error Unable to authenticate, need: BASIC realm="Sonatype Nexus Repository Manager"
34 verbose exit [ 1, true ]

Rush seems to be doing npm view "@frontend-bindings/data-model" time --json --//nexus.foo.example.org:50043/repository/npm-hosted/:authToken=secrettokenstring

Which seem to break, but when I do manually, it works;

npm view "@frontend-bindings/data-model@1.0.0" time --json --registry https://nexus.foo.example.org:50043/repository/npm-hosted/
{
  "created": "2019-02-09T17:37:51.903Z",
  "modified": "2019-06-04T02:58:49.995Z",
  "1.0.0-alpha.2": "2019-02-09T17:37:51.903Z",
  "1.0.0-alpha.3": "2019-02-09T17:58:41.785Z",
  "1.0.0": "2019-06-04T02:58:49.995Z"
}

I'm assuming it's related to figuring out which registry to use when doing publish

rush publish \
    --npm-auth-token ${NEXUS_AUTH_TOKEN} \
    -r 'https://nexus.foo.example.org:50043/repository/npm-hosted/'

My guess, is that any time i happen to do npm login --regsitry=... manually (for other packages), I might just break that publish command with Rush. See PublishAction#_addSharedNpmConfig(). I've convigured PNPM in rush.json, and that's npm doing publishing. It's confusing.

About common/config/rush/.npmrc, I'm unsure WHAT to put, exactly.

Use-case:

  1. Public packages are hosted on /repository/js-group/, so anything is backed up
  2. Own packages published are in /repository/npm-hosted/
  3. Assuming WE CANNOT put anything _authToken in common/config/rush/.npmrc
; file: common/config/rush/.npmrc
registry=//nexus.foo.example.org:50043/repository/js-group/
@frontend-bindings:registry=//nexus.foo.example.org:50043/repository/npm-hosted/
registry=//nexus.foo.example.org:50043/repository/npm-hosted/

Screenshots

20190916-rush-publish

renoirb commented 5 years ago

I might be stuck using, as per Nexus docs

registry = https://nexus.foo.example.org:50043/repository/npm-hosted/
email=renoirb@example.org
always-auth=true
_auth=$NPM_TOKEN
// hopefully this can be written and superseeded by env var
renoirb commented 5 years ago

Well, it seems that if somewhere in the path for .npmrc (e.g. ~/.npmrc) with credentials for Nexus like comment above

NPM config from where to release

; File ~/.npmrc
; Assuming you publish to Nexus anything under namespace @renoirb; Nexus requires, when using such configuration TO ALWAYS have always-auth=true set, with email
@renoirb:registry=https://nexus.foo.example.org:50043/repository/npm-hosted/
email=renoirb@example.org
always-auth=true
_auth=$NPM_TOKEN
; This above hasn't been tried out yet.

And rush

; File common/config/rush/.npmrc
registry=https://nexus.foo.example.org:50043/repository/js-group/
@renoirb:registry=https://nexus.foo.example.org:50043/repository/npm-hosted/

Version policy

cat common/config/rush/version-policies.json

[
  {
    "policyName": "tooling",
    "definitionName": "lockStepVersion",
    "version": "1.0.1",
    "nextBump": "patch"
  }
]

In rush.json

{
  "project": [
    {
       "packageName": "@renoirb/conventions-use-bili",
       "projectFolder": "conventions/use-bili",
       "versionPolicyName": "tooling",
       "reviewCategory": "conventions"
    }
  ]
}

Assuming you've bumped version

rush version --version-policy tooling --bump

Commit affected packages, tag in some way, etc.

To publish packages (i.e. build all matching tools version policy):

rush publish --publish --version-policy tooling --include-all --set-access-level public -r https://nexus.foo.example.org:50043/repository/npm-hosted/
renoirb commented 5 years ago

Updated example and full use-case is now published npmjs.com/search?q=@renoirb and up-to-date.

All works well, let's see what we can improve in the documentation.

What would be nice, but I couldn't figure out yet:

  1. How to create git tag for each package
  2. Now that I can run manually, as maintainer, from his workstation, how does people make this work from CI? What am I missing
octogonz commented 5 years ago

Thanks @renoirb !

1 . How to create git tag for each package

I believe Rush does this automatically during publishing.

2 . Now that I can run manually, as maintainer, from his workstation, how does people make this work from CI? What am I missing

This is what we do in common/config/azure-pipelines/templates/buildAndPublish.yaml (the CI job that gets run by Azure DevOps):

- script: 'node common\scripts\install-run-rush.js version --bump --version-policy $(VersionPolicy) --target-branch $(Build.SourceBranchName)'
  displayName: 'Rush Version'
- script: 'node common\scripts\install-run-rush.js publish --apply --publish --include-all --target-branch $(Build.SourceBranchName) --npm-auth-token $(npmToken) --add-commit-details --set-access-level public'
  displayName: 'Rush Publish'
renoirb commented 5 years ago

I don't understand the --target-branch, if I'm already on master? Or is it expected we MUST ALWAYS bump version from another branch, AND that IT WILL merge into master THEN create git tag, and so on?

renoirb commented 5 years ago

Hi, here's my bash based build script I've just come up with.

#!/usr/bin/env bash

#
# MonoRepo helper utility to manage code releases
#
# Author: Renoir Boulanger <hello@renoirboulanger.com>
#
# Some other cool bash patterns in
# https://gist.github.com/renoirb/361e4e2817341db4be03b8f667338d47
#
# For the release process, refer to this ticket:
# https://github.com/microsoft/web-build-tools/issues/1522
#
# TIP:
# Before commiting changes in this file, make sure you run shellcheck and no messages are shown
#
# > shellcheck bin/monorepo
#
# For publishing, we have to do this from the maintainer's workstation, not GitLab.
#
# To push packages to Nexus, you need to give your Authentication token to Rush.
# The Authentication token is a base64 string of your Nexus registry account credentials.
# Its a semi-column separated values for username:password. Plug in your Nexus username and password.
# Paste the value in ~/.npmrc at `_auth`
#
# > echo -n 'renoir.boulanger:password' | openssl base64
#
# For a successful publish, make sure you have ~/.npmrc with the following lines:
#
# > @frontend-bindings:registry=https://nexus.foo.example.org:50043/repository/npm-hosted/
# > email=renoir.boulanger@example.org
# > always-auth=true
# > _auth=...
#
# Then:
# > bin/monorepo info
# > export VERSION_POLICY=libraries
# > bin/monorepo version
# > bin/monorepo publish
#
# To get a directory with archives:
# > bin/monorepo pack
#

set -e

################################################################################
# Functions. That could be distributed and re-used.

function must_command_exist()
{
    command -v "${1}" >/dev/null 2>&1 || { echo >&2 "Command ${1} must exist."; exit 1; }
}

# https://stackoverflow.com/questions/11362250/in-bash-how-do-i-test-if-a-variable-is-defined-in-u-mode#answer-19874099
function is_var_defined()
{
    if [ $# -ne 1 ]
    then
        echo "Expected exactly one argument: variable name as string, e.g., 'my_var'"
        exit 1
    fi
    # Tricky.  Since Bash option 'set -u' may be enabled, we cannot directly test if a variable
    # is defined with this construct: [ ! -z "$var" ].  Instead, we must use default value
    # substitution with this construct: [ ! -z "${var:-}" ].  Normally, a default value follows the
    # operator ':-', but here we leave it blank for empty (null) string.  Finally, we need to
    # substitute the text from $1 as 'var'.  This is not allowed directly in Bash with this
    # construct: [ ! -z "${$1:-}" ].  We need to use indirection with eval operator.
    # Example: $1="var"
    # Expansion for eval operator: "[ ! -z \${$1:-} ]" -> "[ ! -z \${var:-} ]"
    # Code  execute: [ ! -z ${var:-} ]
    eval "[ ! -z \${$1:-} ]"
    return $?  # Pedantic.
}

function must_var_defined()
{
    is_var_defined "$1" || { echo >&2 "Variable ${1} is not defined and must exist. Aborting."; exit 1; }
}

function error { printf "Error: %s \n " "$@" >&2; exit 1; }

function policy_names () {
  npx strip-json-comments-cli common/config/rush/version-policies.json | jq .[].policyName
}

function list_versions () {
  node common/scripts/install-run-rush.js list --version
}

function action_version () {
  must_var_defined VERSION_POLICY
  node common/scripts/install-run-rush.js version --bump --version-policy "${VERSION_POLICY}" --target-branch "${TARGET_BRANCH=master}"
}

function action_publish () {
  must_var_defined VERSION_POLICY
  must_var_defined REGISTRY
  node common/scripts/install-run-rush.js publish --publish --apply --include-all --target-branch "${TARGET_BRANCH=master}" --version-policy "${VERSION_POLICY}" --set-access-level public -r "${REGISTRY}"
  # ... yeah. Something, in some order. Applying works if I have --target-branch=master, whereas elsewhere it's a space.
  node common/scripts/install-run-rush.js publish --publish --pack --include-all --release-folder dist --apply-git-tags-on-pack --apply --target-branch=master
}

function action_pack () {
  node common/scripts/install-run-rush.js clean
  node common/scripts/install-run-rush.js rebuild
  node common/scripts/install-run-rush.js publish --publish --pack --include-all --release-folder dist
}

function action_info () {
  printf "\n\nAvailable packages:\n"
  list_versions
  printf "\n\nAvailable version policies:\n"
  policy_names
  printf "\n\n"
}

function display_usage() {
    echo -e "\n"
    echo "MonoRepo helper utility to manage code releases."
    echo -e "\nUsage:\n  ${0} <ACTION>\n\n"
    echo -e "<ACTION> being one of:"
    printf "  * %s\n"  "${switchValidActions[@]}"
    echo -e "\n"
    echo -e "Examples:"
    echo -e "  ${0}"
    echo -e "  ${0} pack"
    echo -e "\n"
}

################################################################################
# Sanity checks before executing anything else.

must_command_exist git
must_command_exist jq
must_command_exist node
must_command_exist npx

################################################################################
# VALIDATION LOGIC DATA
switchValidActions=(info version publish pack)
REGISTRY="${MONOREPO_REGISTRY=https://nexus.foo.example.org:50043/repository/npm-hosted/}"

################################################################################
# PRE EXECUTION

if [ $# -ne 1 ]; then
  display_usage
  exit
fi

if [[ ! "${switchValidActions[*]}" =~ ${1} ]]; then
    error "Invalid action \"${1}\". It must be one of: ${switchValidActions[*]}"
fi
ACTION="${1^^}"

################################################################################
# EXECUTION

if [[ $ACTION == "INFO" ]]; then
  action_info
fi
if [[ $ACTION == "VERSION" ]]; then
  action_version
fi
if [[ $ACTION == "PUBLISH" ]]; then
  action_publish
fi
if [[ $ACTION == "PACK" ]]; then
  action_pack
fi
renoirb commented 5 years ago

This worked for adding branch

node common/scripts/install-run-rush.js publish --publish --pack --include-all --release-folder dist --apply-git-tags-on-pack --apply --target-branch=master

But I can't make this to TAG only on changed packages. Unless I make bash NOT die on errors in publish function

renoirb commented 5 years ago

Question is now more: What is the order we should do things when publishing a package.

Assuming:

My understanding is the following (written on top of my head, not exact commands):

rush update
rush clean
rush build
rush test
rush change --target-branch v1.0.0-branch
# rinse repeat above steps, until we want to release
rush version --bump --version-policy foo
rush publish --publish --apply --version-policy foo --target-branch v1.0.0-branch -r https://nexus.foo.example.org:54330/registry/npm-hosted/
rush publish --pack --apply-git-tags-on-pack --include-all --version-policy foo

What confuse me is the use of publish --publish and publish --pack --publish ... to do two things. I'm never sure WHICH ones is about publishing to gitlab, which one DO create tags.

When I succeed creating tags, its only without version policy, and then errors because not all packages changed versions and git errors for existing tag.

What I didn't understand this far was the goal of rush change. I now see that it can be done more than once. But rather after many commits as a way to describe a change spanning multiple package. So it's to help making changelog and link dependents and their impact

alfonsoar commented 4 years ago

Hi @renoirb - Did you ever get the tagging process to work from the CI? i.e. not having the mainter manually do this from his/hers workstation?

What i am trying todo is have the whole release process automated. i.e. when I tag my master branch on gitlab the CI automatically bumps versions, generates changelogs and publishes to nexus.