[BUG] Workspace linking for constructs is not working

[rverma-dev]

Describe the bug

Consider a monorepo with one construct library

const root = new nx_monorepo.NxMonorepoProject({
  defaultReleaseBranch: 'main',
  devDeps: [`aws-prototyping-sdk@${AWS_PDK_VERSION}`, 'lerna@^6.6.1', '@nrwl/devkit@^15.9.2'],
  name: 'nsa',
  projenrcTs: true,
  description: 'Nsl SAAS Accelerator on AWS',
  packageName: '@nsa/aws',
  packageManager: NodePackageManager.PNPM,
  prettierOptions: {
    settings: {
      singleQuote: true,
      tabWidth: 2,
      printWidth: 120,
      trailingComma: TrailingComma.ALL,
      arrowParens: ArrowParens.AVOID,
    },
  },
  gitignore: ['.idea', '.vscode'],
  stale: true,
  autoApproveUpgrades: true,
  autoApproveOptions: {
    allowedUsernames: ['rverma-nsl'],
  },
  autoMerge: true,
  buildWorkflow: true,
  dependabot: true,
  github: true,
  minNodeVersion: '18.0.0',
  typescriptVersion: '~5.0.4',
  vscode: true,
  workspaceConfig: {
    linkLocalWorkspaceBins: true,
  },
});

new awscdk.AwsCdkConstructLibrary({
  parent: root,
  outdir: 'packages/constructs',
  bundledDeps: [
    '@nslhb/eks-blueprints@1.6.21',
    `@aws-sdk/client-iam@${AWS_SDK_VERSION}`,
    `@aws-sdk/client-secrets-manager@${AWS_SDK_VERSION}`,
    '@types/js-yaml@4.0.5',
    'js-yaml@4.1.0',
    'sync-request',
  ],
  peerDeps: ['cdk-nag'],
  devDeps: ['@types/aws-lambda', 'aws-lambda', 'jsii-pacmak@1.80.0' // need to ovveride this, default is very old inaccessible version],
  defaultReleaseBranch: 'main',
  name: '@nsa/construct',
  packageManager: NodePackageManager.PNPM,
  cdkVersion: CDK_VERSION,
  constructsVersion: CONSTRUCT_VERSION,
  minNodeVersion: root.minNodeVersion,
  jestOptions: {
    jestVersion: JEST_VERION, // need to override this as using ts 5.0.4
    jestConfig: {
      detectOpenHandles: true,
    },
  },
  lambdaOptions: {
    runtime: awscdk.LambdaRuntime.NODEJS_18_X,
    bundlingOptions: {
      externals: ['aws-sdk'],
      sourcemap: true,
    },
  },
  jsiiVersion: '5.0.6',
});

Task "build » package" failed when executing "pdk@pnpm-link-bundled-transitive-deps

Expected Behavior

Successful build. Doesn't require a manual copy of the scripts from nx-mono-repo project.

Current Behavior

@nsa/construct: 👾 build » package | pdk@pnpm-link-bundled-transitive-deps packages/constructs
@nsa/construct: /bin/sh: pdk@pnpm-link-bundled-transitive-deps: command not found
@nsa/construct: 👾 Task "build » package" failed when executing "pdk@pnpm-link-bundled-transitive-deps packages/constructs" (cwd: /Users/rverma/dev/aws/nsa/packages/constructs)
@nsa/construct:  ELIFECYCLE  Command failed with exit code 1.

Reproduction Steps

simply use above projen.ts file and build any construct

Possible Solution

Had to copy the scripts from https://github.com/aws/aws-prototyping-sdk/tree/mainline/packages/nx-monorepo/scripts/pnpm to my construct subproject manually.

Additional Information/Context

No response

PDK version used

0.17.0

What languages are you seeing this issue on?

Typescript

Environment details (OS name and version, etc.)

mac m1

[JeremyJonas]

Hi @rverma-nsl, thank you for reporting this issue. Seems that the aws-prototyping-sdk library which bundles all the stable packages in this project does not marshal the "bin" files for each package.

If you use @aws-prototyping-sdk/nx-monorepo directly, then it should work just fine - I have confirmed using npx projen new --from @aws-prototyping-sdk/nx-monorepo.

We are actually going to deprecate the bundled aws-protoypting-sdk package since it causes many similar issues and maintenance without much benefit. I will update the documentation today to replace the usage of it to prevent others from this issue.

Let me know if using @aws-prototyping-sdk/nx-monorepo package directly solves you issues - cheers!

[rverma-dev]

@JeremyJonas seems like bootstrap using above command doesn't work out of box, getting an exception like

Error: Command failed: npm run eslint --if-present
👾 eslint | yarn exec nx run-many --target=eslint --output-style=stream --nx-bail
nx run-many

Run target for multiple listed projects

Options:
      --help               Show help                                                                                                                                                                                                                  [boolean]
      --version            Show version number                                                                                                                                                                                                        [boolean]
  -c, --configuration      This is the configuration to use when performing tasks on projects                                                                                                                                                          [string]
  -t, --targets, --target  Tasks to run for affected projects                                                                                                                                                                               [string] [required]
      --output-style       Defines how Nx emits outputs tasks logs                                                                                                                 [string] [choices: "dynamic", "static", "stream", "stream-without-prefixes"]
      --exclude            Exclude certain projects from being processed                                                                                                                                                                 [string] [default: ""]
      --parallel           Max number of parallel processes [default is 3]                                                                                                                                                                             [string]
      --runner             This is the name of the tasks runner configured in nx.json                                                                                                                                                                  [string]
      --graph              Show the task graph of the command                                                                                                                                                                        [boolean] [default: false]
      --verbose            Prints additional information about the commands (e.g., stack traces)                                                                                                                                                      [boolean]
      --nx-bail            Stop command execution after the first failed task                                                                                                                                                        [boolean] [default: false]
      --nx-ignore-cycles   Ignore cycles in the task graph                                                                                                                                                                           [boolean] [default: false]
      --skip-nx-cache      Rerun the tasks even when the results are available in the cache                                                                                                                                          [boolean] [default: false]
  -p, --projects           Projects to run. (comma/space delimited project names and/or patterns)                                                                                                                                                      [string]
      --all                [deprecated] Run the target on all projects in the workspace                                                                                                                                               [boolean] [default: true]

Examples:
  run-many --target=test                                          Test all projects
  run-many --target=test --projects=proj1,proj2                   Test proj1 and proj2 in parallel
  run-many --target=test --projects=proj1,proj2 --parallel=5      Test proj1 and proj2 in parallel using 5 workers
  run-many --target=test --projects=proj1,proj2 --parallel=false  Test proj1 and proj2 in sequence
  run-many --target=test --projects=*-app --exclude excluded-app  Test all projects ending with `*-app` except `excluded-app`.  Note: your shell may require you to escape the `*` like this: `\*`
  run-many --targets=lint,test,build --all                        Run lint, test, and build targets for all projects. Requires Nx v15.4+

Find more information and examples at https://nx.dev/nx/run-many

Missing required argument: targets
error Command failed.
Exit code: 1
Command: nx
Arguments: run-many
Directory: /Users/rverma/dev/aws/nsb
Output:

👾 Task "eslint" failed when executing "yarn exec nx run-many --target=eslint --output-style=stream --nx-bail" (cwd: /Users/rverma/dev/aws/nsb)

    at checkExecSyncError (node:child_process:885:11)
    at Object.execSync (node:child_process:957:15)
    at exec (/Users/rverma/.npm/_npx/80cfb0dc84733c29/node_modules/projen/lib/util.js:15:19)
    at initProject (/Users/rverma/.npm/_npx/80cfb0dc84733c29/node_modules/projen/lib/cli/cmds/new.js:325:25)
    at initProjectFromModule (/Users/rverma/.npm/_npx/80cfb0dc84733c29/node_modules/projen/lib/cli/cmds/new.js:279:11)
    at Object.handler (/Users/rverma/.npm/_npx/80cfb0dc84733c29/node_modules/projen/lib/cli/cmds/new.js:94:16)
    at Object.runCommand (/Users/rverma/.npm/_npx/80cfb0dc84733c29/node_modules/projen/node_modules/yargs/build/index.cjs:446:48)
    at Object.parseArgs [as _parseArgs] (/Users/rverma/.npm/_npx/80cfb0dc84733c29/node_modules/projen/node_modules/yargs/build/index.cjs:2706:53)
    at Object.runCommand (/Users/rverma/.npm/_npx/80cfb0dc84733c29/node_modules/projen/node_modules/yargs/build/index.cjs:410:36) {
  status: 1,
  signal: null,
  output: [
    null,
    null,
    <Buffer f0 9f 91 be 20 1b 5b 33 37 6d 1b 5b 34 6d 65 73 6c 69 6e 74 1b 5b 32 34 6d 20 7c 20 79 61 72 6e 20 65 78 65 63 20 6e 78 20 72 75 6e 2d 6d 61 6e 79 20 ... 5094 more bytes>
  ],
  pid: 34835,
  stdout: null,
  stderr: <Buffer f0 9f 91 be 20 1b 5b 33 37 6d 1b 5b 34 6d 65 73 6c 69 6e 74 1b 5b 32 34 6d 20 7c 20 79 61 72 6e 20 65 78 65 63 20 6e 78 20 72 75 6e 2d 6d 61 6e 79 20 ... 5094 more bytes>
}

[rverma-dev]

If we ignore above error, I believe the issue is still not fully addressed. I am expecting that with the above changes in place I should be able to run package target without specifying the bin

e.g.

import { NxMonorepoProject } from "@aws-prototyping-sdk/nx-monorepo";
import { awscdk } from 'projen';
import { NodePackageManager } from "projen/lib/javascript";

const AWS_SDK_VERSION = '^3.316.0';
const CDK_VERSION = '2.76.0';
const CONSTRUCT_VERSION = '10.2.4';
const JEST_VERION = '^29.1.0';

const project = new NxMonorepoProject({
  defaultReleaseBranch: "main",
  devDeps: ["@aws-prototyping-sdk/nx-monorepo"],
  name: "nsb",
  projenrcTs: true,
  packageManager: NodePackageManager.PNPM,
});

new awscdk.AwsCdkConstructLibrary({
  parent: project,
  outdir: 'packages/constructs',
  bundledDeps: [
    `@aws-sdk/client-iam@${AWS_SDK_VERSION}`,
    `@aws-sdk/client-secrets-manager@${AWS_SDK_VERSION}`,
  ],
  defaultReleaseBranch: 'main',
  name: '@nsa/construct',
  packageManager: NodePackageManager.PNPM,
  cdkVersion: CDK_VERSION,
  constructsVersion: CONSTRUCT_VERSION,
  minNodeVersion: project.minNodeVersion,
  jestOptions: {
    jestVersion: JEST_VERION,
    jestConfig: {
      detectOpenHandles: true,
    },
  },
  author: 'Rohit',
  authorAddress: 'rohit.verma@nslhub.com',
  repositoryUrl: 'https://github.com/rverma-nsl/nsl-saas-accelerator.git',
});
project.synth();

However it still results in same exception

@nsa/construct: /bin/sh: pdk@pnpm-link-bundled-transitive-deps: command not found
@nsa/construct: 👾 Task "package" failed when executing "pdk@pnpm-link-bundled-transitive-deps packages/constructs" (cwd: /Users/rverma/dev/aws/nsb/packages/constructs)

To resolve this I need to specify bin linkings explicitly like

bin: {
    'pdk@pnpm-link-bundled-transitive-deps':
      './node_modules/@aws-prototyping-sdk/nx-monorepo/scripts/pnpm/link-bundled-transitive-deps.ts',
// or local path to avoid chmod a+X above location
  },

[rverma-dev]

@JeremyJonas it seems not working if using awscdk.AwsCdkConstructLibrary and now there isn't any workaround. The error statement also changed

👾 build » package | pdk-pnpm-link-bundled-transitive-deps packages/constructs
@nsa/construct: /Users/rverma/dev/aws/nsa/node_modules/.pnpm/@aws-prototyping-sdk+nx-monorepo@0.17.4_@pnpm+logger@5.0.0_nx@15.9.2_projen@0.71.28/node_modules/@aws-prototyping-sdk/nx-monorepo/scripts/pnpm/link-bundled-transitive-deps.ts:4
@nsa/construct: import * as path from "node:path";
@nsa/construct: ^^^^^^
@nsa/construct: SyntaxError: Cannot use import statement outside a module
@nsa/construct:     at internalCompileFunction (node:internal/vm:73:18)
@nsa/construct:     at wrapSafe (node:internal/modules/cjs/loader:1176:20)
@nsa/construct:     at Module._compile (node:internal/modules/cjs/loader:1218:27)
@nsa/construct:     at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
@nsa/construct:     at Object.require.extensions.<computed> [as .ts] (/Users/rverma/dev/aws/nsa/node_modules/.pnpm/ts-node@10.9.1_@types+node@18.15.12_typescript@5.0.4/node_modules/ts-node/src/index.ts:1608:43)
@nsa/construct:     at Module.load (node:internal/modules/cjs/loader:1117:32)
@nsa/construct:     at Function.Module._load (node:internal/modules/cjs/loader:958:12)
@nsa/construct:     at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
@nsa/construct:     at phase4 (/Users/rverma/dev/aws/nsa/node_modules/.pnpm/ts-node@10.9.1_@types+node@18.15.12_typescript@5.0.4/node_modules/ts-node/src/bin.ts:649:14)
@nsa/construct:     at bootstrap (/Users/rverma/dev/aws/nsa/node_modules/.pnpm/ts-node@10.9.1_@types+node@18.15.12_typescript@5.0.4/node_modules/ts-node/src/bin.ts:95:10)
@nsa/construct: 👾 Task "build » package" failed when executing "pdk-pnpm-link-bundled-transitive-deps packages/constructs" (cwd: /Users/rverma/dev/aws/nsa/packages/constructs)
@nsa/construct:  ELIFECYCLE  Command failed with exit code 1.

[zsstiers]

I am also encountering the most recently posted error. The behavior I seem to be encountering from ts-node is that it doesn't default to treating pdk-pnpm-link-bundled-transitive-deps file as a module and so fails to import. Maybe either compiling it to JS and directly using node or just using a different execution engine would fix it (tsx seems to work)?

[JeremyJonas]

@zsstiers Thanks for reporting - I have re-opened. I think changing to tsx (https://www.npmjs.com/package/tsx) is probably a good approach, and will be useful in the toolchain for similar issues later on.

Feel free to send a PR for that if your able to - otherwise will probably be next week before can get to resolving this.

[agdimech]

Resolved in #546