search

vercel / turborepo

Build system optimized for JavaScript and TypeScript, written in Rust
https://turbo.build/repo/docs
MIT License
26.21k stars 1.81k forks source link

npx create-turbo@latest fails with `[Circular *1]` error #2293

Closed kathodler closed 1 year ago

kathodler commented 2 years ago

What version of Turborepo are you using?

1.5.6

What package manager are you using / does the bug impact?

npm, pnpm

What operating system are you using?

Linux (NixOS)

Describe the Bug

I cannot install turborepo with the create-turbo@latest installer because I get an error.

Steps to reproduce:

npx create-turbo@latest
# notice failure in output
cd my-turborepo
npm install 
# Get the following error below

Also occurs in a similar way with pnpm have not yet tried yarn

Versions

npm --version
8.19.1

pnpm --version
7.13.3

node --version
v18.9.1

cat /etc/os-release
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"
BUILD_ID="22.11pre416790.f634d427b02"
DOCUMENTATION_URL="https://nixos.org/learn.html"
HOME_URL="https://nixos.org/"
ID=nixos
LOGO="nix-snowflake"
NAME=NixOS
PRETTY_NAME="NixOS 22.11 (Raccoon)"
SUPPORT_URL="https://nixos.org/community.html"
VERSION="22.11 (Raccoon)"
VERSION_CODENAME=raccoon
VERSION_ID="22.11"

Expected Behavior

No error.

To Reproduce

$ npx create-turbo@latest

>>> TURBOREPO

>>> Welcome to Turborepo! Let's get you set up with a new codebase.

? Where would you like to create your turborepo? ./my-turborepo
? Which package manager do you want to use? npm

>>> Created a new turborepo with the following:

 - apps/web: Next.js with TypeScript
 - apps/docs: Next.js with TypeScript
 - packages/ui: Shared React component library
 - packages/eslint-config-custom: Shared configuration (ESLint)
 - packages/tsconfig: Shared TypeScript `tsconfig.json`

>>  Installing dependencies...
Aborting installation.
  npm install has failed.

❯ ls
my-turborepo  prototype
❯ cd my-turborepo
❯ npm install
npm ERR! code 1
npm ERR! path /home/ktdlr/gh/sharelane/my-turborepo/node_modules/turbo
npm ERR! command failed
npm ERR! command sh -c -- node install.js
npm ERR! node:internal/errors:484
npm ERR!     ErrorCaptureStackTrace(err);
npm ERR!     ^
npm ERR!
npm ERR! <ref *1> Error: spawnSync /home/ktdlr/gh/sharelane/my-turborepo/node_modules/turbo/bin/turbo ENOENT
npm ERR!     at Object.spawnSync (node:internal/child_process:1110:20)
npm ERR!     at spawnSync (node:child_process:857:24)
npm ERR!     at Object.execFileSync (node:child_process:900:15)
npm ERR!     at validateBinaryVersion (/home/ktdlr/gh/sharelane/my-turborepo/node_modules/turbo/install.js:25:6)
npm ERR!     at /home/ktdlr/gh/sharelane/my-turborepo/node_modules/turbo/install.js:308:5 {
npm ERR!   errno: -2,
npm ERR!   code: 'ENOENT',
npm ERR!   syscall: 'spawnSync /home/ktdlr/gh/sharelane/my-turborepo/node_modules/turbo/bin/turbo',
npm ERR!   path: '/home/ktdlr/gh/sharelane/my-turborepo/node_modules/turbo/bin/turbo',
npm ERR!   spawnargs: [ '--version' ],
npm ERR!   error: [Circular *1],
npm ERR!   status: null,
npm ERR!   signal: null,
npm ERR!   output: null,
npm ERR!   pid: 0,
npm ERR!   stdout: null,
npm ERR!   stderr: null
npm ERR! }
npm ERR!
npm ERR! Node.js v18.9.1

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ktdlr/.npm/_logs/2022-10-21T12_28_20_023Z-debug-0.log
nathanhammond commented 2 years ago

The error is here:

Error: spawnSync /home/ktdlr/gh/sharelane/my-turborepo/node_modules/turbo/bin/turbo ENOENT

I suspect you have a bad local package cache:

npm cache clean --force
rm -rf `pnpm store path`
rm -rf /home/ktdlr/gh/sharelane/my-turborepo
cd /home/ktdlr/gh/sharelane/
pnpm dlx create-turbo@latest

I'm closing this because I do not suspect that this is an issue with turbo itself. Let me know if this doesn't address your issue and we'll be happy to reopen!

kathodler commented 2 years ago

That didn't address the issue. The file exists (binary) and pnpm/npm cache has been cleared. Are there any global paths turbo expect to be available/writable? Installing next (pnpm create next-app) works fine.

nathanhammond commented 2 years ago

node install.js is the post-install script for installing turbo. Can you try the smallest possible test?

mkdir issue-2293 && cd issue-2293 && npm init && npm install turbo --save-dev

kathodler commented 2 years ago 
❯ mkdir issue-2293 && cd issue-2293 && npm init && npm install turbo --save-dev
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help init` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (issue-2293)
version: (1.0.0)
description:
entry point: (index.js)
test command:
git repository:
keywords:
author:
license: (ISC)
About to write to /home/ktdlr/gh/issue-2293/package.json:

{
  "name": "issue-2293",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

Is this OK? (yes)
npm ERR! code 1
npm ERR! path /home/ktdlr/gh/issue-2293/node_modules/turbo
npm ERR! command failed
npm ERR! command sh -c -- node install.js
npm ERR! node:internal/errors:484
npm ERR!     ErrorCaptureStackTrace(err);
npm ERR!     ^
npm ERR!
npm ERR! <ref *1> Error: spawnSync /home/ktdlr/gh/issue-2293/node_modules/turbo/bin/turbo ENOENT
npm ERR!     at Object.spawnSync (node:internal/child_process:1110:20)
npm ERR!     at spawnSync (node:child_process:857:24)
npm ERR!     at Object.execFileSync (node:child_process:900:15)
npm ERR!     at validateBinaryVersion (/home/ktdlr/gh/issue-2293/node_modules/turbo/install.js:25:6)
npm ERR!     at /home/ktdlr/gh/issue-2293/node_modules/turbo/install.js:308:5 {
npm ERR!   errno: -2,
npm ERR!   code: 'ENOENT',
npm ERR!   syscall: 'spawnSync /home/ktdlr/gh/issue-2293/node_modules/turbo/bin/turbo',
npm ERR!   path: '/home/ktdlr/gh/issue-2293/node_modules/turbo/bin/turbo',
npm ERR!   spawnargs: [ '--version' ],
npm ERR!   error: [Circular *1],
npm ERR!   status: null,
npm ERR!   signal: null,
npm ERR!   output: null,
npm ERR!   pid: 0,
npm ERR!   stdout: null,
npm ERR!   stderr: null
npm ERR! }
npm ERR!
npm ERR! Node.js v18.9.1

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/ktdlr/.npm/_logs/2022-10-22T08_39_18_286Z-debug-0.log
kathodler commented 2 years ago

So this time the file does not exist: 

❯ ls -la  /home/ktdlr/gh/issue-2293/node_modules/turbo/bin/turbo
ls: cannot access '/home/ktdlr/gh/issue-2293/node_modules/turbo/bin/turbo': No such file or directory
kathodler commented 2 years ago

In Nixos occasionally things live in different places to normal Linux and certain config folders can be readonly. I tried ubuntu on a VM and it works fine. I could try nixos on a vm with a more minimal config but I need to download the iso etc. which will take time. Don't know for certain but it might be some kind of Nixos incompatibility based on an assumption. Would be good to hear from other Nixos users.

rafaeltab commented 2 years ago

I am experiencing a similar or the same issue, the issue occurs inside the build of a Dockerfile using the following command: yarn dlx turbo prune --scope=@kurabu/api --docker I tried this same command on every version down to 1.5.5, which is the first time it succeeds, the command is then: yarn dlx turbo@1.5.5 prune --scope=@kurabu/api --docker.

The logs are:

➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed in 1s 524ms
➤ YN0000: ┌ Fetch step
➤ YN0013: │ turbo-linux-64@npm:1.6.1 can't be found in the cache and will be fetched from the remote registry
➤ YN0013: │ turbo@npm:1.6.1 can't be found in the cache and will be fetched from the remote registry                                                   
➤ YN0000: └ Completed in 6s 195ms
➤ YN0000: ┌ Link step
➤ YN0007: │ turbo@npm:1.6.1 must be built because it never has been before or the last one failed
➤ YN0009: │ turbo@npm:1.6.1 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-ddfa7c86/build.log)
➤ YN0000: └ Completed in 0s 309ms
➤ YN0000: Failed with errors in 8s 37ms

and the /tmp/xfs-ddfa7c86/build.log file

# This file contains the result of Yarn building a package (turbo@npm:1.5.6)
# Script name: postinstall

node:internal/errors:857
  const err = new Error(message);
              ^

Error: Command failed: node /tmp/xfs-8fd21b48/dlx-899/node_modules/turbo/bin/turbo --version
/tmp/xfs-8fd21b48/dlx-899/node_modules/turbo/bin/turbo:13
    throw e;
    ^

<ref *1> Error: spawnSync /tmp/xfs-8fd21b48/dlx-899/node_modules/turbo-linux-64/bin/turbo ENOENT
    at Object.spawnSync (node:internal/child_process:1119:20)
    at spawnSync (node:child_process:847:24)
    at Object.execFileSync (node:child_process:890:15)
    at Object.<anonymous> (/tmp/xfs-8fd21b48/dlx-899/node_modules/turbo/bin/turbo:6:28)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10)
    at Module.load (node:internal/modules/cjs/loader:1033:32)
    at Function.Module._load (node:internal/modules/cjs/loader:868:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:22:47 {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawnSync /tmp/xfs-8fd21b48/dlx-899/node_modules/turbo-linux-64/bin/turbo',
  path: '/tmp/xfs-8fd21b48/dlx-899/node_modules/turbo-linux-64/bin/turbo',
  spawnargs: [ '--version' ],
  error: [Circular *1],
  status: null,
  signal: null,
  output: null,
  pid: 924,
  stdout: null,
  stderr: null
}

    at checkExecSyncError (node:child_process:861:11)
    at Object.execFileSync (node:child_process:896:15)
    at validateBinaryVersion (/tmp/xfs-8fd21b48/dlx-899/node_modules/turbo/install.js:25:6)
    at /tmp/xfs-8fd21b48/dlx-899/node_modules/turbo/install.js:305:5 {
  status: 1,
  signal: null,
  output: [
    null,
    Buffer(0) [Uint8Array] [],
    Buffer(1185) [Uint8Array] [
       47, 116, 109, 112,  47, 120, 102, 115,  45,  56, 102, 100,
       50,  49,  98,  52,  56,  47, 100, 108, 120,  45,  56,  57,
       57,  47, 110, 111, 100, 101,  95, 109, 111, 100, 117, 108,
      101, 115,  47, 116, 117, 114,  98, 111,  47,  98, 105, 110,
       47, 116, 117, 114,  98, 111,  58,  49,  51,  10,  32,  32,
       32,  32, 116, 104, 114, 111, 119,  32, 101,  59,  10,  32,
       32,  32,  32,  94,  10,  10,  60, 114, 101, 102,  32,  42,
       49,  62,  32,  69, 114, 114, 111, 114,  58,  32, 115, 112,
       97, 119, 110,  83,
      ... 1085 more items
    ]
  ],
  pid: 917,
  stdout: Buffer(0) [Uint8Array] [],
  stderr: Buffer(1185) [Uint8Array] [
     47, 116, 109, 112,  47, 120, 102, 115,  45,  56, 102, 100,
     50,  49,  98,  52,  56,  47, 100, 108, 120,  45,  56,  57,
     57,  47, 110, 111, 100, 101,  95, 109, 111, 100, 117, 108,
    101, 115,  47, 116, 117, 114,  98, 111,  47,  98, 105, 110,
     47, 116, 117, 114,  98, 111,  58,  49,  51,  10,  32,  32,
     32,  32, 116, 104, 114, 111, 119,  32, 101,  59,  10,  32,
     32,  32,  32,  94,  10,  10,  60, 114, 101, 102,  32,  42,
     49,  62,  32,  69, 114, 114, 111, 114,  58,  32, 115, 112,
     97, 119, 110,  83,
    ... 1085 more items
  ]
}

The problem does not only occur with the prune command.

Reproducing Dockerfile:

FROM node:16.18-alpine AS base
# 1.5.6-1.6.1 will all error
# 1.5.5 will not
ENV TURBO_VERSION 1.5.6 
ENV YARN_VERSION 3.2.4
RUN yarn policies set-version $YARN_VERSION
RUN yarn dlx turbo@$TURBO_VERSION --version

More info: Yarn version: 3.2.4 Problematic turbo versions: 1.5.6-1.6.1 Architectures: At least linux/amd64 and linux/arm64

The yarn dlx turbo --version command, with the same yarn version, DOES succeed on my regular windows PRO 11 machine, but not inside a docker build.

kathodler commented 1 year ago

Can confirm the above test works on my Nixos setup with version 1.5.5:

❯ mkdir issue-2293 && cd issue-2293 && npm init && npm install turbo@1.5.5 --save-dev
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help init` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (issue-2293)
version: (1.0.0)
description:
entry point: (index.js)
test command:
git repository:
keywords:
author:
license: (ISC)
About to write to /home/ktdlr/gh/issue-2293/package.json:

{
  "name": "issue-2293",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC"
}

Is this OK? (yes)

added 2 packages, and audited 3 packages in 2s

found 0 vulnerabilities
nathanhammond commented 1 year ago

Fantastic. This is probably enough information for me to get a root cause. Unfortunately that root cause is possibly "switched to CGO."

nathanhammond commented 1 year ago

@rafaeltab did you get the docker/libc fix in place for Alpine? (Search issues.)

rafaeltab commented 1 year ago

@nathanhammond I have never heard of that before, I'll take a look at it right now for ya.

rafaeltab commented 1 year ago

@nathanhammond

Indeed what you seem to be describing is the issue. I ended up finding this repository: https://github.com/Docker-Hub-frolvlad/docker-alpine-glibc/blob/master/Dockerfile

It adds glibc to the alpine docker image, I applied the same commands to the docker image in https://github.com/vercel/turborepo/issues/2293#issuecomment-1287876271

Resulting in the following Dockerfile

FROM node:16.18-alpine AS base

ENV TURBO_VERSION 1.6.1
ENV YARN_VERSION 3.2.4

ENV LANG=C.UTF-8

# Here we install GNU libc (aka glibc) and set C.UTF-8 locale as default.

RUN ALPINE_GLIBC_BASE_URL="https://github.com/sgerrand/alpine-pkg-glibc/releases/download" && \
    ALPINE_GLIBC_PACKAGE_VERSION="2.35-r0" && \
    ALPINE_GLIBC_BASE_PACKAGE_FILENAME="glibc-$ALPINE_GLIBC_PACKAGE_VERSION.apk" && \
    ALPINE_GLIBC_BIN_PACKAGE_FILENAME="glibc-bin-$ALPINE_GLIBC_PACKAGE_VERSION.apk" && \
    ALPINE_GLIBC_I18N_PACKAGE_FILENAME="glibc-i18n-$ALPINE_GLIBC_PACKAGE_VERSION.apk" && \
    apk add --no-cache --virtual=.build-dependencies wget ca-certificates && \
    echo \
        "-----BEGIN PUBLIC KEY-----\
        MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEApZ2u1KJKUu/fW4A25y9m\
        y70AGEa/J3Wi5ibNVGNn1gT1r0VfgeWd0pUybS4UmcHdiNzxJPgoWQhV2SSW1JYu\
        tOqKZF5QSN6X937PTUpNBjUvLtTQ1ve1fp39uf/lEXPpFpOPL88LKnDBgbh7wkCp\
        m2KzLVGChf83MS0ShL6G9EQIAUxLm99VpgRjwqTQ/KfzGtpke1wqws4au0Ab4qPY\
        KXvMLSPLUp7cfulWvhmZSegr5AdhNw5KNizPqCJT8ZrGvgHypXyiFvvAH5YRtSsc\
        Zvo9GI2e2MaZyo9/lvb+LbLEJZKEQckqRj4P26gmASrZEPStwc+yqy1ShHLA0j6m\
        1QIDAQAB\
        -----END PUBLIC KEY-----" | sed 's/   */\n/g' > "/etc/apk/keys/sgerrand.rsa.pub" && \
    wget \
        "$ALPINE_GLIBC_BASE_URL/$ALPINE_GLIBC_PACKAGE_VERSION/$ALPINE_GLIBC_BASE_PACKAGE_FILENAME" \
        "$ALPINE_GLIBC_BASE_URL/$ALPINE_GLIBC_PACKAGE_VERSION/$ALPINE_GLIBC_BIN_PACKAGE_FILENAME" \
        "$ALPINE_GLIBC_BASE_URL/$ALPINE_GLIBC_PACKAGE_VERSION/$ALPINE_GLIBC_I18N_PACKAGE_FILENAME" && \
    apk add --no-cache \
        "$ALPINE_GLIBC_BASE_PACKAGE_FILENAME" \
        "$ALPINE_GLIBC_BIN_PACKAGE_FILENAME" \
        "$ALPINE_GLIBC_I18N_PACKAGE_FILENAME" && \
    \
    rm "/etc/apk/keys/sgerrand.rsa.pub" && \
    (/usr/glibc-compat/bin/localedef --force --inputfile POSIX --charmap UTF-8 "$LANG" || true) && \
    echo "export LANG=$LANG" > /etc/profile.d/locale.sh && \
    \
    apk del glibc-i18n && \
    \
    rm "/root/.wget-hsts" && \
    apk del .build-dependencies && \
    rm \
        "$ALPINE_GLIBC_BASE_PACKAGE_FILENAME" \
        "$ALPINE_GLIBC_BIN_PACKAGE_FILENAME" \
        "$ALPINE_GLIBC_I18N_PACKAGE_FILENAME"

RUN yarn policies set-version $YARN_VERSION
RUN yarn dlx turbo@$TURBO_VERSION --version

Which builds without failing.

Using the non-alpine version of node also works:

FROM node:16.18 AS base
# 1.5.6-1.6.1 will all error
# 1.5.5 will not
ENV TURBO_VERSION 1.6.1
ENV YARN_VERSION 3.2.4

RUN yarn policies set-version $YARN_VERSION
RUN yarn dlx turbo@$TURBO_VERSION --version

The differences between these two files can be summarized as follows:

nathanhammond commented 1 year ago

So, @kathodler, I suspect that this is a nixos/libc thing. I know nothing about nix, but I'm hoping that you do since you're running it. 😅

Is there a fix that looks similar to what @rafaeltab describes for nix?

nathanhammond commented 1 year ago

@rafaeltab RUN apk add --no-cache libc6-compat

rafaeltab commented 1 year ago

@nathanhammond that works beautifully!

FROM node:16.18-alpine AS base
# 1.5.6-1.6.1 will all error
# 1.5.5 will not
ENV TURBO_VERSION 1.6.1
ENV YARN_VERSION 3.2.4

RUN apk add --no-cache libc6-compat

RUN yarn policies set-version $YARN_VERSION
RUN yarn dlx turbo@$TURBO_VERSION --version

And it is even smaller than the other fix, only 124MB!

dlip commented 1 year ago

Native tools need to be rebuilt or patched to work with nix. I have create a flake overlay which builds turbo and turbo-tooling if you want to try it. Repo at https://github.com/dlip/turbo

    turbo = {
      url = "github:dlip/turbo";
    };
nathanhammond commented 1 year ago

@dlip's comment confirms for me that the issue here is Nix. We don't officially support Nix, but are open to supporting it. If somebody wishes to open a PR to enable support in our build system and add a workflow to run our test suite against Nix we will accept it as a second tier platform (tests run on a schedule, not per-PR).

Pegasust commented 1 year ago

@dlip Thank you for putting effort to this. I'm new to Nix, and I've been trying different things to make this work. Currently, I want to provide a devShell for a turbo-repo with Prisma. It seems like the Prisma portion is okay. Could you provide a small turbo-repo example using your flake?

Pegasust commented 1 year ago

After some struggle, I was able to get this working with a devShell. I basically read from https://github.com/prisma/prisma/issues/3026#issuecomment-927258138 to make Prisma on NixOS possible. To sum up, it requires direnv (impure build & variable exports?) which exports variables to override the binary path. I missed this when reading dlip's comment

Here's a short snippet of how my setup is like.

Turbo version must match

Dlip's repository is on 1.6.2 at the time of writing, so we need to make sure our package.json reflects this. 

// package.json
{
  ...
  "devDependencies": {
    ...
    "turbo": "1.6.2"
  }
}

Until Nix support is official, we need to maintain a fork of turbo to this updated recently.

nix parts

# flake.nix
{
  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    turbo.url = "github:dlip/turbo";
    # ...
  };
  outputs = {self, nixpkgs, turbo,...}:
  let overlays = [turbo.overlay];
    system = "x86_64-linux";
    pkgs = import nixpkgs {inherit system overlays;};
  in {
    # see comment from Prisma's ticket
    nativeBuildInputs = [pkgs.bashInteractive];
    buildInputs = [
      pkgs.nodejs-18_x pkgs.nodePackages.pnpm #... (I have prisma here)
      pkgs.turbo # injected from the overlay
    ];
    # variable is documented here: https://github.com/vercel/turbo/blob/main/CONTRIBUTING.md#updating-turbo
    shellHook = ''
      export TURBO_BINARY_PATH="${pkgs.turbo}/bin/turbo"
    '';
  };
}
# .envrc
if command -v nix-shell &> /dev/null
then
  use flake
fi

Hope this helps!

dlip commented 1 year ago

@Pegasust nice work mate, that looks good. Sorry I've been too busy this week to reply. I'll try to put up a pr on nixpkgs so we can all use it easily

dlip commented 1 year ago

I have created the nixpkgs pr here: https://github.com/NixOS/nixpkgs/pull/200788