Full cache build still very slow

[MichealXie]

What version of Turborepo are you using?

1.2.16

What package manager are you using / does the bug impact?

Yarn v1

What operating system are you using?

Mac

Describe the Bug

When I run turbo run build --filter=./packages/*, it can successfully cache and replaying, but the speed is very slow compare to

The image says build 102 job with 370ms, but my build speed (without remote cache) is

and the build command are all like:

    "build": "npm-run-all -p build:es build:cjs",
    "build:cjs": "tsc -p tsconfig.build.json -m commonjs --outDir dist/lib --declaration false",
    "build:es": "tsc -p tsconfig.build.json -m esNext --outDir dist/esm --declarationDir dist/types",

Expected Behavior

As fast as you advertise, 25 packages build time within 1 second.

To Reproduce

I don't know it's a bug or design in this way, and all my code are company's code so sorry that I can't show it.

[gsoltis]

For slow builds is it possible for you get a profile of your execution? turbo run build --filter=./packages/* --profile=my-build.prof? That will help us see where turbo is spending time.

Another avenue of investigation is regarding size of your outputs. Do you have an estimate for the file size of the artifacts you're producing in a build? This is probably the dist/ folders.

[MichealXie]

@gsoltis The 25 package's dist folder size add together is around 100mb.

I try run turbo run build --filter=./packages/* --profile=my-build.prof & turbo run build --filter=./packages/* both take about 20 seconds with FULL TURBO.

Here is the prof result:

If you need anything else please give me a comment, I really want to boost my build speed in my local env.

[nathanhammond]

@MichealXie We actually need that profile file itself so that we can investigate it. Can you attach it? Or you can send it to me privately: nathan.hammond@vercel.com

[vincent-lecrubier-skydio]

Hi, I have a pretty nasty case of this here on turbo@1.4.6, turbo@1.3.4 and turbo@1.2.16, on node@14.18.2 and node@16.17.0:

Uncached build:

Full turbo build:

I am going to email the profile to nathan.hammond@vercel.com now.

Interestingly, this time went from 30s to 3min after I migrated one of our applications from CRA to NextJS. Could such a change cause that ?

[tknickman]

Interesting, thanks for sending the trace over - we'll take a look.

For anyone else who finds this in the future - please send traces to turbo-core@vercel.com so we don't only drown @nathanhammond with these and we can hopefully get you a resolution faster 😄

[nathanhammond]

@tknickman that email address doesn't accept external messages. 😅

[tknickman]

In that case. Keep drowning @nathanhammond! 😂

[nathanhammond]

Given that the second run was about the same amount of time, I have a suspicion that the caches are failing to restore. Matching by key, but during a failure to restore we may then incorrectly report the cache hit.

[nathanhammond]

(I haven't reviewed profiles yet, will forward.)

[vincent-lecrubier-skydio]

Ok so on my side, I have clearly identified that the one nextjs workspace is causing trouble: removing it takes the fully cached time down to 16s.

I don't think my config is doing anything exceptional, and my nextjs project is also pretty basic. Not clear why it is causing trouble. 🤔

[nathanhammond]

Still just looking at the total durations, the difference between your runs and when you pop out that one task makes me feel pretty good about my hypothesis.

[vincent-lecrubier-skydio]

Cool! Let me know if you think there is a workaround / solution I need to take action on, or if you plan on a fix on your side

[vincent-lecrubier-skydio]

Thanks to @nathanhammond I was able to find the source of my problem:

I was adding my .next folder as output of my build step 😱 . This folder includes caches, and these caches were 8GB, hence the long time to restore cache.

So in my case the fix is to take my turbo.json config file in essence from this:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "outputs": [".next/**", "build/**", "out/**", "gen/**", "dist/**"],
    }
  }
}

to this:

{
  "$schema": "https://turborepo.org/schema.json",
  "pipeline": {
    "build": {
      "outputs": ["build/**", "out/**", "gen/**", "dist/**"],
    }
  }
}

This took down my full turbo time back to 4 seconds 👍

[mehulkar]

Looks like this is resolved with better config, thanks for reporting the update @vincent-lecrubier-skydio!