vercel / turborepo

Build system optimized for JavaScript and TypeScript, written in Rust
https://turbo.build/repo/docs
MIT License
26.46k stars 1.84k forks source link

Strip in-workspace devDependencies when pruning #1100

Open marvinroger opened 2 years ago

marvinroger commented 2 years ago

Describe the feature you'd like to request

Hi,

First of all, thanks for this awesome project, it's really powerful!

We're currently using the great turbo prune feature, and we noticed a potential optimization. Imagine a project app having two "in-workspace" dependencies:

Currently, running turbo prune --scope=app will generate a directory out/ with app, lib-config and tooling. In our case, we run turbo prune in our CI after all packages have been built. Therefore, we don't need the devDependencies in our out/ directory.

Ideally, our pruned out/ directory would contain only app and lib-config, which are the only packages needed at runtime.

Describe the solution you'd like

Having an optional --production flag that would ignore the devDependencies would be great.

Describe alternatives you've considered

I don't think there are any alternatives, except keeping the devDependencies, which is frankly not a big deal. It's just less optimal... But, isn't Turborepo all about optimization? 😉

Ghirigoro commented 2 years ago

This not always just a question of optimizing file size. I have a situation with a backend project that is strictly node, but uses some browser-based utilities when testing. In order to build those browser dependencies that get pulled in I also have to manually pull in the resources required to build those dependencies (for example I have separate base tsconfig.json files for node and for browser at the root of the monorepo). It's not the end of the world but you start to quickly lose the point of pruning if you're pulling in half your monorepo and littering your docker images with resources that are only used to workaround this issue.

P.S. - I'm realizing my tone probably comes off as more stern than I mean to. I agree it's not a huge deal but it would make Turborepo even more awesome if it wasn't the case :)

azertyalex commented 1 year ago

@joemckenney care to share your rm-dev-deps.mjs script referenced in issue #1488 ? I'm having the same problem as you, 1GB docker image

joemckenney commented 1 year ago

@joemckenney care to share your rm-dev-deps.mjs script referenced in issue #1488 ? I'm having the same problem as you, 1GB docker image

@azertyalex Here's a gist https://gist.github.com/joemckenney/fd55f02ab0bf905283423aadeeca80ca

The script uses a package we built called @dopt/wutils - which assumes you use pnpm. It's open source but not published to any registry - you can find it here https://github.com/dopt/odopt/tree/main/packages/%40dopt/wutils and take inspiration from it or copy pieces to recreate the functionality.

Some additional context - this problem was solved by our shift from yarn to pnpm and pnpm's deploy command.

azertyalex commented 1 year ago

okay, thanks! I stumbled upon pnpm myself in the last hour, that seems the way to go

gajus commented 1 year ago

Tagging core team member to make sure this gets seen since it is a pretty major issue @gsoltis

Annoyingly, pnpm v8 introduced a regression where it broke pnpm deploy so there isn't even a fallback option.

Related pnpm issues:

mehulkar commented 1 year ago

A --production flag does seem like a good idea.

cc @chris-olszewski, how hard would this be to do? I got a little lost reading the prune code with the transitive_closure stuff, so I wasn't able to get an easy answer

chris-olszewski commented 12 months ago

This is something we do want to add, but a few things need to happen first:

mehulmpt commented 7 months ago

Faced this today where one of our packages required some devDependencies for building it but turbo prune is pulling them inside production build the way our pipeline is built. We have to write a workaround for this but it would be great if it was natively available.

wrporter commented 4 months ago

Just came across this problem when dockerizing my app today. The only decent workaround I could think of is to extract packages that are devDependencies in my apps to a separate repo, publish those to a registry, and install them that way. Definitely not a great workaround in cases where you don't want to publish those.

WINOFFRG commented 3 months ago

Facing same issue my workspace has some tooling packages in devDeps and they are getting bundled in image. Some packages like prettier typescript are quite heavy like 22MB

image
abcdefghiraj commented 2 months ago

I spent some time struggling with the same issue. I am using pnpm and I tried pnpm deploy, but that messes with the symlinks. If I add the option to inject, then turbo complains.

Till there is a fix, here is a possible workaround - I'm compiling the build output to a single file (using vercel/ncc - can also be tsup or something similar) and only copying that output to the final layer of my docker image, and I am at a third of the previous size. Be mindful of the pitfalls of this approach and make sure you test all code paths by bundling and running this locally before going to production