yarnpkg / yarn

The 1.x line is frozen - features and bugfixes now happen on https://github.com/yarnpkg/berry
https://classic.yarnpkg.com
Other
41.44k stars 2.73k forks source link

[FR] a way to ignore packages #4611

Open farfromrefug opened 7 years ago

farfromrefug commented 7 years ago

Right now in npm or yarn it's impossible to ignore packages. This makes it very difficult to package small node_modules folder.

Let me explain: In my apps i use many packages, some for which i need to keep optionalDependencies like noble. Some i dont want and which makes the node_modules folder a lot bigger, like log4js optional dependencies.

The consequence of that is that i can't use --ignore-optional.

So for example if i want, after a yarn install --flat --prod cleanup and remove log4js optionalDependencies, i simply cant! I would need to get the dependency tree for all the optional modules, make sure there are not used by anyone else, and then remove the folders manualy.

The best solution, even if not perfect, would be to be able in the package.json to add "ignored" packages. Those would be ignored in yarn install and thus would not alter the dependency tree. I would then simply have to add all log4js optional dependencies in my ignored section.

Would that be possible?

tvald commented 6 years ago

I second this request, both to manage package size and also to suppress misbehaving optional packages.

The Problem To elaborate, a package we rely on (good-package) has an optional dependency (bad-package) with unmanaged dependencies. That is, bad-package requires you to separately download and build some libfoo before running yarn install, which we never do because bad-package isn't useful to us. Unfortunately, this causes yarn install to always spew a lengthy warning when bad-package fails to install.

This optional dependency is obviously misbehaving, but we can't fix it, so instead we'd like to ignore it. We can't just use --ignore-optional everywhere, because we do need some optional dependencies like fsevents.

The Suggestion I suggest the repeatable flag --suppress-optional <package-name>, which would cause <package-name> to be ignored if <package-name> is a purely optional dependency, and would cause an error if <package-name> is actually required.

(For reference, see yarnpkg/yarn/pull/5059 for discussion of when a package is truly optional...)

tvald commented 6 years ago

Also, could we edit the title for clarity: Add way to ignore individual optional packages?

farfromrefug commented 6 years ago

@tvald i am not sure about the title cause maybe you you want to ignore non optional packages too!

tvald commented 6 years ago

If a non-optional package were suppressed, that would presumably break all packages which directly or transitively depend on it, including the top-level package.

It would be great if you can illustrate a case where a non-optional package can be suppressed without breaking things. Otherwise, I'd prefer the smaller and safer capability to suppress only transitively optional packages.

Do note that a required dependency of an optional dependency is considered optional, unless referenced elsewhere (again, transitively) as required.

farfromrefug commented 6 years ago

@tvald node-pre-gyp is one simple case. it comes with a lot of unecessary data for production. I already ignore it by removing it. And in my package.json i include my own package to replace it which only include what s necessary for production.

tvald commented 6 years ago

That's far more complicated, then. I'll split off a separate feature request if that's the kind of scenario you want to support with this issue. /shrug

zevdg commented 6 years ago

Another specific use case for this feature was raised in https://github.com/firebase/firebase-js-sdk/issues/546#issuecomment-370596013

Namely, firestore is an isomorphic package that includes large platform specific dependencies. If you are using their package for a website, it would be nice to able to tell yarn that it can ignore the dependencies that aren't specific to that platform.

petrgazarov commented 6 years ago

Another scenario where this can help is with Node.js usage on AWS lambda. Lambda has a 50MB code limit. Many packages (including one created by our team, for example) tend to include a large dependency tree. To fit into the space limit, one would tell yarn to ignore certain packages, assuming you know what you are doing and are ignoring only dependencies that you are not using in your code.

aneilbaboo commented 6 years ago

Another use case - fsevents:

A yarn.lock file generated on a Mac will include this dependency, so when code is deployed to a CI system running ubuntu, yarn finishes with an exit code 1. It would be nice to ignore this dependency in a particular environment.

Et0san commented 6 years ago

Still not a thing? I'm having problems rn with chromedriver getting overridden while bootstrapping, and I don't want that because it breaks while trying to download the .zip (behind corp proxy). So I want to tell yarn not to touch chromedriver, so it does not override my local modifications that should make it work -_- (I mean, I read through chromedriver/install.js, and I did what it does by myself in like twenty seconds...) Please tell me if you have a workaround, though

GongT commented 5 years ago

Some package have multiple function, and if I only use part of them, then some required dependencies can suppressed.
Currently we can use tools like rollup to do this After install.

(🤔 ok, I think this is rare)

I'm using gulp-sass, it's default to use node-sass, also can config to use sass instead.
But they write node-sass in dependencies... 💢

tvald commented 5 years ago

if I only use part of them, then some required dependencies can suppressed

Definitely a reasonable use case that I have encountered. We also use rollup in the build stage to reduce build artifact size.

GongT commented 5 years ago

Now look at my stupid.json: 🤣

{
    "name": "test",
    "version": "1.0.0",
    "private": true,
    "license": "WTFPL",
    "dependencies": {
        "debug": "^3.2.6"
    },
    "optionalDependencies": {
        "ms": "https://registry.npmjs.org/package-that-cannot-be-installed/-/package-that-cannot-be-installed-0.0.0.tgz"
    },
    "resolutions": {
        "ms": "https://registry.npmjs.org/package-that-cannot-be-installed/-/package-that-cannot-be-installed-0.0.0.tgz"
    }
}

This will install debug without required ms dependency

favna commented 4 years ago

Adding directly to @GongT's comment and since this issue is still open. I've released my own version of a similar idea but unlike the pacakge-that-cannot-be-installed package, this one doesn't force the usage of the --ignore-scripts flag for yarn install.

You can find it here:

Implementation:

{
    "name": "stupid.json",
    "version": "1.0.0",
    "private": true,
    "license": "WTFPL",
    "dependencies": {
        "debug": "^3.2.6"
    },
    "resolutions": {
        "ms": "https://registry.yarnpkg.com/@favware/skip-dependency/-/skip-dependency-1.0.2.tgz"
    }
}

Honestly don't ask me why. I don't know. It just bothered me that the previous one required --ignore-scripts 🤣.

Sidenote: it does really work. I was able to filter out jest-environment-jsdom from a pure NodeJS application as a test, yeeting out 5MB from the node_modules folder. If you're crunching for space this can help I guess.

tqwewe commented 4 years ago

This is also something needed for my project which is suffering from issues with scrypt.js and node 12.

Would love to install https://www.npmjs.com/package/ganache-core with --ignore-optionals set permanently.

JasonKleban commented 4 years ago

Jest doesn't need babel anymore, so I'd love to suppress the need to download that sprawling set of dependencies even before Jest is prepared to remove it as a dependency (though it wouldn't be employed, by configuration).

mrvisser commented 4 years ago

I'm building a Lambda function that has a dependency that references aws-sdk. The lambda environment provides aws-sdk for you so I don't want to bundle it (it adds about 6MB to the package size), but I can't exclude it.

I'm using workspaces so I have:

/package.json
  - dependencies:
    - aws-sdk
/lambda-workspace-a/package.json
  - dependencies:
    - external-module-depends-on-aws-sdk
/lambda-workspace-b/package.json

When I build my lambda environment, only my /lambda-workspace-a/package.json and lambda-workspace-b/package.json files are installed in order to package up the lambdas individually. This is great because my "provided" dependencies are in /package.json and my "runtime" dependencies are in the workspaces themselves.

But since external-module-depends-on-aws-sdk is also a runtime dependency and includes aws-sdk, it's forcing me to to include aws-sdk in my lambda package.

thorpep138 commented 3 years ago

Also requesting this feature; it greatly helps when optional dependencies have vulnerabilities that one would like to ignore 😄

Airkro commented 3 years ago

I have some dependencies only use in CI, some dependencies not use in CI.

I think ignore can make the installation faster, and save disk space.

I tried optionalDependencies but is not good enough.

neollog commented 2 years ago

I have some dependencies that need to be installed ONLY in a build image within a multi stage docker container. Now how can I do this since devDependencies is also used???

nettybun commented 1 year ago

You can do this yourself without any deps by creating a ""package"" anywhere in your project and then pointing yarn to it as a resolution/override such as "underscore": "link:./.yarn/empty-package". This has the added benefit of being entirely customizable for each import usecase.

Here's my empty-package source - file tree on the right, and contents vertically on the left:

image

This is how it's used in my actual project:

image

I could have put it in /packages/ but I like that to stay as "my real code".

Tested in Berry.

KartikSoneji commented 1 year ago

Following up on @heyheyhello 's idea, there is an empty-npm-package on npm, that also works:

{
    "resolutions": {
        "package-to-replace": "npm:empty-npm-package"
    }
}
ArshanKhanifar commented 1 year ago

I have some private bundledDependencies that yarn attempts to install from npm package registry even though I've added the resolutions field :(

  "bundledDependencies": [
    "@decent.xyz/box-common",
    "@decent.xyz/box-hooks",
    "@decent.xyz/box-ui",
    "@decent.xyz/box-evm"
  ],
  "resolutions": {
    "@decent.xyz/box-common": "npm:empty-npm-package",
    "@decent.xyz/box-hooks": "npm:empty-npm-package",
    "@decent.xyz/box-ui": "npm:empty-npm-package",
    "@decent.xyz/box-evm": "npm:empty-npm-package"
  },
  "optionalDependencies": {
    "@decent.xyz/box-common": "workspace:*",
    "@decent.xyz/box-hooks": "workspace:*",
    "@decent.xyz/box-ui": "workspace:*",
    "@decent.xyz/box-evm": "workspace:*"
  },

This works perfectly with npm and pnpm but not with yarn.

ArshanKhanifar commented 1 year ago

I fixed this by publishing empty packages to npm with the same name as my private packages. For some reason yarn wants to ensure those packages exist on the registry before it tries to install/do anything.

garmoshka-mo commented 9 months ago

there are plenty of poorly managed libraries in npm, yarn don't react to this yet, vote for this issue to bring yarn's attention to the problem and make a nice solution, e.g. "exclude-libraries": ["lib1", "lib2"] since solution "package-to-exlude": "npm:empty-npm-package" is a bit of a trick. But many thangs to KartikSoneji for inventing it! ⭐️⭐️⭐️⭐️