[RRFC] Optional install

gotbahn commented 4 years ago

Motivation ("The Why")

Motivation ("The Why")

As a Continuous Integration maintainer, I'm highly interested in fast build jobs. Installing dependencies one of the key parts here.

Caching helps significantly speed up, but:

it's not stale, dependencies can be updated pretty often, so cache invalidation as well
not all dependencies are cachable

I highly appreciate that npm has --production flag feature, that allows us to avoid devDependcies installation, this technique significantly reduces install time for delivery jobs.

It would be so nice if such behavior could be more eloquent and we can define or exclude certain packages from being installed. Lemme give some example

Example

Let's say we running some test jobs in parallel, based on type:

static, uses eslint, stylelint, typescript, etc.
units, uses jest, redux-mock-store, etc.
end-to-end, that uses some dev-server, cypress, cypress plugin, etc.

{
  "devDependencies": {
    "cypress": "*",
    "eslint": "*",
    "express": "*",
    "jest": "*",
    "stylelint": "*",
    "redux-mock-store": "*",
    "typescript": "*",
  },
  "dependencies": {
    "react": "*",
    "redux": "*",
    "webpack": "*",
  }
}

How

Current Behaviour

Each of the jobs installs all dependencies. Hower "static" and "unit" don't need packages from end-to-end, but have to install it together with all "devDependencies".

Desired Behaviour

I have no strong vision of how API should look like in the end, but can give a couple of ideas:

Exclude packages

As an additional argument, that accepts list of package to be ignored upon install

npm i --exclude=express,cypress

Unique listing

Assemble own unique lists

{
  "staticDependencies": {
    "eslint": "*",
    "stylelint": "*",
    "typescript": "*",
  },
  "unitDependencies": {
    "jest": "*",
    "redux-mock-store": "*"
  },
  "e2eDependencies": {
    "cypress": "*",
    "express": "*",
  },
  "dependencies": {
    "react": "*",
    "redux": "*",
    "webpack": "*",
  }
}

Can follow the same rule as --production, and install packages from "dependencies" & unique list. For example

npm i --dependencies=static

or

npm i --staticDependencies

References

Sorry, I haven't found anything useful to ref

maapteh commented 3 years ago

Just think about having stuff into a mono-repo. I do the same for example our e2e setup is in its own package with its own deps and only runs on the stage it needs to be runned. So not installing any e2e related when not needed. The job depends on the build stage obviously but i guess its clear what a mono repo can do for you :)

Another way is making dependencies yourself for example all linters in one package. Then during the job you can install them while not being part of your package lib.

Having cypress for example as package is a total waste of energy, you can think about creating an image for it. Why download it, is has a huge post install phase and thats really a waste of energy :) Create the container and be done with it.

Hope it helps.

aladdin-add commented 3 years ago

same issue: we want to skip installing e2e related devDependencies (e.g. puppeteer) - it's very time-costing!

but I suggest something like optionalDevDependencies - it's just the same as optionalDependencies

related discussion: https://github.com/npm/npm/issues/3870

darcyclarke commented 3 years ago

@gotbahn apologies for the delay in bubbling this up - I've added the "Agenda" label to this issue & it should get discussed at our next Open RFC call on Wednesday.

As a side note, you might actually be able to get this type of behaivour by utilizing workspaces when we ship the next round of improvements which include filtered/scoped installs (ex. npm i --workspace=static / npm i -w=static or npm i -w=static -w=staticDependencies...)

gotbahn commented 3 years ago

As a side note, you might actually be able to get this type of behaivour by utilizing workspaces when we ship the next round of improvements which include filtered/scoped installs

Oh, nice! Scoped installs sounds exciting. Looking forward to try it out.

npm / rfcs