[Feature] Explore prepare all projects Vs prepare affected project

[tschaffter]

What projects is this feature for?

No response

Description

The strategy adopted by Sage Monorepo is to prepare all the projects with workspace-install, then keep the dependencies up to date with git hooks.

Pros:

• Assumes that the dependencies are ready when running task like lint, build, etc. so no need to check dependencies when running these tasks (faster).

Cons:

• We are making the assumption that the dependencies are ready, which is strongly promoted but can't be guaranteed.
• The solution does not scale as disk space and initialization time increase with the number of projects.

Alternative

An alternative is to always check that the dependencies are up to date before running tasks that require them. Initializing the workspace would require less time and disk space.

Tasks

• For each type of project defined by their programming language and package manager, evaluate the time it takes to CHECK that the dependencies are up to date.
• Decide whether it is acceptable to adopt the alternative strategy

Anything else?

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[tschaffter]

TIme needed to check Python dependencies

Schematic API:

$ hyperfine --warmup 1 --prepare 'nx prepare-python schematic-api' --runs 10 'nx prepare-python schematic-api'

Benchmark 1: nx prepare-python schematic-api
  Time (mean ± σ):     13.447 s ±  0.728 s    [User: 11.333 s, System: 0.907 s]
  Range (min … max):   11.974 s … 14.580 s    10 runs

$ nx prepare-python schematic-api

> nx run schematic-api:prepare-python

Using virtualenv: /workspaces/sage-monorepo/apps/schematic/api/.venv
Requirement already satisfied: cython<3.0.0 in ./.venv/lib/python3.10/site-packages (0.29.36)
Collecting pyyaml==5.4.1
  Using cached PyYAML-5.4.1-cp310-cp310-linux_x86_64.whl
Installing collected packages: pyyaml
  Attempting uninstall: pyyaml
    Found existing installation: PyYAML 6.0.1
    Uninstalling PyYAML-6.0.1:
      Successfully uninstalled PyYAML-6.0.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
schematic-db 0.0.29 requires PyYAML<7.0,>=6.0, but you have pyyaml 5.4.1 which is incompatible.
schematicpy 23.7.1 requires PyYAML<7.0.0,>=6.0.0, but you have pyyaml 5.4.1 which is incompatible.
Successfully installed pyyaml-5.4.1
Installing dependencies from lock file
Package operations: 0 installs, 1 update, 0 removals
  • Updating pyyaml (5.4.1 -> 6.0.1)
Installing the current project: schematic-api (0.1.0)

 ——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target prepare-python for project schematic-api (12s)

When commenting out the patch (TODO add more details):

# poetry run pip install "cython<3.0.0"
# poetry run pip install --no-build-isolation pyyaml==5.4.1

$ hyperfine --warmup 1 --prepare 'nx prepare-python schematic-api' --runs 10 'nx prepare-python schematic-api'
Benchmark 1: nx prepare-python schematic-api
  Time (mean ± σ):      4.851 s ±  0.360 s    [User: 4.129 s, System: 0.541 s]
  Range (min … max):    4.522 s …  5.639 s    10 runs

Time needed to check R dependencies

$ hyperfine --warmup 1 \
    --prepare 'nx prepare-r openchallenges-api-client-r' \
    --runs 10 'nx prepare-r openchallenges-api-client-r'

Benchmark 1: nx prepare-r openchallenges-api-client-r
  Time (mean ± σ):      3.985 s ±  0.298 s    [User: 3.476 s, System: 0.841 s]
  Range (min … max):    3.505 s …  4.599 s    10 runs

Time needed to check Java dependencies

$ hyperfine --warmup 1 \
    --prepare 'nx prepare-java openchallenges-challenge-service' \
    --runs 10 'nx prepare-java openchallenges-challenge-service'

Benchmark 1: nx prepare-java openchallenges-challenge-service
  Time (mean ± σ):      2.185 s ±  0.246 s    [User: 2.548 s, System: 0.400 s]
  Range (min … max):    1.828 s …  2.697 s    10 runs

Time needed to check npm dependencies

When using yarn:

$ hyperfine --warmup 1 --prepare 'yarn install --immutable' --runs 10 'yarn install --immutable'
Benchmark 1: yarn install --immutable
  Time (mean ± σ):      6.444 s ±  0.630 s    [User: 7.176 s, System: 0.912 s]
  Range (min … max):    5.770 s …  7.670 s    10 runs

When using pnpm:

$ hyperfine --warmup 1 --prepare 'pnpm install --frozen-lockfile' --runs 10 'pnpm install --frozen-lockfile'
Benchmark 1: pnpm install --frozen-lockfile
  Time (mean ± σ):      3.922 s ±  0.284 s    [User: 4.149 s, System: 0.813 s]
  Range (min … max):    3.582 s …  4.476 s    10 runs