Open tschaffter opened 11 months ago
Type of EC2 instance used: t3.2xlarge
(see docs)
Starting a fresh dev container + removing node_modules
:
vscode@108c376e44d7:/workspaces/sage-monorepo$ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 80G 38G 43G 48% /
tmpfs 64M 0 64M 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/nvme0n1p1 80G 38G 43G 48% /vscode
none 16G 160K 16G 1% /tmp
Preparing all the environments. This operation needs to be done only once!
$ time workspace-install
...
➤ YN0000: Done with warnings in 1m 7s
✔ nx run openchallenges-keycloak:prepare (390ms)
✔ nx run openchallenges-service-registry:prepare (407ms)
✔ nx run openchallenges-api-gateway:prepare (433ms)
✔ nx run openchallenges-mariadb:prepare (385ms)
✔ nx run openchallenges-challenge-to-elasticsearch-service:prepare (355ms)
✔ nx run schematic-api:prepare (397ms)
✔ nx run openchallenges-kaggle-to-kafka-service:prepare (309ms)
✔ nx run openchallenges-organization-service:prepare (340ms)
✔ nx run openchallenges-challenge-service:prepare (374ms)
✔ nx run openchallenges-mysqld-exporter:prepare (366ms)
✔ nx run openchallenges-image-service:prepare (374ms)
✔ nx run openchallenges-config-server:prepare (373ms)
✔ nx run openchallenges-elasticsearch:prepare (382ms)
✔ nx run openchallenges-auth-service:prepare (427ms)
✔ nx run openchallenges-core-service:prepare (390ms)
✔ nx run openchallenges-user-service:prepare (403ms)
✔ nx run openchallenges-api-docs:prepare (409ms)
✔ nx run openchallenges-prometheus:prepare (432ms)
✔ nx run openchallenges-postgres:prepare (386ms)
✔ nx run openchallenges-notebook:prepare (438ms)
✔ nx run openchallenges-grafana:prepare (464ms)
✔ nx run openchallenges-rstudio:prepare (434ms)
✔ nx run openchallenges-thumbor:prepare (364ms)
✔ nx run openchallenges-zipkin:prepare (399ms)
✔ nx run openchallenges-mongo:prepare (362ms)
✔ nx run openchallenges-vault:prepare (358ms)
✔ nx run openchallenges-minio:prepare (369ms)
✔ nx run openchallenges-kafka:prepare (391ms)
✔ nx run openchallenges-apex:prepare (372ms)
✔ nx run schematic-api-docs:prepare (416ms)
✔ nx run openchallenges-app:prepare (425ms)
✔ nx run schematic-notebook:prepare (444ms)
✔ nx run synapse-rstudio:prepare (455ms)
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target prepare for 33 projects (5s)
View logs and investigate cache misses at https://cloud.nx.app/runs/MSTcuhdHQX
✔ nx run openchallenges-app-config-data:prepare-java (4s)
✔ nx run shared-java-util:prepare-java (820ms)
✔ nx run openchallenges-kafka-admin:prepare-java (954ms)
✔ nx run openchallenges-kafka-model:prepare-java (1s)
✔ nx run openchallenges-service-registry:prepare-java (1s)
✔ nx run openchallenges-api-gateway:prepare-java (1s)
✔ nx run openchallenges-kafka-consumer:prepare-java (920ms)
✔ nx run openchallenges-kafka-producer:prepare-java (1s)
✔ nx run openchallenges-challenge-to-elasticsearch-service:prepare-java (1s)
✔ nx run openchallenges-kaggle-to-kafka-service:prepare-java (934ms)
✔ nx run openchallenges-organization-service:prepare-java (978ms)
✔ nx run openchallenges-challenge-service:prepare-java (941ms)
✔ nx run openchallenges-config-server:prepare-java (1s)
✔ nx run openchallenges-image-service:prepare-java (997ms)
✔ nx run openchallenges-auth-service:prepare-java (796ms)
✔ nx run openchallenges-user-service:prepare-java (863ms)
✔ nx run openchallenges-core-service:prepare-java (850ms)
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target prepare-java for 17 projects (20s)
View logs and investigate cache misses at https://cloud.nx.app/runs/5j7bZtTDjC
✔ nx run schematic-api-client-python:prepare-python (6s)
✔ nx run openchallenges-notebook:prepare-python (11s)
✔ nx run schematic-notebook:prepare-python (10s)
✔ nx run schematic-api:prepare-python (2m)
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target prepare-python for 4 projects (2m)
View logs and investigate cache misses at https://cloud.nx.app/runs/IH7LFwfyVx
✔ nx run openchallenges-api-client-r:prepare-r (3m)
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target prepare-r for project openchallenges-api-client-r (3m)
View logs and investigate cache misses at https://cloud.nx.app/runs/9uaJqVf5pk
real 6m18.141s
user 10m19.576s
sys 2m38.068s
Running the same command again. Nothing has changed but it still takes some time to the project to verify that they don't need to install anything. The time varies depending on the programming language used.
➤ YN0000: └ Completed in 0s 976ms
➤ YN0000: ┌ Fetch step
➤ YN0000: └ Completed in 1s 810ms
➤ YN0000: ┌ Link step
➤ YN0000: └ Completed in 1s 495ms
➤ YN0000: Done with warnings in 5s 181ms
✔ nx run openchallenges-keycloak:prepare (353ms)
✔ nx run openchallenges-api-gateway:prepare (383ms)
✔ nx run openchallenges-service-registry:prepare (457ms)
✔ nx run openchallenges-mariadb:prepare (343ms)
✔ nx run openchallenges-challenge-to-elasticsearch-service:prepare (348ms)
✔ nx run schematic-api:prepare (435ms)
✔ nx run openchallenges-kaggle-to-kafka-service:prepare (289ms)
✔ nx run openchallenges-challenge-service:prepare (316ms)
✔ nx run openchallenges-organization-service:prepare (433ms)
✔ nx run openchallenges-mysqld-exporter:prepare (448ms)
✔ nx run openchallenges-elasticsearch:prepare (419ms)
✔ nx run openchallenges-image-service:prepare (373ms)
✔ nx run openchallenges-config-server:prepare (342ms)
✔ nx run openchallenges-core-service:prepare (305ms)
✔ nx run openchallenges-user-service:prepare (329ms)
✔ nx run openchallenges-auth-service:prepare (323ms)
✔ nx run openchallenges-notebook:prepare (318ms)
✔ nx run openchallenges-prometheus:prepare (412ms)
✔ nx run openchallenges-api-docs:prepare (364ms)
✔ nx run openchallenges-postgres:prepare (392ms)
✔ nx run openchallenges-thumbor:prepare (395ms)
✔ nx run openchallenges-grafana:prepare (371ms)
✔ nx run openchallenges-zipkin:prepare (345ms)
✔ nx run openchallenges-rstudio:prepare (391ms)
✔ nx run openchallenges-minio:prepare (347ms)
✔ nx run openchallenges-vault:prepare (361ms)
✔ nx run openchallenges-mongo:prepare (396ms)
✔ nx run openchallenges-kafka:prepare (340ms)
✔ nx run openchallenges-apex:prepare (409ms)
✔ nx run openchallenges-app:prepare (358ms)
✔ nx run schematic-api-docs:prepare (383ms)
✔ nx run synapse-rstudio:prepare (328ms)
✔ nx run schematic-notebook:prepare (364ms)
———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target prepare for 33 projects (4s)
View logs and investigate cache misses at https://cloud.nx.app/runs/aTFiBqQoRW
✔ nx run openchallenges-app-config-data:prepare-java (792ms)
✔ nx run shared-java-util:prepare-java (939ms)
✔ nx run openchallenges-kafka-model:prepare-java (854ms)
✔ nx run openchallenges-kafka-admin:prepare-java (831ms)
✔ nx run openchallenges-service-registry:prepare-java (912ms)
✔ nx run openchallenges-api-gateway:prepare-java (815ms)
✔ nx run openchallenges-kafka-consumer:prepare-java (934ms)
✔ nx run openchallenges-kafka-producer:prepare-java (877ms)
✔ nx run openchallenges-challenge-to-elasticsearch-service:prepare-java (740ms)
✔ nx run openchallenges-kaggle-to-kafka-service:prepare-java (960ms)
✔ nx run openchallenges-organization-service:prepare-java (1s)
✔ nx run openchallenges-challenge-service:prepare-java (1s)
✔ nx run openchallenges-image-service:prepare-java (1s)
✔ nx run openchallenges-config-server:prepare-java (1s)
✔ nx run openchallenges-auth-service:prepare-java (890ms)
✔ nx run openchallenges-core-service:prepare-java (941ms)
✔ nx run openchallenges-user-service:prepare-java (847ms)
———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target prepare-java for 17 projects (16s)
View logs and investigate cache misses at https://cloud.nx.app/runs/Ul872VD3pc
✔ nx run schematic-api-client-python:prepare-python (5s)
✔ nx run openchallenges-notebook:prepare-python (9s)
✔ nx run schematic-api:prepare-python (11s)
✔ nx run schematic-notebook:prepare-python (9s)
———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target prepare-python for 4 projects (14s)
View logs and investigate cache misses at https://cloud.nx.app/runs/3A3qdoyABt
✔ nx run openchallenges-api-client-r:prepare-r (3s)
———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target prepare-r for project openchallenges-api-client-r (3s)
View logs and investigate cache misses at https://cloud.nx.app/runs/292emRi0Gm
real 0m46.165s
user 1m15.760s
sys 0m9.433s
vscode@108c376e44d7:/workspaces/sage-monorepo$ df -h
Filesystem Size Used Avail Use% Mounted on
overlay 80G 42G 39G 52% /
tmpfs 64M 0 64M 0% /dev
tmpfs 16G 0 16G 0% /sys/fs/cgroup
shm 64M 0 64M 0% /dev/shm
/dev/nvme0n1p1 80G 42G 39G 52% /vscode
none 16G 2.2M 16G 1% /tmp
TODO: Check how much time installing Node.js dependencies take
Here are the projects that take time to install:
openchallenges-api-client-r
takes 3 minutesschematic-api
takes 2 minutesThe total size of the dependencies for the existing projects is 4 GB.
Preparing 17 Java projects takes 20 seconds, which is really fast. Note that Java projects must be prepared sequentially because of a concurrency issue.
$ nx run-many --target=build
✔ nx run schematic-api-description:build (2s)
✔ nx run openchallenges-api-description:build-individuals (3s)
✔ nx run openchallenges-api-description:build (990ms)
✔ nx run openchallenges-api-client-angular-deprecated:build:production (6s)
✔ nx run openchallenges-api-client-angular:build:production (6s)
✔ nx run openchallenges-app-config-data:build (35s)
✔ nx run openchallenges-app-config-data:install (5s)
✔ nx run shared-java-util:build (40s)
✔ nx run openchallenges-kafka-model:build (39s)
✔ nx run shared-java-util:install (4s)
✔ nx run openchallenges-kafka-model:install (5s)
✔ nx run schematic-api-client-angular:build:production (5s)
✔ nx run openchallenges-kafka-admin:build (14s)
✔ nx run openchallenges-kafka-admin:install (15s)
✔ nx run openchallenges-service-registry:build (26s)
✔ nx run openchallenges-api-gateway:build (26s)
✔ nx run synapse-api-description:build (2s)
✔ nx run openchallenges-kafka-consumer:build (12s)
✔ nx run openchallenges-db-cli:build (3s)
✔ nx run openchallenges-kafka-producer:build (6s)
✔ nx run openchallenges-kafka-consumer:install (6s)
✔ nx run schematic-api-docs:build (2s)
✔ nx run openchallenges-kafka-producer:install (3s)
✔ nx run openchallenges-api-docs:build (2s)
✔ nx run openchallenges-infra:build (10s)
✔ nx run openchallenges-config-server:build (5s)
✔ nx run openchallenges-app:build:production (1m)
✔ nx run openchallenges-organization-service:build (1m)
✔ nx run openchallenges-challenge-service:build (34s)
✔ nx run openchallenges-api-client-r:build (2m)
✔ nx run openchallenges-auth-service:build (7s)
✔ nx run openchallenges-user-service:build (11s)
✔ nx run openchallenges-image-service:build (23s)
✔ nx run openchallenges-challenge-to-elasticsearch-service:build (16s)
✔ nx run openchallenges-core-service:build (33s)
✔ nx run openchallenges-kaggle-to-kafka-service:build (5s)
✔ nx run schematic-app:build:production (31s)
———————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target build for 30 projects and 7 tasks they depend on (4m)
What projects is this feature for?
No response
Description
Currently the tools that maintain the Node.js/Python/Java/R environments are applied to all the projects. However, doing so may be a waste of time and resources (mainly storage space) for projects that a developer is not contributing too.
For example, developers who work on Schematic shouldn't ideally have to prepare the OC projects. On the other hand,
schematic-api
comes with many dependencies that take about two minutes to install (see logs below).There are different strategies we may explore to save resources:
Approach 1
The first method is to rely on Nx to run the preparation tasks for the
affected
projects. One edge case is that when cloning the repo, there are no affected projects and so nothing could be built without preparing some project first.A potential solution may be to initialize the workspace by preparing all the projects (potentially in combination with solution 2), and from then only prepare again the projects that are affected.
Another solution is to make "build", "test", "lint", etc. tasks depends on the preparation tasks. This approach would not be worth it if running the preparation task on project that have already been prepare takes time.
Approach 2
Here the idea would be to allow the developers to define the list of products they want to work with. E.g. "openchallenges", "schematic", "synapse", etc.
Approach 3
This approach consist in evaluating whether running the preparation task on a project that has already been prepared and has not changed is fast or if it takes time. If preparing the environment again is super fast, then developers will not waste time. However, storage space would still be wasted.
Notes
package.json
and some configuration files. When that happens, all the project will have their environments prepared if we rely on the list of affected projects.cc: @andrewelamb @linglp @milen-sage
Anything else?
No response
Code of Conduct