Strategy for using with a Lerna Monorepo

Undistraction commented 2 years ago

I'm struggling to find the correct approach to using yalc with a Lerna monorepo and projects that consume it.

Monorepo structure

packages
  - alpha
  - bravo 
  - charlie

My first approach was to run yalc publish from each package within the monorepo, then yalc add for each of those packages within the repo that needed to consume the packages from the monorepo.

While this appears to work, and sets all the monorepo packages to the correct file path, it doesn't resolve interdependencies within the monorepo packages. If alpha depends on bravo, for example, its dependency is still pointed at the versioned package, not at the version inside the yalc repo.

My next approach was to try using yarn add for each package in the monorepo as well, adding the generated .yalc to their package.json's files list, however this results in yalc recursively uploading the .yalc directory from each package to the repo, so I'm assuming this approach is not recommended.

What is the correct way to publish packages within a monorepo so that interdependent packages use the yalc repo?

[Edit] I guess one solution is to use yarn's resoutions in the consuming repo to force all the packages from the monorepo to use the versions in the .yalc directory.

wclr commented 2 years ago

Yalc is just a way to quickly share a package across projects in one's dev environment, it just copies content to .yalc folder, and may add ref in package.json, that is all what it is supposed to do, don't try to involve it into more complicated scenarios.

mathiasgheno commented 2 years ago

@Undistraction I'm using yalc to sync one monorepo and one front-end app with another packages of another monorepo. I'll create a text and share my architecture with you as soon as I finish it. I'm using lerna for both monorepos.

Undistraction commented 2 years ago

@mathiasgheno thank you. It would be really helpful to see your approach.

We publish all our packages out from the monorepo out at the same version, so the approach I've take is:

push all packages from within monorepo
add all packages within consuming frontend (If I stopped here there would be problems as the package versions for interdependent packages in the monorepo are not pointing to the any .yalc sir)
grab all the monorepo packages from the consuming frontend's dependencies and add them to it's resolutions (we use yarn) with the format "**/@example/name": "file: path/to/.yalc" which forces the package version in all our modules no matter how deeply nested.
I then have a script that removes yalc and removes the resolutions for when I want to publish and consume packages.

This seems like a good approach as it uses yalc minimally and is quick and easy to enable and disable.

mathiasgheno commented 2 years ago

@Undistraction

(English is not my primary language, let my know if I did not explain right)

In my project we have one repository that is agnostic of bussiness where we put all the code that can be shared accross projects. I'll call it Shared. Shared is a monorepo and we're using it with fixed version ─ so, when I update one package version I need to update all my packages.

I have another repository that is using the same strategy, I'll call it ProjectA. This project follows one bussiness domain, so all my packages that are related with that is defined inside that monorepo.

Still, I have one projects that are not monorepos that are using the Shared packages. I'll call it ProjectB. This project call be an UI or a server project.

Overview of Relations

graph LR;
    ProjectA --> Shared;
    ProjectB --> Shared;

Overview Modules of `Shared`

graph LR;
    Shared --> u["@shared/utils"];
    Shared --> c["@shared/components"];

Overview Modules of `ProjectA`

graph LR;
    ProjectA --> ui["@project-a/ui"];
    ProjectA --> s1["@project-a/server-1"];
    ProjectA --> s2["@project-a/server-2"];
    ProjectA --> tp["@project-a/types"];

My project Shared does not have any internal dependency inside the same repo ─ they're used by ProjectA and ProjectB. I'm using lerna to generate my package versions and to make easy to define individual packages. [1]

My project ProjectA does have a lot of internal dependency of internal packages. This project is using fixed version, so Lerna is what I'm using to controll all the relations inside that monorepo.

Where does I use `Yalc`?

I'm using Yalc to link my Shared packages with ProjectA and ProjectB.

ProjectB is the easy one. I'm just running yalc add * (* is the name of the package) and yalc remove --all and yalc makes everything works. Inside the Shared I do an yalc publish for each package I want to have the latest development version.

ProjectA is similar, but I need to be more carefull, because would be not ok to run those commands at each package. The packages that are related with the domain are linked by lerna ─ so I don't need yalc. The Shared packages are linked by yalc. So, in the root of my monorepo I have exclusive NPM script to run yalc by demmand. See one example below.

lerna exec --scope=@project-a/ui npm run yalc:add

The script yalc:add is defined inside the package and that script is responsable of making the relationship between what yalc add * needs to do [2]. You can see one example of that below.

yalc add @shared/components @shared/components

Still for my ProjectA I have another script in the root that does the oposite: `yalc:remove.

How does I update my `Shared` version inside projects?

When I finish my work locally I publish all my Shared package at one private NPM registry. So, inside my root project I run npm install package@latest and npm will figure out the new version and update my package.json for me. In the lerna world the idea is the same but I use lerna exec --scope @project-a/ui -- npm install @shared/component@latest. I have one script to help my do that in all my packages (Sorry it is in pt-br. I dont have time to translate now, let me know if you would like to have a english version).

Considerations

I did have a lot of problems related with React and Lerna. If you project is new I would like to share with you PNPM. PNPM seems to make the process of using React and monorepoes easier. The reason is that PNPM will flat all the React dependencies in the root of the monorepo ─ solving the problem "Invalid Hooks Calls" that borders me a lot. I hope I helped somehow.

[1] - The only exception is one ESLint packages that is used everywhere. I do not recommend you to use this strategy. Make Eslint a monorepo config, not one package.

[2] - I don't have so many shared inside one package. So my yalc add * will add all my Shared for simplicity.

Undistraction commented 2 years ago

@mathiasgheno thanks for taking the time to write that up. This is the solution we ended up with:

Our setup:

one monorepo containing ~50 packages
four frontend repos that consume packages from the monorepo
packages in the monorepo have complex inter-dependencies
We publish all packages at the same version on each publish
We use yarn (and have access to its resolutions feature)

solution:

In the monorepo we run a script that effectively:

Runs the following for every package: yarn yalc push --store-folder ~/.yalc --pure --content
watches all our package dirs for changes and when it detects one runs the script above

In our frontends we run a script that:

Runs through all our namespaced packages in the package.json and runs yalc add --link for each set (dependencies and devDependencies)
Adds yarn resolutions for all these dependencies to the package.json. This ensures that any interdependencies between packages also use packages form the .yalc directory.

At this point, I can make a change in the monorepo and it is propagated to the ~/.yalc dir and from there to the frontend, where it is picked up by fast-refresh and updated in the client.

I'm currently using this for dev flow, including deploying changes to our alpha-servers. When deploying to staging (or production), we also have a script in the frontend that removes yalc, and I publish out the packages to npm from the monorepo as before.

There are a few rough edges I'm ironing out. Definitely a big time-saver though.

pinkynrg commented 1 year ago

@Undistraction could you share the relevant parts of your setup (package.json / etc...)?

fabricioAburto commented 1 year ago

@wclr When u say: "don't try to involve it into more complicated scenarios" Imagine the scientists or computer engineers with that mentality that you express. We were still writing in stone. Where is the software compossibility principle?

I have a screw that is for concrete, so does it mean that it is not suitable for wood? Should I not use it? What do you think?

Are you from Iran by any chance? xD I have some collegues with that mentality there.

fabricioAburto commented 1 year ago

@Undistraction Thanks man for sharing your solution. At the end concrete screws works for wood too. :)

wclr / yalc