Multiple library in one repo

jogelin commented 7 years ago

Hello,

I am following the issue #6510 on angular cli and, one of the last reaction of @shlomiassaf is a list of key features that a library template should provide.

One point that library template should take in account is the "monorepo" case. Is there a way to manage multiple library with angular-librarian ?

gonzofish commented 7 years ago

There's not currently, but that's something I'd be interested in adding (I responded to the comment you're referencing on #6510). I've never worked in a monorepo, so I'm not sure of all the considerations that need to be made.

Are there any special concerns you have for such a feature?

jogelin commented 7 years ago

I saw your comment and sorry but I forgot to thank you for you work. I think it helps a lot of developers !

My main concern is when you have to manage a lot of libraries, you cannot maintain the same number of repositories and you don't want to duplicate same configurations.

I am working in a big company with many applications and we are creating a lot of libraries (+50) which are shared accross all apps. We try to keep the size of the apps as minimal as we can and all duplicated code are packaged in libraries.

I am comming from Java backend development and even if there are a lot of bad points, I think the way that we use to create libraries/microservices and share them is very easy. You can create a new module (with maven or gradle or other), you put your code into, you add a dependency to it and that its ! You can inherit configurations, share libraries, share packaging configurations, etc..

I would love to be able to do the same in JS/Angular development, just moving an Angular module into a folder, add a package.json and that it. But I have to know a lot of tools (webpack, karma, etc...) and even knowing those, it's not enough. I need to creates scripts, system links during development, etc...

But I still like to work in Angular, I just hope we could get the benefits of the backend development ;)

gonzofish commented 7 years ago

Thanks 👍 Your gratitude means a lot (and keeps me motivated 😆)

I think that your request ties in with @tobi-or-not-tobi request in #42--I know that generating a module is not the only part of this problem, but I think being able to have sub-modules could be a good step 1.

After that, I think being able to publish individual sub-modules would be a good step. I'll do some reading to get familiar with how to best set up a monorepo and work towards this feature.

Right now I'm working on a solid upgrade strategy (#38) and then I think this issue & #42 will be my next focus.

jogelin commented 7 years ago

Yes, publishing individual sub-modules would be a very good step !

You can find some links about monorepo here: build-angular-monorepo-and-app.

It's a repo that I started long time ago to try to solve this issue but I have no time and to be honest, no enough knowledge to maintain this repo...my bad :(

Again, thanks for your job and good luck !

shlomiassaf commented 7 years ago

Hi guys

Most of the hard work for supporting multiple libraries is around mapping.

Each library must map to its public import token, so other libraries and the demo app will access it by the public import and not the relative import.

This mapping is then applied for each processes. Webpack, TypeScript (paths) and optional others (e.g jest).

This has to be explicit for every library and not via a baseDir / moduleDir definition. Since we might have scope.

To key is having a good foundation to support this mapping, once you have it - going through the different process is easy

Sent from my iPhone

On Jun 21, 2017, at 4:55 PM, jogelin notifications@github.com wrote:

Yes, publishing individual sub-modules would be a very good step !

You can find some links about monorepo here: build-angular-monorepo-and-app.

It's a repo that I started long time ago to try to solve this issue but I have no time and to be honest, no enough knowledge to maintain this repo...my bad :(

Again, thanks for your job and good luck !

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jogelin commented 7 years ago

@shlomiassaf I completly agree !

Do you have any advices, ideas about how this mapping shall be done ?

I think it should be provided by the npm environment:

It could help to have a local repository by default (like the maven repo in Java) where we could publish our libraries before publishing on a remote repo.
Having some conventions in the packages.json to be able to inherit from a parent, share configurations, etc...

We shoudn't have to write scripts to update configuration files or use npm link, etc... The tools (webpack, typescript, ...), the framework (Angular, React, etc...) and the IDEs should just implement those conventions....

shlomiassaf commented 7 years ago

If you don't want to use scripts it means you want the users to write those mapping definitions and all other complex configurations.

This will not work, it's advanced and requires deep understanding of how every system works, it's just something that people will not do, they want it to work magically. Otherwise they need to dig deep into each tool, which is not realistic.

For example, with webpack you need to configure an alias, that's just the root folder of each lib.

For TypeScript you need to set the "paths" relative to "baseUrl" those must be to the entry file of the library.

For Jest you need to configure regexp, also to the entry.

Now, in all of the above the convention differs. For example, TypeScript wildcard are different then jest...

Now make it work for scoped vs non-scoped...

Try to figure how to align the output for.

Make sure it inlines resources...

You see where I'm getting...

I'm telling you it took me a long time to understand th eco system and I like to dig deep and understand. Most people just want it to work.

Sent from my iPhone

On Jun 22, 2017, at 11:35 AM, jogelin notifications@github.com wrote:

@shlomiassaf I completly agree !

Do you have any advices, ideas about how this mapping shall be done ?

I think it should be provided by the npm environment:

It could help to have a local repository by default (like the maven repo in Java) where we could publish our libraries before publishing on a remote repo. Having some conventions in the packages.json to be able to inherit from a parent, share configurations, etc... We shoudn't have to write scripts to update configuration files or use npm link, etc... The tools (webpack, typescript, ...), the framework (Angular, React, etc...) and the IDEs should just implement those conventions....

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jogelin commented 7 years ago

Again, I agree with you ;) I don't want to write mapping definitions, I don't want know all tools deeply, I just want it to work magically :p

You said that each tool work differently...this is my main concern...why each tools create a new way ? Why they cannot agree on a common way ? I think it's because there is no common way..., rules..., a standard defined by an "official organism"...

Because of that, you have to write scripts, conventions, you need to dig deep understand and thanks for that :))

shlomiassaf commented 7 years ago

The reason each tool works differently is simple, they are built by different teams and for different purposes.

Webpack is a bundler, TypeScript is a compiler so they might have different requirements.

It's almost impossible to regulate this thing, to many opinions, ego and work.

Sent from my iPhone

On Jun 22, 2017, at 12:58 PM, jogelin notifications@github.com wrote:

Again, I agree with you ;) I don't want to write mapping definitions, I don't want know all tools deeply, I just want it to work magically :p

You said that each tool work differently...this is my main concern...why each tools create a new way ? Why they cannot agree on a common way ? I think it's because there is no common way..., rules..., a standard defined by an "official organism"...

Because of that, you have to write scripts, conventions, you need to dig deep understand and thanks for that :))

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jogelin commented 7 years ago

To regulate opinions, ego and work, I agree, it's complicate but not impossible.

Sorry to come back that again but:

	Java	Javascript
Bundlers	Maven, Gradle, ... They produce same target folder which is packaged in the same jar extension You can depends on another library without taking in account the Bundler	Gulp, Webpack, systemJs, rollup ... They produce many different output folder and each one has it own way to bundle : es5/esm, umd, fesm, ... You have to bundle in all formats to be compatible with the bundler of the user
Compilers	Javac, Scala, Groovy, Kotlin,... They compile different languages to bytecode which is interpreted by the same JVM	Typescript, Babel, jsx ?, ... They compile different languages to es5/es6 which is interpreted by many differents browsers and nodeJS
Frameworks	Native Java, Spring, EJB, ... They specify different ways of working	Native JS, React, Angular, ... They specify different ways of working

Why in java, many different teams could agree on the same formats/structures ? Why not in JS ?
Why in java, when you depend on a library, you don't care about bundler that the library use ? Why in JS you need to publish many different bundle ?
Why in java, when you depend on a library, you don't care about the framework that the library use ?
Why in java, you don't have to think about modules relations ?

The main reason is that the community agreed on some conventions, structure and when a new tools is on the market, it respect those conventions.

In JS everybody try to create its own guidelines, code style, bundler, etc....which is good because it's encouraging to produce interesting stuff more performant, etc... But the good ideas should be copied and centralized, standartized somewhere.

I don't know how and why and I am not saying they are no "wars" in the JVM community but it works. Because of your incredible job with those kind of libraries and with the job of big team like Angular, I am sure (I hope ;)), we are going on a common way to make it work

gonzofish commented 7 years ago

Unfortunately, this isn't really a good spot to be having the "why can't JS have a common way of doing things?" conversation. It's both a blessing and a curse of the JS world--the better place to figure it out would be with TC-39

Hope I'm not coming off like a jerk. The topic is just outside the scope of the project

gonzofish commented 7 years ago

To get this working appropriately, the generated project structure would have to change significantly.

Currently, all the files live under src:

|_examples/
|_  src/
  |_some-component/
    |_some-component.component.html
    |_some-component.component.scss
    |_some-component.component.spec.ts
    |_some-component.component.ts
  |_services/
    |_first.service.spec.ts
    |_first.service.ts
    |_second.service.spec.ts
    |_second.service.ts
  |_my-library.module.ts
  |_index.ts
  |_test.js
  |_vendor.ts
|_tasks/
|_webpack/
|_index.ts
// ... other files

But with a monorepo approach, we need something more like:

|_examples/
|_packages/
  |_my-library/
    |_src/
      |_some-component/
      |_services/
      |_my-library.module.ts
      |_index.ts
    |_index.ts
  |_other-library/
    |_src/
      |_other-component/
      |_services/
      |_my-library.module.ts
      |_index.ts
  |_third-library/
    |_src/
      |_third-component/
      |_services/
      |_my-library.module.ts
      |_index.ts
|_tasks/
|_webpack/
// ...other files

It looks relatively straight forward...the migration would be minor (taking the existing src and moving it into packages/{{ name }}).

Some questions still:

Would this structure work for someone who doesn't want a monorepo?
- If not, how do we support both structures effectively?
What are the other common files that should go in the root?
- Is test.js a common file?
- Does each package need its own examples?
- Does there need to be a single unifying examples at the root?

jogelin commented 7 years ago

I don't think it is necessary to rename the src folder. Source is still source code.

After testing multiple library seeds and monorepo client like lerna, ora, yerna, etc..., I prefer the file structure of angular-library-starter. It is supporting one or multiple library, with or without scope. It supports also a demo application which is very useful in a library monorepo.

Some ideas/features about monorepo:

Inherit from the parent package.json: during the build, be able to inject in the package.json some properties like author, repo, license, ...
Be able to manage local version with placeholders like angular team is doing. If a library depends on another local one, the version is injected in the package.json
Give the possibility to inherit other configurations like tests,
Don't force people to follow only one release process by tagging automatically, etc...give the possibility to do it but don't force it.
Some scripts/command to synchronize libraries configurations like doing npm install in each library or synchronize the tsconfig or rollup file file, etc... usually, it is a bootstrap command
Some scripts/command to be able to execute a command in each library
The hooks of angular-library-starter is also very useful to interact with configurations and to avoid file multiplication !

gonzofish commented 7 years ago

Progress is definitely being made in the monorepo branch...

I'm taking cues from both @shlomiassaf's project and @filipesilva's angular-quickstart-lib for inspiration/copying

gonzofish commented 7 years ago

I'm going to push this off for now

While I've made progress I'm adverse to publishing a v1.0.0 that diverges so much from how people who've been using it have had their projects structured.

In the future, when I support this, projects can be setup with a --monorepo or --m flag. Additionally, I'd like to allow projects to toggle to a monorepo setup if they'd like to after initial setup.

gonzofish commented 7 years ago

Another heads up...#59 is a small step toward achieving monorepos...

The code is done for #59, just need to do some more testing--but it at least adds scoped package support

gonzofish / angular-librarian

Multiple library in one repo #43