Convert to lerna monorepo

hulkish commented 7 years ago

I think there is a lot of value in this. If you agree, I'd like to submit a PR which accomplishes this.

For starters, this will allow sqwuak to be a.bit more of a useful tool potentially outside of webpack. I can think of several great usages of this for my own projects, for example.

TheLarkInn commented 7 years ago

@webpack-contrib/admin-team @alistairjcbrown What do you think about this.

I'm 100% on board. Opens up some awesome opportunities for collab too.

TheLarkInn commented 7 years ago

@hulkish Would you be able to outline a bit more in detail how you would break it up?

hulkish commented 7 years ago

@TheLarkInn Sure: here's a rough draft:

current structure:

├── webpack-canary/
│   ├── lib/
│   │   ├── generate-install-object/
│   │   │   └── test/
│   │   ├── get-dependency-examples/
│   │   │   └── test/
│   │   ├── install-webpack-and-dependency
│   │   │   └── test/
│   │   ├── logger/
│   │   ├── run-dependency-with-webpack/
│   │   │   └── test/
│   │   └── test/
│   ├── scripts/
│   ├── squawk/
│   ├── LICENSE
│   ├── README.md
│   ├── index.js
│   ├── package.json
│   └── squawk.js

new structure:

├── webpack-canary/
│   ├── packages/
│   │   ├── generate-install-object/
│   │   │   ├── lib/
│   │   │   ├── test/
│   │   │   └── package.json // package name: webpack-canary-generate-install-object
│   │   ├── get-dependency-examples/
│   │   │   ├── lib/
│   │   │   ├── test/
│   │   │   └── package.json // package name: webpack-canary-get-dependency-examples
│   │   ├── install-webpack-and-dependency
│   │   │   ├── lib/
│   │   │   ├── test/
│   │   │   └── package.json // package name: webpack-canary-install-webpack-and-dependency
│   │   ├── logger/
│   │   │   ├── lib/
│   │   │   ├── test/
│   │   │   └── package.json // package name: webpack-canary-logger
│   │   ├── regressions
│   │   │   ├── lib/
│   │   │   ├── test/
│   │   │   └── package.json // package name: webpack-canary-regressions
│   │   ├── run-dependency-with-webpack/
│   │   │   ├── lib/
│   │   │   ├── test/
│   │   │   └── package.json // package name: webpack-canary-run-dependency-with-webpack
│   │   └── scripts/
│   │   │   ├── lib/
│   │   │   ├── test/
│   │   │   └── package.json // package name: webpack-canary-scripts
│   │   └── squawk/
│   │   │   ├── bin/
│   │   │   │   └── squawk.js // ?
│   │   │   ├── lib/
│   │   │   │   └── squawk.js // ?
│   │   │   ├── test/
│   │   │   └── package.json // package name: webpack-canary-squawk
│   ├── LICENSE
│   ├── README.md
│   ├── index.js
│   ├── lerna.json
│   ├── package.json
│   └── yarn.lock

Note: Could also namespace each package name using @webpack-canary. For example: @webpack-canary/squawk

Note: All packages/* (including root of monorepo) will have "private": true in their package.json's, unless the particular package(s) are intended to be published.

`webpack-canary/lerna.json`:

version mode: fixed/locked

I think this makes the most sense, as opposed to the independent mode.

{
"lerna": "<lerna version>",
"version": "<version for each webpack-canary package>",
"npmClient": "yarn",
"packages": [
"packages/*"
]
}

`webpack-canary/package.json`:

  [..]
  "private": true,
  "ava": {
    "files": [
      "packages/*/src/**/*.test.js",
      "packages/*/test/**/*.test.js",
      "./test/**/*.test.js"
    ],
    "source": [
      "packages/*/src/**/*.js",
      "!packages/*/lib/**/*.js"
    ],
    "require": [
      "babel-register"
    ]
  },
  "scripts": {
    [..]
    "bootstrap": "lerna bootstrap",
    "test": "nyc ava"
  }

running scripts example

yarn
yarn bootstrap # recursively installs external dependencies for each package then links each webpack-canary package which depend on each other
yarn test

Why?

Easier to maintain & share much of the tooling created for webpack-canary.
I think it'd be great to expose much of this tooling to where users could perhaps implement this into their application stack's testing phase. Allowing them to test when their webpack config breaks!
Nobody is going to want to correctly repackage everything once it's already done the non-lerna way. 😆

Hope that paints a clearer picture.

michael-ciniawsky commented 7 years ago

I'm not familar with webpack-canary at all. Is it even in use somehwere yet 🙃 ? A monorepo seems to be a good solution since testing all that stuff is complex and the more each subject can localized (into a package) the better. Please excuse if the following thinking out loud makes little sense, but what I mean is

webpack-canary/bench (Benchmarks, for e.g Vue, React, Angular, ... (like TodoMVC but for webpack))
webpack-canary/unit (Unit Tests)
webpack-canary/e2e (Interagtion Tests, for e.g Vue, React, Angular, Svelte ...)
webpack-canary/[logs|doctor|inspector] (Logger/Inspector of various things webpack)
webpack-canary/ci (Run/manage different CI setups)
webpack-canary/versions (Run/manage webpack versions)

If possible each could run standalone and there is an intreface for framwork authors to setup a webpack-canary project to test his/her work against.

Difficult to do, anyone interested in trying to do this should a have the change to do so imho 😛 good luck... 😅 :D

TheLarkInn commented 7 years ago

@michael-ciniawsky This was a seed project that was created from a research spike by @alistairjcbrown. I believe he is now back to being busy with work but the project is in a really great place still.

@michael-ciniawsky Ideally the project would work like NodeJS's Canary in the Gold Mine. But with the architecture for the project that Alistair built, ideally framework authors, plugin / loader authors, and even us (webpack core) could leverage this library to do any of the things like what you've mentioned.

We'd also want to do smoke testing against bug prone libraries like node-libs-browsers etc.

TheLarkInn commented 7 years ago

@hulkish I think this is great looking what you have so far. Only one note:

Note: Could also namespace each package name using @webpack-canary. For example: @webpack-canary/squawk

We haven't 100% decided on using scoped packages yet and if we do we'll want to make it as an org wide decision. So for now, feel free to leave that piece out.

joshwiens commented 7 years ago

@hulkish - Perhaps highlighting the benefits you eluded to in the original post would be beneficial, particularly to me who currently isn't seeing them.

From an org perspective ...

Giving canary it's own scope, under an org which will eventually be scoped is an anti-pattern as npm scopes are generally organization level. At the same time, if you are going to go the monorepo route, it should be scoped to a common parent.
The point of contrib was to standardize, if canary will somehow benefit from this restructuring in a way that i'm just not seeing more power to everyone but at that point it should be moved out of contrib.
Nothing currently under contrib wouldn benefit from a monorepo & given our charter really never will. Loaders & plugins are intended to do a single job, the nested package thing just isn't applicable.

Personally ...

I honestly don't see the point here outside of monorepos are the new in thing. Conceptually, they simplify maintenance of multiple large projects who share a common parent. You can scope & install portions as needed, which is great in certain situations but currently is not applicable to webpack-canary.

It simply isn't complicated / cumbersome enough to warrant complicating the repo, ci, publishing.
This project hasn't been touched in months, refactoring it into a monorepo has no benefit on the maintenance side at this time.

IMO something like this should be tabled until the need warrants the added complexity at which time, it would have outgrown webpack-contrib and should start cultivating it's own mini-ecosystem much like contrib is currently.

TheLarkInn commented 7 years ago

Should @hulkish proceed with something like separate repo's broken out instead?

joshwiens commented 7 years ago

I'm struggling to see how it's beneficial in either case at this point in time. We are talking about splitting out hundreds of lines of code into 8 separate libs of which one may or may not be beneficial as a standalone lib to a subset of the community.

As things are right now, it just feels like complexity for the sake of complexity. If you look at this as benefit vs. reward, I have a hard time finding a way where this comes out as a net gain.

Somebody sell me on it because we are talking about adding what is effectively 7 more libs of the org to maintain at some level, I'd want to see a larger community need before doing something like this.

TheLarkInn commented 7 years ago

That's fair. @hulkish is there no way to reuse existing functionality in other projects as it stands?

Ideally, I agree with @d3viant0ne that it may be more work than necessary and yield to his judgement, meanwhile I'm excited about this project or even pieces of it, being more widely used, and contributed to. If there is a blocker to this, I want to find a middle ground.

joshwiens commented 7 years ago

If we are going to do it, go big with it not middle of the road.

Scope it as @webpack-canary/*, give it a chance to grow and allow it to be the single exception to the standardization rule with the intent that if adoption grows, it will become a standalone mini-ecosystem.

If people honestly believe that publishing webpack-canary as multiple libs will be beneficial to consumers and help facilitate community adoption, i'm all for it regardless of personal opinions.

michael-ciniawsky commented 7 years ago

@TheLarkInn Yep that's what I meant Canary in the Goldmine like NodeJS, but with various recommended webpack configs for different popular frameworks/setups

This repo will/is highly complex in general, personally I wouldn't be even sure if it is possible to make a clear separation of concerns and if individual packages could run standalone, but since there is a ton of work left try and see what's possible first with a monorepo-ish architecture in mind ? The question of how to organise it maybe to early atm. I'm in favour of avoiding a monolith where possible, but would this be really the case for webpack-canary? Testing with different versions in different CI setups etc sounds sadly pretty depended on each another (I could be wrong though, like mentioned before not really the right person to answer this :D)

A compromise while triaging this could also be a monorepo branch , a git branch -D is cheap, it's up to the person if he/she wants to invest the time and work then, I personally have no problem deleting my god damn code the moment a better solution is found 😛, but that might not be for everyone...

hulkish commented 7 years ago

Not sure how to respond, lots of mixed responses here. Im gathering generally that this idea is rejected?

The scoped packages thing was an off handed suggestion. I'm not married to the idea.

@d3viant0ne This comment kind of caught me off guard:

I honestly don't see the point here outside of monorepos are the new in thing.

That truly was not my thinking. I saw value in moving to a monorepo. That's the only reason I suggested it.

That said... webpack was once considered the "new in thing". When did we arrive at discrediting solely because something is new?

I'm a new contributor to webpack core eco system. So, I'm bound to get it wrong at first.

I just want to help. But, if the majority rules against this then that's cool too.

webpack-contrib / webpack-canary