search

rescript-lang / rescript-compiler

The compiler for ReScript.
https://rescript-lang.org
Other
6.68k stars 445 forks source link

Pinned packages support #4361

Closed BlueHotDog closed 3 years ago

BlueHotDog commented 4 years ago

https://github.com/BlueHotDog/reason_monorepo

Monorepo support via common tooling(specifically in the above repo Lerna) is broken. Monorepo support for JS projects is usually being done by using symlinks to support cross-project dependencies - That's the most basic and common workflow. Another possible monorepo approach is by using Yarn workspaces, which utilizes symlinking even more by hoisting all dependencies to the "root".

Currently, bucklescript is unable to handle any kind of monorepo setup, which, after trying to bang my head, is impossible to solve and leads to a very weird workflow for any ambitious, project(e.g a modern SPA + Nodejs server in Reason).

The example project above illustrates all the problems, but just to summarize what i've encountered:

Links to previous discussions that died out and might be related: https://github.com/BuckleScript/bucklescript/issues/3521 https://github.com/BuckleScript/bucklescript/pull/4047

As there's no roadmap for reasonml. any type of communication regarding stale issue would really help.

TheSpyder commented 3 years ago

I stopped using bsb -w recently when I realised it runs via JS fs.watch(). I've been using a little wrapper to watchexec that works quite well, and it sounds like it will be perfect for pinned packages. I'll give the dev release a go soon.

fakenickels commented 3 years ago

I was using redemon but I was looking for dropping the extra dependencies

On Thu, 26 Nov 2020 at 09:32 Andrew Herron notifications@github.com wrote:

I stopped using bsb -w recently when I realised it runs via JS fs.watch(). I've been using a little wrapper to watchexec that works quite well, and it sounds like it will be perfect for pinned packages. I'll give the dev release a go soon.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-734272021, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJZJADZOZ5PYHVKPQR6ULLSRZDENANCNFSM4M4ORL5A .

-- Cheers, Gabriel Rubens

BlueHotDog commented 3 years ago

I'm having a weird issue while trying to build on netlify.. 

2:30:48 PM: ────────────────────────────────────────────────────────────────
2:30:48 PM:   1. Build command from Netlify app                             
2:30:48 PM: ────────────────────────────────────────────────────────────────
2:30:48 PM: ​
2:30:48 PM: $ yarn build
2:30:48 PM: yarn run v1.22.10
2:30:48 PM: $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production webpack --mode production
2:30:49 PM: $ bsb -make-world -clean-world
2:30:49 PM: bsb: error: loading 'build.ninja': No such file or directory
2:30:49 PM: Failed
2:30:49 PM: Dependency on reason-react
2:30:49 PM: bsb: error: build.ninja:2: expected '=', got ':'
2:30:49 PM: g_finger :=
2:30:49 PM:          ^ near here
2:30:49 PM: Failure: /opt/build/repo/node_modules/bs-platform/linux/ninja.exe
2:30:49 PM: Location: /opt/build/repo/node_modules/reason-react/lib/bs
2:30:49 PM: error Command failed with exit code 2.
2:30:49 PM: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2:30:49 PM: error Command failed with exit code 2.
2:30:49 PM: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
2:30:49 PM: ​
2:30:49 PM: ────────────────────────────────────────────────────────────────
2:30:49 PM:   "build.command" failed                                        
2:30:49 PM: ────────────────────────────────────────────────────────────────
2:30:49 PM: ​
2:30:49 PM:   Error message
2:30:49 PM:   Command failed with exit code 2: yarn build
2:30:49 PM: ​
2:30:49 PM:   Error location
2:30:49 PM:   In Build command from Netlify app:
2:30:49 PM:   yarn build
2:30:49 PM: ​
2:30:49 PM:   Resolved config
2:30:49 PM:   build:
2:30:49 PM:     base: /opt/build/repo/packages/demo
2:30:49 PM:     command: yarn build
2:30:49 PM:     commandOrigin: ui
2:30:49 PM:     environment:
2:30:49 PM:       - REVIEW_ID
2:30:49 PM:       - YARN_VERSION
2:30:49 PM:     publish: /opt/build/repo/packages/demo/build
2:30:49 PM:   redirects:
2:30:49 PM:     - from: /demo/assets/*
Coobaha commented 3 years ago

Here is an example where package with pinned dev dependency is not building https://github.com/Coobaha/bsb-smallest-monorepo-example

try to run yarn build and try to build packages/app on its own (yarn build:app is doing this)

More on this report is here https://forum.rescript-lang.org/t/call-for-testing-bs-platform-8-4-0-dev-2/661/19

packages
├── app                                  # depends on my-lib, dev depends on my-shared-dev-lib, pinned deps: my-shared-dev-lib, my-lib
├── my-lib                               # dev depends on my-shared-dev-lib, pinned deps: my-shared-dev-lib
└── my-shared-dev-lib

calling build via lerna (yarn build which just builds every package in order that is resolved by lerna) works just fine, but if you will try to build packages/app via yarn build or root alias for this yarn build:app compiler will throw:

bsb: [1/6] dev/DevDemo.ast
bsb: [2/6] src/MyLib.ast
bsb: [3/6] dev/DevDemo.d
bsb: [4/6] src/MyLib.d
bsb: [5/6] dev/DevDemo.cmj
FAILED: dev/DevDemo.cmj

  We've found a bug for you!
  /Projects/bsb-smallest-monorepo-example/packages/my-lib/dev/DevDemo.re:1:1-12

  1 │ DevLib.print("Hello, BuckleScript and Reason!");

  The module or file DevLib can't be found.
  - If it's a third-party dependency:
    - Did you list it in bsconfig.json?
    - Did you run `bsb` instead of `bsb -make-world`
      (latter builds third-parties)?
  - Did you include the file's directory in bsconfig.json?

Notice that i am build app but dependency that is missing in bsc is from my-lib

bobzhang commented 3 years ago

@BlueHotDog reading the log, it seems that you are mixing the old version of the bsb with the latest

bobzhang commented 3 years ago

@TheSpyder the watcher is decoupled with the build. You are recommended to use your own if you have a better candidate on your platform

Hongbo-bb commented 3 years ago

another thing is that -clean-world -make-world instead of -make-world -clean-world

On Thu, Nov 26, 2020 at 8:48 PM Danni Friedland notifications@github.com wrote:

I'm having a weird issue while trying to build on netlify..

2:30:48 PM: ──────────────────────────────────────────────────────────────── 2:30:48 PM: 1. Build command from Netlify app 2:30:48 PM: ──────────────────────────────────────────────────────────────── 2:30:48 PM: ​ 2:30:48 PM: $ yarn build 2:30:48 PM: yarn run v1.22.10 2:30:48 PM: $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production webpack --mode production 2:30:49 PM: $ bsb -make-world -clean-world 2:30:49 PM: bsb: error: loading 'build.ninja': No such file or directory 2:30:49 PM: Failed 2:30:49 PM: Dependency on reason-react 2:30:49 PM: bsb: error: build.ninja:2: expected '=', got ':' 2:30:49 PM: g_finger := 2:30:49 PM: ^ near here 2:30:49 PM: Failure: /opt/build/repo/node_modules/bs-platform/linux/ninja.exe 2:30:49 PM: Location: /opt/build/repo/node_modules/reason-react/lib/bs 2:30:49 PM: error Command failed with exit code 2. 2:30:49 PM: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. 2:30:49 PM: error Command failed with exit code 2. 2:30:49 PM: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. 2:30:49 PM: ​ 2:30:49 PM: ──────────────────────────────────────────────────────────────── 2:30:49 PM: "build.command" failed 2:30:49 PM: ──────────────────────────────────────────────────────────────── 2:30:49 PM: ​ 2:30:49 PM: Error message 2:30:49 PM: Command failed with exit code 2: yarn build 2:30:49 PM: ​ 2:30:49 PM: Error location 2:30:49 PM: In Build command from Netlify app: 2:30:49 PM: yarn build 2:30:49 PM: ​ 2:30:49 PM: Resolved config 2:30:49 PM: build: 2:30:49 PM: base: /opt/build/repo/packages/demo 2:30:49 PM: command: yarn build 2:30:49 PM: commandOrigin: ui 2:30:49 PM: environment: 2:30:49 PM: - REVIEW_ID 2:30:49 PM: - YARN_VERSION 2:30:49 PM: publish: /opt/build/repo/packages/demo/build 2:30:49 PM: redirects: 2:30:49 PM: - from: /demo/assets/*

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-734279661, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSV6OQBZKAYWONEWW2TSRZFCXANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

Hongbo-bb commented 3 years ago

The thing is that we treat packages in three categories which are handled slightly differently:

For a build, we have to decide which module format to be chosen, if we have multiple main entry points, there would be no single source of truth.

On Thu, Nov 26, 2020 at 9:29 PM Alexander Ryzhikov notifications@github.com wrote:

Here is an example where package with pinned dev dependency is not building https://github.com/Coobaha/bsb-smallest-monorepo-example

try to run yarn build and try to build packages/app on its own (yarn build:app is doing this)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-734298841, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSVVBWS7T5S4XC7WQB3SRZJ2HANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

Coobaha commented 3 years ago

For a build, we have to decide which module format to be chosen, if we have multiple main entry points, there would be no single source of truth.

Yes this is exactly how I see the main entry bsconfig too! Please check this https://github.com/Coobaha/bsb-smallest-monorepo-example and try to build packages/app as an entry point/root build, it will raise mentioned error for transitive pinned dev dependency..

Hongbo-bb commented 3 years ago

it works for me, see the attachment

On Fri, Nov 27, 2020 at 11:20 AM Alexander Ryzhikov < notifications@github.com> wrote:

For a build, we have to decide which module format to be chosen, if we have multiple main entry points, there would be no single source of truth.

Yes this is exactly how I see the main entry bsconfig too! Please check this https://github.com/Coobaha/bsb-smallest-monorepo-example and try to build packages/app as an entry point/root build, it will raise mentioned error for transitive pinned dev dependency..

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-734625154, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSWWQVA46WSEPDKOASLSR4LI3ANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

Coobaha commented 3 years ago

Interesting.. Might be related to the difference between fs, as I am using macos? I will add ubuntu github build

Coobaha commented 3 years ago

@Hongbo-bb https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1461940833?check_suite_focus=true

but build via lerna passes (job title is wrong) 😨 https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1461940796?check_suite_focus=true

UPD: Linux also fails to build https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1461964839?check_suite_focus=true

TheSpyder commented 3 years ago

@TheSpyder the watcher is decoupled with the build. You are recommended to use your own if you have a better candidate on your platform

This wasn't clear before, but the chat around custom watchers and pinned packages made me look deeper at what bsb -w actually does, and I'm not happy with it anymore 😂

I'll post some details on the forum soon about what switched to, it is cross platform. Using all native tooling means that for single-file incremental recompiles bsb can be done before I've released cmd+s on my keyboard. It's made an already fast feedback loop practically instant.

TheSpyder commented 3 years ago

I have spent the day trial-converting the big project I'm working on to pinned packages (using 8.4.0-dev.4). This was quite an effort, it's been running with a single flat bsconfig.json since September 2019 so we had some accidental circular package dependencies.

I am quite happy so far, I'm able to use public in sources specifications to only export the API modules from my big packages. This will later become namespace:true but I don't want to rename anything just yet.

My feedback:

TheSpyder commented 3 years ago

I'm confused about where I need to specify pinned-dependencies. Is it only at the entry point?

Looks like yes. I removed all of the pinned-dependencies from anywhere but the entry point and it behaves the same. This definitely needs to be mentioned in the docs once this feature is released, imo.

Coobaha commented 3 years ago

Related to the above, I'm confused about where I need to specify pinned-dependencies. Is it only at the entry point? Or in every bsconfig.json? I've been doing it in every file (which is why it's so annoying) but now I'm wondering if that was necessary.

Yeah it is annoying, I wrote a script for our monorepo that scans all monorepo packages:

  1. Adds all monorepo packages from package.json as pinned dependency
  2. Checks bsconfig for missing RE dependencies from package.json and warns
  3. Checks package.json for missing deps, dev deps, pinned-deps from bsconfig

Maybe bsb can support this out of the box at some point?

BlueHotDog commented 3 years ago

@BlueHotDog reading the log, it seems that you are mixing the old version of the bsb with the latest

hmm, just triple verified, using the latest, the failure happens on github actions/netlify/vercel so very hard to debug. But i also added "bs-platform": "^8.4.0-dev.4" to the monorepo resolution, and it still fails :(

bobzhang commented 3 years ago

@fakenickels

If a pinned dep has a source dir with the type dev the compiler gives an error saying it can't find the bs-dev-dependencies of the pinned dep (Jest in this case). Is that an intended behavior?

For pinned package, the type dev directory will get compiled, and bs-dev-dependencies needs to be installed

bobzhang commented 3 years ago

@Coobaha I tried again, it works for me:

bsb-smallest-monorepo-example$yarn build
yarn run v1.19.2
$ lerna run build --concurrency 1 --stream
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Executing command in 3 packages: "yarn run build"
@myorg/my-shared-dev-lib: $ bsb -make-world
@myorg/my-shared-dev-lib: Dependency Finished
@myorg/my-shared-dev-lib: bsb: [1/3] src/DevLib.ast
@myorg/my-shared-dev-lib: bsb: [2/3] src/DevLib.d
@myorg/my-shared-dev-lib: bsb: [3/3] src/DevLib.cmj
@myorg/my-lib: $ bsb -make-world
@myorg/my-lib: Dependency pinned on @myorg/my-shared-dev-lib
@myorg/my-lib: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib 
@myorg/my-lib: bsb: [1/3] src/DevLib.ast
@myorg/my-lib: bsb: [2/3] src/DevLib.d
@myorg/my-lib: bsb: [3/3] src/DevLib.cmj
@myorg/my-lib: bsb: [1/1] DevLib.cmt
@myorg/my-lib: Dependency Finished
@myorg/my-lib: bsb: [1/6] dev/DevDemo.ast
@myorg/my-lib: bsb: [2/6] src/MyLib.ast
@myorg/my-lib: bsb: [3/6] dev/DevDemo.d
@myorg/my-lib: bsb: [4/6] src/MyLib.d
@myorg/my-lib: bsb: [5/6] dev/DevDemo.cmj
@myorg/my-lib: bsb: [6/6] src/MyLib.cmj
@myorg/app: $ bsb -make-world
@myorg/app: Dependency pinned on @myorg/my-lib
@myorg/app: bsb: [1/6] dev/DevDemo.ast
@myorg/app: bsb: [2/6] src/MyLib.ast
@myorg/app: bsb: [3/6] dev/DevDemo.d
@myorg/app: bsb: [4/6] src/MyLib.d
@myorg/app: bsb: [5/6] dev/DevDemo.cmj
@myorg/app: bsb: [6/6] src/MyLib.cmj
@myorg/app: bsb: [1/2] DevDemo.cmt
@myorg/app: bsb: [2/2] MyLib.cmt
@myorg/app: Dependency pinned on @myorg/my-shared-dev-lib
@myorg/app: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib 
@myorg/app: bsb: [1/3] src/DevLib.ast
@myorg/app: bsb: [2/3] src/DevLib.d
@myorg/app: bsb: [3/3] src/DevLib.cmj
@myorg/app: bsb: [1/1] DevLib.cmt
@myorg/app: Dependency Finished
@myorg/app: bsb: [1/2] dev/DevDemo-App.cmj
@myorg/app: bsb: [2/2] src/Demo-App.cmj
lerna success run Ran npm script 'build' in 3 packages in 1.7s:
lerna success - @myorg/app
lerna success - @myorg/my-lib
lerna success - @myorg/my-shared-dev-lib
✨  Done in 2.54s.
bsb-smallest-monorepo-example$yarn build:app
yarn run v1.19.2
$ cd packages/app && yarn build
$ bsb -make-world
Dependency pinned on @myorg/my-lib
bsb: [2/2] dev/DevDemo.cmj
bsb: [2/2] DevDemo.cmt
Dependency pinned on @myorg/my-shared-dev-lib
Dependency Finished
bsb: [2/2] src/Demo-App.cmj
✨  Done in 0.65s.
bsb-smallest-monorepo-example$
bobzhang commented 3 years ago

@TheSpyder

Whether by accident or design, it's really nice that dev dependencies are able to disconnect what would normally be a circular dependency.

It is intentional, we want to cut dependency as much as we can.

The design decision (probably made years ago) to not auto-include transitive dependencies is annoying

This is intentional, I don't see any package manager take transitive dependencies into account, I think this is also modular, users have to list its real dependencies.

Related to the above, I'm confused about where I need to specify pinned-dependencies. Is it only at the entry point?

Yes, I will document it once released.

Before I started migrating to pinned packages, bsb -make-world was completing more or less instantly. Now, even with zero changes, it seems to rebuilding all pinned packages every time it runs

If you want great performance, I would suggest you stay with true monorepo. If you split it into multiple packages, the build system always need check the status of different packages, it takes some time. If your dependency changes, it will trigger a lot of rebuild

bobzhang commented 3 years ago

@BlueHotDog anyway it could be reproduced? The error message does show that you are using an old version of ninja

bobzhang commented 3 years ago

Hi all, I think most concerns are resolved on master. Since this issue gets very lengthy I will close this issue, there will be some bugs, you can file it in a new issue.

Coobaha commented 3 years ago

@Coobaha I tried again, it works for me:

bsb-smallest-monorepo-example$yarn build
yarn run v1.19.2
$ lerna run build --concurrency 1 --stream
lerna notice cli v3.22.1
lerna info versioning independent
lerna info Executing command in 3 packages: "yarn run build"
@myorg/my-shared-dev-lib: $ bsb -make-world
@myorg/my-shared-dev-lib: Dependency Finished
@myorg/my-shared-dev-lib: bsb: [1/3] src/DevLib.ast
@myorg/my-shared-dev-lib: bsb: [2/3] src/DevLib.d
@myorg/my-shared-dev-lib: bsb: [3/3] src/DevLib.cmj
@myorg/my-lib: $ bsb -make-world
@myorg/my-lib: Dependency pinned on @myorg/my-shared-dev-lib
@myorg/my-lib: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib 
@myorg/my-lib: bsb: [1/3] src/DevLib.ast
@myorg/my-lib: bsb: [2/3] src/DevLib.d
@myorg/my-lib: bsb: [3/3] src/DevLib.cmj
@myorg/my-lib: bsb: [1/1] DevLib.cmt
@myorg/my-lib: Dependency Finished
@myorg/my-lib: bsb: [1/6] dev/DevDemo.ast
@myorg/my-lib: bsb: [2/6] src/MyLib.ast
@myorg/my-lib: bsb: [3/6] dev/DevDemo.d
@myorg/my-lib: bsb: [4/6] src/MyLib.d
@myorg/my-lib: bsb: [5/6] dev/DevDemo.cmj
@myorg/my-lib: bsb: [6/6] src/MyLib.cmj
@myorg/app: $ bsb -make-world
@myorg/app: Dependency pinned on @myorg/my-lib
@myorg/app: bsb: [1/6] dev/DevDemo.ast
@myorg/app: bsb: [2/6] src/MyLib.ast
@myorg/app: bsb: [3/6] dev/DevDemo.d
@myorg/app: bsb: [4/6] src/MyLib.d
@myorg/app: bsb: [5/6] dev/DevDemo.cmj
@myorg/app: bsb: [6/6] src/MyLib.cmj
@myorg/app: bsb: [1/2] DevDemo.cmt
@myorg/app: bsb: [2/2] MyLib.cmt
@myorg/app: Dependency pinned on @myorg/my-shared-dev-lib
@myorg/app: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib 
@myorg/app: bsb: [1/3] src/DevLib.ast
@myorg/app: bsb: [2/3] src/DevLib.d
@myorg/app: bsb: [3/3] src/DevLib.cmj
@myorg/app: bsb: [1/1] DevLib.cmt
@myorg/app: Dependency Finished
@myorg/app: bsb: [1/2] dev/DevDemo-App.cmj
@myorg/app: bsb: [2/2] src/Demo-App.cmj
lerna success run Ran npm script 'build' in 3 packages in 1.7s:
lerna success - @myorg/app
lerna success - @myorg/my-lib
lerna success - @myorg/my-shared-dev-lib
✨  Done in 2.54s.
bsb-smallest-monorepo-example$yarn build:app
yarn run v1.19.2
$ cd packages/app && yarn build
$ bsb -make-world
Dependency pinned on @myorg/my-lib
bsb: [2/2] dev/DevDemo.cmj
bsb: [2/2] DevDemo.cmt
Dependency pinned on @myorg/my-shared-dev-lib
Dependency Finished
bsb: [2/2] src/Demo-App.cmj
✨  Done in 0.65s.
bsb-smallest-monorepo-example$

@bobzhang could you also try yarn clean && yarn build:app? While your local builds are passing please check github builds here https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/386395351

TheSpyder commented 3 years ago

If you want great performance, I would suggest you stay with true monorepo. If you split it into multiple packages, the build system always need check the status of different packages, it takes some time.

The performance isn't a problem if nothing builds. It's the incorrect rebuilding that is a problem.

If your dependency changes, it will trigger a lot of rebuild

Even with no changes it triggers a lot of rebuild. This doesn't happen without pinned packages. I'll see if updating my example repo reproduces the problem and file a new issue.

ixzzd commented 3 years ago

I'm having a weird issue while trying to build on netlify..

@BlueHotDog I have the same error building app on linux.

@bobzhang Looks like linux ninja binary is outdated, is it possible?

Hongbo-bb commented 3 years ago

Indeed, it seems ninja binary was not updated on Linux.. Egor Chemokhonenko notifications@github.com于2020年11月30日 周一下午7:16写道：

I'm having a weird issue while trying to build on netlify..

@BlueHotDog https://github.com/BlueHotDog I have the same error building app on linux.

@bobzhang https://github.com/bobzhang Looks like linux ninja binary is outdated, is it possible?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735722905, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSXSDJQDFU5KOLI4DDTSSN5H5ANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

BlueHotDog commented 3 years ago

@Hongbo-bb could you please ping when i can test again?

Hongbo-bb commented 3 years ago

sure thing

On Tue, Dec 1, 2020 at 1:15 AM Danni Friedland notifications@github.com wrote:

@Hongbo-bb https://github.com/Hongbo-bb could you please ping when i can test again?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735921688, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSRWXV5TKZFHAVDJMRDSSPHMNANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

Hongbo-bb commented 3 years ago

Hi, thanks for reporting this issue, should be fixed here: https://github.com/rescript-lang/rescript-compiler/pull/4854

On Mon, Nov 30, 2020 at 2:31 PM Alexander Ryzhikov notifications@github.com wrote:

@Coobaha https://github.com/Coobaha I tried again, it works for me:

bsb-smallest-monorepo-example$yarn build

yarn run v1.19.2

$ lerna run build --concurrency 1 --stream

lerna notice cli v3.22.1

lerna info versioning independent

lerna info Executing command in 3 packages: "yarn run build"

@myorg/my-shared-dev-lib: $ bsb -make-world

@myorg/my-shared-dev-lib: Dependency Finished

@myorg/my-shared-dev-lib: bsb: [1/3] src/DevLib.ast

@myorg/my-shared-dev-lib: bsb: [2/3] src/DevLib.d

@myorg/my-shared-dev-lib: bsb: [3/3] src/DevLib.cmj

@myorg/my-lib: $ bsb -make-world

@myorg/my-lib: Dependency pinned on @myorg/my-shared-dev-lib

@myorg/my-lib: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/my-lib/node_modules/@myorg/my-shared-dev-lib

@myorg/my-lib: bsb: [1/3] src/DevLib.ast

@myorg/my-lib: bsb: [2/3] src/DevLib.d

@myorg/my-lib: bsb: [3/3] src/DevLib.cmj

@myorg/my-lib: bsb: [1/1] DevLib.cmt

@myorg/my-lib: Dependency Finished

@myorg/my-lib: bsb: [1/6] dev/DevDemo.ast

@myorg/my-lib: bsb: [2/6] src/MyLib.ast

@myorg/my-lib: bsb: [3/6] dev/DevDemo.d

@myorg/my-lib: bsb: [4/6] src/MyLib.d

@myorg/my-lib: bsb: [5/6] dev/DevDemo.cmj

@myorg/my-lib: bsb: [6/6] src/MyLib.cmj

@myorg/app: $ bsb -make-world

@myorg/app: Dependency pinned on @myorg/my-lib

@myorg/app: bsb: [1/6] dev/DevDemo.ast

@myorg/app: bsb: [2/6] src/MyLib.ast

@myorg/app: bsb: [3/6] dev/DevDemo.d

@myorg/app: bsb: [4/6] src/MyLib.d

@myorg/app: bsb: [5/6] dev/DevDemo.cmj

@myorg/app: bsb: [6/6] src/MyLib.cmj

@myorg/app: bsb: [1/2] DevDemo.cmt

@myorg/app: bsb: [2/2] MyLib.cmt

@myorg/app: Dependency pinned on @myorg/my-shared-dev-lib

@myorg/app: Duplicated package: bs-platform /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-lib/node_modules/bs-platform (chosen) vs /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib/node_modules/bs-platform in /Users/hongbozhang/git/bsb-smallest-monorepo-example/packages/app/node_modules/@myorg/my-shared-dev-lib

@myorg/app: bsb: [1/3] src/DevLib.ast

@myorg/app: bsb: [2/3] src/DevLib.d

@myorg/app: bsb: [3/3] src/DevLib.cmj

@myorg/app: bsb: [1/1] DevLib.cmt

@myorg/app: Dependency Finished

@myorg/app: bsb: [1/2] dev/DevDemo-App.cmj

@myorg/app: bsb: [2/2] src/Demo-App.cmj

lerna success run Ran npm script 'build' in 3 packages in 1.7s:

lerna success - @myorg/app

lerna success - @myorg/my-lib

lerna success - @myorg/my-shared-dev-lib

✨ Done in 2.54s.

bsb-smallest-monorepo-example$yarn build:app

yarn run v1.19.2

$ cd packages/app && yarn build

$ bsb -make-world

Dependency pinned on @myorg/my-lib

bsb: [2/2] dev/DevDemo.cmj

bsb: [2/2] DevDemo.cmt

Dependency pinned on @myorg/my-shared-dev-lib

Dependency Finished

bsb: [2/2] src/Demo-App.cmj

✨ Done in 0.65s.

bsb-smallest-monorepo-example$

@bobzhang https://github.com/bobzhang could you also try yarn clean && yarn build:app? While your local builds are passing please check github builds here https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/386395351

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735580001, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSQVKQKMVPLRTQNJXA3SSM32DANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

Hongbo-bb commented 3 years ago

Hi, it would be helpful if you can make a reproducible repo, I am planning to cut a release early next week, thanks

On Mon, Nov 30, 2020 at 2:31 PM Andrew Herron notifications@github.com wrote:

If you want great performance, I would suggest you stay with true monorepo. If you split it into multiple packages, the build system always need check the status of different packages, it takes some time.

The performance isn't a problem if nothing builds. It's the incorrect rebuilding that is a problem.

If your dependency changes, it will trigger a lot of rebuild

Even with no changes it triggers a lot of rebuild. This doesn't happen without pinned packages. I'll see if updating my example repo reproduces the problem and file a new issue.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735580027, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSQPQO3NCLALS6HMEJLSSM32LANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

TheSpyder commented 3 years ago

Hi, it would be helpful if you can make a reproducible repo, I am planning to cut a release early next week, thanks …

yes, sorry, I've been busy but I'll give it a go tonight or tomorrow

TheSpyder commented 3 years ago

I couldn't make my example reason monorepo reproduce the issue so I'll try making it more complex. In the meantime I switched back to my main repo and I was wrong about something:

Whether by accident or design, it's really nice that dev dependencies are able to disconnect what would normally be a circular dependency.

It is intentional, we want to cut dependency as much as we can.

I must have put my repository into a weird state to make that compile. The problem is that it's still a circular dependency:

So unless the dev sources can be compiled in a separate pass I'll need to unwind those dependency links. On a clean repo it's giving me a module not found error compiling the tests of library 1, which is a bit misleading but oh well.

TheSpyder commented 3 years ago

Looks like the weird state I was seeing where no changes caused a full rebuild were related to the cycle that wasn't really a cycle. With a clean checkout and no fake cycles it's fine. Sorry about that.

It's still unusable for large monorepo projects due to this thing you said earlier

If your dependency changes, it will trigger a lot of rebuild

But I'm going to log a new issue for that as it is a separate discussion. Pinned packages seem to be working as designed.

Hongbo-bb commented 3 years ago

Hi, I made a release 8.4.0 under dev tag, it should work on Linux/Windows/Mac

On Tue, Dec 1, 2020 at 1:15 AM Danni Friedland notifications@github.com wrote:

@Hongbo-bb https://github.com/Hongbo-bb could you please ping when i can test again?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-735921688, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSRWXV5TKZFHAVDJMRDSSPHMNANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

Coobaha commented 3 years ago

@Hongbo-bb amazing! it seems to pass, i've noticed that linux dist seems to be incorrectly packed? https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1505808007?check_suite_focus=true

Build passes on mac and uses correct bsb :)

BlueHotDog commented 3 years ago

Doesnt work :(

$ bsb -clean-world -make-world
--
11:52:11.055 | bsb: error: loading 'build.ninja': No such file or directory
11:52:11.056 | Failed
11:52:11.060 | bs-platform version mismatch Running bsb 8.4.0-dev.1 (/vercel/workpath0/node_modules/bs-platform) vs vendored 8.4.0 (/vercel/workpath0/node_modules/bs-platform)
11:52:11.076 | error Command failed with exit code 2.
11:52:11.076 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:52:11.096 | error Command failed with exit code 2.
11:52:11.096 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
11:52:11.105 | Error: Command "yarn build" exited with 2
11:52:13.238 | Done with "package.json"
Hongbo-bb commented 3 years ago

Indeed, it seems I made a mistake for publishing on linux. Should be good now -- 8.4.1

On Sun, Dec 6, 2020 at 5:44 PM Alexander Ryzhikov notifications@github.com wrote:

@Hongbo-bb https://github.com/Hongbo-bb amazing! it seems to pass, i've noticed that linux dist seems to be incorrectly packed? https://github.com/Coobaha/bsb-smallest-monorepo-example/runs/1505808007?check_suite_focus=true

Build passes on mac and uses correct bsb :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-739479013, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSTIAYKWTL2E23YXVTLSTNHAXANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

BlueHotDog commented 3 years ago

Just tried again :( I'm a bit lost between all the versions, but seems to not work still: Still getting this weird error.

19:51:04.978 | [4/4] Building fresh packages...
-- | --
19:51:16.633 | Done in 112.46s.
19:51:17.028 | yarn run v1.22.4
19:51:17.058 | $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production yarn js:build
19:51:17.264 | $ bsb -clean-world -make-world
19:51:17.336 | bsb: error: loading 'build.ninja': No such file or directory
19:51:17.336 | Failed
19:51:17.341 | bs-platform version mismatch Running bsb 8.4.0-dev.1 (/vercel/workpath0/node_modules/bs-platform) vs vendored 8.4.0 (/vercel/workpath0/node_modules/bs-platform)
19:51:17.352 | error Command failed with exit code 2.
19:51:17.352 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.369 | error Command failed with exit code 2.
19:51:17.369 | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.377 | Error: Command "yarn build " exited with 2
19:51:19.526 | Done with "package.json"
Hongbo-bb commented 3 years ago

can you double check that if your bs-platform is indeed 8.4.1?

On Mon, Dec 7, 2020 at 1:55 AM Danni Friedland notifications@github.com wrote:

Just tried again :( I'm a bit lost between all the versions, but seems to not work still: Still getting this weird error.

19:51:04.978 [4/4] Building fresh packages...
19:51:16.633 Done in 112.46s.
19:51:17.028 yarn run v1.22.4
19:51:17.058 $ NODE_ENV=production yarn re:cleanbuild && NODE_ENV=production yarn js:build
19:51:17.264 $ bsb -clean-world -make-world
19:51:17.336 bsb: error: loading 'build.ninja': No such file or directory
19:51:17.336 Failed
19:51:17.341 bs-platform version mismatch Running bsb 8.4.0-dev.1 (/vercel/workpath0/node_modules/bs-platform) vs vendored 8.4.0 (/vercel/workpath0/node_modules/bs-platform)
19:51:17.352 error Command failed with exit code 2.
19:51:17.352 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.369 error Command failed with exit code 2.
19:51:17.369 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
19:51:17.377 Error: Command "yarn build " exited with 2
19:51:19.526 Done with "package.json"

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-739538789, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSWKJDS4VK6N66J27ITSTPASLANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

BlueHotDog commented 3 years ago

i did, also made sure yarn.lock has 8.4.1. it fails the same way on chromatic/vercel

Hongbo-bb commented 3 years ago

The log is weird, what's the output of npx bsb -v? it should say 8.4.1 on your platform

On Mon, Dec 7, 2020 at 3:27 PM Danni Friedland notifications@github.com wrote:

i did, also made sure yarn.lock has 8.4.1. it fails the same way on chromatic/vercel

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-739727366, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSSNDE3IVXYBSWNT64TSTR7UZANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

BlueHotDog commented 3 years ago

Yeah, super weird. and it works locally on my OSX. the problem is, i dont have shell access to any of these platforms. pinging you on Discord to maybe give you some other way to look.

Coobaha commented 3 years ago

It seems to be passing on Linux, before it was failing with the same error as you have @BlueHotDog https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/403993626

Can you reproduce this error with my repo and vercel build?

BlueHotDog commented 3 years ago

It seems to be passing on Linux, before it was failing with the same error as you have @BlueHotDog https://github.com/Coobaha/bsb-smallest-monorepo-example/actions/runs/403993626

Can you reproduce this error with my repo and vercel build?

Just tried, and seems to work. well. no idea what is happening.. and no idea what this error means :( The thing is that it also fails on Chromatic and they dont share the same cache/build pipeline

TheSpyder commented 3 years ago

But I'm going to log a new issue for that as it is a separate discussion

I won't have time to log the issue until tomorrow, probably, but I have added pinned packages support to my example monorepo (it's probably a good way to show how pinned packages work, actually) and there's a blurb in the readme about why it isn't quite good enough for large projects.

Hongbo-bb commented 3 years ago

Changing ExampleStyles.re in b causes package c to rebuild even though c does not use it. Changing ATest.re - which is a development source file and therefore cannot have any impact on the exported code - triggers a complete rebuild of packages b and c.

This could be fixed but I think it still won't match with the expectation of true monorepo -- when you do change the interface type, it would trigger the whole project rebuild. In that case, the benefit of such minor improvement is not that high?

On Tue, Dec 8, 2020 at 4:56 AM Andrew Herron notifications@github.com wrote:

But I'm going to log a new issue for that as it is a separate discussion

I won't have time to log the issue until tomorrow, probably, but I have added pinned packages support to my example monorepo https://github.com/TheSpyder/reason_monorepo (it's probably a good way to show how pinned packages work, actually) and there's a blurb in the readme about why it isn't quite good enough for large projects.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-740175050, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSRWWQEVTNDSBNGZHPLSTU6PZANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

Hongbo-bb commented 3 years ago

@Andrew Herron andrew.herron@gmail.com I made an experimental PR in #4863 to support more fine-grained dependency tracking. This is the best that we can achieve given we treat the package as a black box. It works best when you write interface file for your packages

On Tue, Dec 8, 2020 at 11:53 AM Hongbo Zhang hongboz@seas.upenn.edu wrote:

Changing ExampleStyles.re in b causes package c to rebuild even though c does not use it. Changing ATest.re - which is a development source file and therefore cannot have any impact on the exported code - triggers a complete rebuild of packages b and c.

This could be fixed but I think it still won't match with the expectation of true monorepo -- when you do change the interface type, it would trigger the whole project rebuild. In that case, the benefit of such minor improvement is not that high?

On Tue, Dec 8, 2020 at 4:56 AM Andrew Herron notifications@github.com wrote:

But I'm going to log a new issue for that as it is a separate discussion

I won't have time to log the issue until tomorrow, probably, but I have added pinned packages support to my example monorepo https://github.com/TheSpyder/reason_monorepo (it's probably a good way to show how pinned packages work, actually) and there's a blurb in the readme about why it isn't quite good enough for large projects.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-740175050, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSRWWQEVTNDSBNGZHPLSTU6PZANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

-- -- Regards, Hongbo

TheSpyder commented 3 years ago

Thanks! I look forward to testing it, I do usually have more interfaces it was just an example

Hongbo-bb commented 3 years ago

Hi @Andrew Herron andrew.herron@gmail.com I published a version 8.4.2-dev.1 for Mac, let me know if it works for you

On Tue, Dec 8, 2020 at 3:13 PM Andrew Herron notifications@github.com wrote:

Thanks! I look forward to testing it, I do usually have more interfaces it was just an example

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rescript-lang/rescript-compiler/issues/4361#issuecomment-740430668, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADLRMSVEVMTNIABWORTXDULSTXGZBANCNFSM4M4ORL5A .

-- -- Regards, Hongbo

TheSpyder commented 3 years ago

I published a version 8.4.2-dev.1 for Mac, let me know if it works for you

That's beautiful, thank you! 😁

So the change means that saving an implementation will no longer trigger dependent rebuilds, but saving an interface still does? I tested by swapping back and forth between pinned packages and a single flat bsconfig.json, then saving one of my .rei files (no changes but it causes the file to be built).

The single namespace is able to avoid dependent modules rebuilding in this scenario when the interface contents remains the same. As a pinned package, that same change causes all dependent modules to rebuild; presumably because the build system is operating in new contexts for dependencies and only checks the file timestamp not whether the interface actually changed.

This means it's still slower to develop on than the single namespace but it's vastly improved from 8.4.1 and I think that tips the balance such that team will overlook the speed difference to get the improved monorepo experience.