Why yarn? Any chance pnpm could eventually be supported (in the name of saving disk space)?

matronator commented 3 years ago

I looked (okay, not that hard, but I definitely didn't see it mentioned in the tutorial), but couldn't find an explanation of why is yarn a requirement? Can you please clarify to me, why yarn?

I primarily use pnpm as my package manager, mostly because I'm slightly worried about my 8 years old MacBook's internal SSD running out of lifecycles due to excessive npm install garbage. And yarn isn't much better in this, it installs just as much garbage as npm. So anyways, onto my second question:

Is there any possibility in the future, that npm/pnpm would be supported as well and not just yarn?

Thank you!

peterp commented 3 years ago

@matronator Yup, there is no explicit reason, other than we have some technical debt that we've been cleaning up. We're getting there!

@dac09 Could we run an experiment on the RedwoodJS project side to see if it's possible?

thedavidprice commented 3 years ago

We use Yarn for Yarn Workspaces. NPM did not support Workspaces until recently, which may mean we can switch in the future.

General FYI: I did an experiment two weeks ago using Yarn 2 with the Redwood Framework — even without pnpm there were major breaking changes and substantial behavior changes to dependency management and resolutions. Importantly, they are handling Workspaces differently in 2. Nothing a dealbreaker, but it did make me inclined to wait for Yarn 3 GA before re-evalutating.

ajcwebdev commented 3 years ago

If we decide to move forward with Yarn 3 we should take a look at what a migration to pnpm or any other package manager would look like and how it would compare to upgrading. There's a lot of innovation happening right now in this space so it'll be worth taking some time to make sure we're future proofing ourselves as much as possible.

jtoar commented 3 years ago

StackBlitz also teased their own package manager; they haven't released it yet (I'm assuming they will one day)—just another one to consider.

thedavidprice commented 3 years ago

Early days, but NPM v7 might be a case similar to Yarn v2 in regards to some big, fundamental changes. Hot take. Needs verification.

Albert-Gao commented 3 years ago

Early days, but NPM v7 might be a case similar to Yarn v2 in regards to some big, fundamental changes. Hot take. Needs verification.

Correct, I ditched npm v7 and started using Yarn 1 because of the breaking change in v7, just do not want to use that --legacy-peer-dep arg. 😂

ajcwebdev commented 3 years ago

Yarn 3 announcement

Grist for the mill

While the migration from Yarn 1 to Yarn 2 brought some discomfort, the migration to Yarn 3 should prove easier - regardless of the version you come from. The user-facing breaking changes we made this season are mostly little details that may only affect you in very specific cases:

Node 10 isn't supported anymore

Plug'n'Play hooks are now called .pnp.cjs (vs .pnp.js)

Virtual folders are now called __virtual__ (vs $virtual)

The editor SDKs have been moved to @yarnpkg/sdks

Etc; full list here

Even for Yarn 1 users, migrating from 1 to 3 should be easier: we made it so that Yarn will detect when this situation arises to then automatically enable the node-modules linker. That alone should address most of the problems you may have been hitting when attempting the upgrade - and for everything else, make sure to take a look at our Migration Guide which got significantly improved over the past year.

ajcwebdev commented 3 years ago

From Maël, lead Yarn maintainer:

I'd recommend to attempt the migration and ping us on Discord once you hit a blocker - if you use the n_m linker it should be straightforward, although I don't know what "workspaces [are handled] differently in 2" was referring to 🤔. But I'm sure if you find specific problems our lil' Discord community will be able to help 🙂.

thedavidprice commented 3 years ago

Interesting. I'm curious about taking this for a test run. Although I have strong doubt about this quote in particular --> "Even for Yarn 1 users, migrating from 1 to 3 should be easier". Lerna and Yarn Workspaces got into a fight when I tried to upgrade to v2 for a POC.

Overall, this feels like good news. I just think it's going to take about 6 months to see where the ecosystem lands in terms of support. E.g. even if we upgrade to Yarn v3 for the framework, will projects using v3 have build support on Netlify, Vercel, Render, etc...

peterp commented 3 years ago

I'm trying to get to the point where we don't dictate the package manager for Redwood projects.

My goal is that we create single binary that encases the build and design time packages. The user will only install the run-time packages: express, react, apollo-graphql, etc.

Then we don't have to rely on workspaces and a user is free to use whatever they prefer as a package manager.

A bunch of added advantages: crazy fast installations.

larixer commented 3 years ago

@thedavidprice

Overall, this feels like good news. I just think it's going to take about 6 months to see where the ecosystem lands in terms of support. E.g. even if we upgrade to Yarn v3 for the framework, will projects using v3 have build support on Netlify, Vercel, Render, etc...

You can use node_modules installs with Yarn 2 and Yarn 3. With Yarn 3 you can also benefit from content-addressable storage (nmMode: hardlinks-global), when all your unique files from different packages will be shared across multiple projects in the same place on disk - the same idea as used by pnpm and requested by @matronator to save disk space, but without modifications to node_modules layout via symlinks, hence it should be compatible to all existing libraries for npm.

You don't need to wait for ecosystem to be ready for Yarn 3 if you will continue to use node_modules.

thedavidprice commented 3 years ago

Hi @larixer! Thank you for jumping in here.

These all sound like great benefits. Definitely want to look more into understanding nmMode: hardlinks-global — sounds very nice! And know that all along I've been excited and impressed with the heavy lifting the Yarn team has been doing. We literally couldn't have built Redwood the way we did without Yarn.

I did use node_modules in my attempt to upgrade to Yarn v2 in our Framework monorep. What I discovered was a lot of emerging resolution issues, likely due to improvements that revealed issues we've had along. More importantly, our use and setup of Yarn Workspaces threw a lot of errors and broke the scripts/tools we have in place. If you're curious (and have time for some suggestions), take a look at our scripts in the project's root package.json here: https://github.com/redwoodjs/redwood/blob/d9d385244ec3a5a6d6b8da5c58dc3295676f54e2/package.json#L24-L38

Overall, it felt like there was a path forward. Just not a clean one.

But know I'm motivated to give Yarn v3 a trial run in the near future!

larixer commented 3 years ago

What I discovered was a lot of emerging resolution issues, likely due to improvements that revealed issues we've had along. More importantly, our use and setup of Yarn Workspaces threw a lot of errors and broke the scripts/tools we have in place. If you're curious (and have time for some suggestions), take a look at our scripts in the project's root package.json here: https://github.com/redwoodjs/redwood/blob/d9d385244ec3a5a6d6b8da5c58dc3295676f54e2/package.json#L24-L38

Yes, I'm curious and willing to take a look. I am not sure what I'm looking at, you referenced one commit and I don't understand what exactly have you tried to do and what problems have you faced? Do you have some Pull Request where you have tried to migrate to Yarn 2 with description of problems you faced? Alternatively are you comfortable to pop on our discord: https://discord.com/invite/yarnpkg where we can discuss the issues you faced in real time?

thedavidprice commented 3 years ago

Well, look at that "Upgrade Framework to Yarn v3" #3154

😉

thedavidprice commented 3 years ago

@matronator thank you for your patience while I hijacked this issue to resolve one of the tech debt issues! Now back to focusing on your original question:

Is there any possibility in the future, that npm/pnpm would be supported as well and not just yarn?

Now that the Framework is using Yarn v3, we have identified the outstanding issue(s) to resolve in order to use Yarn v3 in a Redwood project. My assumption is that we need to resolve the same issues to be able to use pnpm (instead of Yarn) for a Redwood Project. See:

https://github.com/redwoodjs/redwood/issues/3517

@jtoar is going to take a look at requiring the dependencies (and therefore binaries) correctly within our CLI Package. Any/all help would be appreciated, including results from testing pnpm with a fresh Redwood Project. No pressure.

matronator commented 3 years ago

@thedavidprice No worries 😉

Okay, I tried running the tutorial with pnpm instead of yarn and got stuck trying to run the project. I ran into similar scoping issues with pnpm run as with Yarn 3 - it doesn't execute child binaries. I had a look at https://github.com/redwoodjs/redwood/issues/3517, but wasn't quite sure where the createRequire code snippet should go to 😔

Anyways, running pnpm init redwood-app kinda worked, but didn't finish all the steps.

yarn create redwood-app ./redwoodblog
// pnpm equivalent is
pnpm init redwood-app ./redwoodblog

It failed on the yarn install step, but now that I'm reading through the output, it might be because of the node version being incompatible with one or more packages (errored on babel-jest@27.2.0).

Click to show the full output


```node
  ✔ Creating Redwood app
    ✔ Checking node and yarn compatibility
    ✔ Creating directory '/Users/matronator/Documents/Work.nosync/tests/redwoodblog'
  ❯ Installing packages
    ✖ Running 'yarn install'... (This could take a while)
      → info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
    Convert TypeScript files to JavaScript
    Generating types

Error: Command failed with exit code 1: yarn install
warning @redwoodjs/core > webpack-dev-server > sockjs > uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
warning @redwoodjs/core > @redwoodjs/cli > @prisma/sdk > temp-write > uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
warning @redwoodjs/core > webpack-dev-server > url > querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
warning @redwoodjs/core > @redwoodjs/testing > @storybook/react > @pmmmwh/react-refresh-webpack-plugin > native-url > querystring@0.2.1: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
warning @redwoodjs/core > @redwoodjs/testing > @storybook/builder-webpack5 > webpack-hot-middleware > querystring@0.2.1: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
warning @redwoodjs/core > svg-react-loader > css > urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
warning @redwoodjs/core > svg-react-loader > css > source-map-resolve > urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
warning @redwoodjs/core > babel-timing > webpack > watchpack > watchpack-chokidar2 > chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.
warning @redwoodjs/core > babel-timing > webpack > watchpack > watchpack-chokidar2 > chokidar > fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
warning @redwoodjs/core > svg-react-loader > css > source-map-resolve > resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
warning @redwoodjs/core > @redwoodjs/testing > @storybook/react > @storybook/core > @storybook/core-server > @storybook/builder-webpack4 > webpack-dev-middleware > webpack-log > uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
error babel-jest@27.2.0: The engine "node" is incompatible with this module. Expected version "^10.13.0 || ^12.13.0 || ^14.15.0 || >=15.0.0". Got "14.11.0"
error Found incompatible module.
yarn install v1.22.4
info No lockfile found.
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
    at makeError (/Users/matronator/.npm/_npx/ac33808a139eaad4/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/Users/matronator/.npm/_npx/ac33808a139eaad4/node_modules/execa/index.js:118:26)
    at runMicrotasks ()
    at processTicksAndRejections (internal/process/task_queues.js:93:5) {
  shortMessage: 'Command failed with exit code 1: yarn install',
  command: 'yarn install',
  escapedCommand: '"yarn install"',
  exitCode: 1,
  signal: undefined,
  signalDescription: undefined,
  stdout: 'yarn install v1.22.4\n' +
    'info No lockfile found.\n' +
    '[1/5] Validating package.json...\n' +
    '[2/5] Resolving packages...\n' +
    '[3/5] Fetching packages...\n' +
    'info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.',
  stderr: 'warning @redwoodjs/core > webpack-dev-server > sockjs > uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.\n' +
    'warning @redwoodjs/core > @redwoodjs/cli > @prisma/sdk > temp-write > uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.\n' +
    'warning @redwoodjs/core > webpack-dev-server > url > querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.\n' +
    'warning @redwoodjs/core > @redwoodjs/testing > @storybook/react > @pmmmwh/react-refresh-webpack-plugin > native-url > querystring@0.2.1: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.\n' +
    'warning @redwoodjs/core > @redwoodjs/testing > @storybook/builder-webpack5 > webpack-hot-middleware > querystring@0.2.1: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.\n' +
    'warning @redwoodjs/core > svg-react-loader > css > urix@0.1.0: Please see https://github.com/lydell/urix#deprecated\n' +
    'warning @redwoodjs/core > svg-react-loader > css > source-map-resolve > urix@0.1.0: Please see https://github.com/lydell/urix#deprecated\n' +
    'warning @redwoodjs/core > babel-timing > webpack > watchpack > watchpack-chokidar2 > chokidar@2.1.8: Chokidar 2 will break on node v14+. Upgrade to chokidar 3 with 15x less dependencies.\n' +
    'warning @redwoodjs/core > babel-timing > webpack > watchpack > watchpack-chokidar2 > chokidar > fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.\n' +
    'warning @redwoodjs/core > svg-react-loader > css > source-map-resolve > resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated\n' +
    'warning @redwoodjs/core > @redwoodjs/testing > @storybook/react > @storybook/core > @storybook/core-server > @storybook/builder-webpack4 > webpack-dev-middleware > webpack-log > uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.\n' +
    'error babel-jest@27.2.0: The engine "node" is incompatible with this module. Expected version "^10.13.0 || ^12.13.0 || ^14.15.0 || >=15.0.0". Got "14.11.0"\n' +
    'error Found incompatible module.',
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false,
  context: [Object: null prototype] {}
}

Warning: Directory '/Users/matronator/Documents/Work.nosync/tests/redwoodblog' was created. However, the installation could not complete due to an error.

npm ERR! code 1
npm ERR! path /Users/matronator/Documents/Work.nosync/tests
npm ERR! command failed
npm ERR! command sh -c create-redwood-app "./redwoodblog"

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/matronator/.npm/_logs/2021-10-27T21_22_06_886Z-debug.log
```

So I tried running pnpm install manually in all the folders with a package.json file, which kinda worked, I think, as nothing crashed or errored this time.

But then I couldn't figure out how to run the project without Yarn as pnpm couldn't reach down the tree for the "redwood" command when trying pnpm run redwood dev or just pnpm redwood dev.

thedavidprice commented 3 years ago

@matronator thank you for trying this out and reporting back here! I think your experience verifies what we've been assuming is the next steps on our end:

resolve Yarn warnings across the RW framework packages
resolve CLI bin requirements

I'm not sure if there's a path forward directly from a project, unfortunately.

guilt commented 1 year ago

Is this something being looked at, especially with Bun making it to 1.0?

mwiemer-microsoft commented 1 year ago

Also here because of Bun 1.0, would love to finally escape the slow installs of yarn and npm, ref https://github.com/oven-sh/bun/issues/4207 and Bun - a fast all-in-one JavaScript toolkit for more info, integration seems blocked on changes here and it's been nearly two years since a substantial update :(

thedavidprice commented 1 year ago

@guilt @mwiemer-microsoft

When Bun, as a runtime, can support full node compatibility and, for example, work with Fastify and Pino, it would be very interesting to explore running Redwood on Bun. As of this time, I'm unsure how Bun could even be used to run Redwood's dev server (Fastify and Vite currently).

There's a lot to like about Bun. And I hope it pushes the ecosystem in the right direction. But given that Redwood already solves, out of the box, the challenges of toolchain integration, I'd be curious to better understand what it is that is currently of high enough value to make a swap? And, additionally, if you're aware of what those tradeoffs might be?

If you're about to answer "performance", please do take a look at this excellent writeup by @mcollina: https://buff.ly/3r9Vhm0

Here's an important excerpt:

In fact, pnpm --prefer-offline flag has roughly the same behavior of bun install and it yields performance in the same ballpark...

And, it also just so happens that Yarn 3 is as performant as pnpm (I know, right? Who knew 🤷‍♂️): Package Manager Benchmarks

I know Node performance has been lagging, but I'm very optimistic about Node 20. And I'm very hopeful that Bun will further push Node's overall performance roadmap: https://blog.rafaelgss.dev/state-of-nodejs-performance-2023

🚢 Shippers Gotta Ship

Lastly, I get that Redwood simply might not support the tools you want to use. Or, frankly, support you easily swapping out tools you use when you want to. There are a lot of things we could to better to this end.

But we choose not to. For one reason, we build with Redwood and we've chosen to accept good enough.

For a more important reason, we're investing our limited time and resouces in shipping innovative new features instead of futzing around with tooling. ('Cause, frankly, it's a lot more interesting and we don't really like futzing around with tooling even though we signed up for it as framework authors.)

New Features: Redwood Realtime and Mailer + Studio

If you haven't checked out v6.2 yet, I suggest you do because I hope you'll get excited about Redwood Realtime and Redwood Mailer API + Studio integration.

React SSR, Streaming, and Server Components

We have an experimental version of Redwood SSR + Streaming that's availble. And we created a very fun production app that runs on Fly.io, using Neon.tech Postgres, with Resend.com for emails. You can learn more about that here.

Lastly, we're well into being the second major framework to suport RSC, which you can try as an experimental feature today.

These are the things that are important to us, along with the other items on our Bighorn Epoch roadmap: redwoodjs.com/roadmap

This priorities might not resonate with you and your requirements. I totally get that. But if they do resonate, Redwood is an open source project for a reason — we welcome and would love your help! 🤗

mwiemer-microsoft commented 1 year ago

Thanks @thedavidprice for the thorough response, anything is better than nothing when it comes to longstanding open issues. I'll admit I'm no expert here--I got through half the Redwood tutorial when it was v2, that was 6+ months ago. I was impressed. Bun obviously has some catchy graphs advertising speed and a low-level language behind it that makes those claims almost believable, but I haven't built anything with it yet. I was asking more out of curiosity and hope than anything grounded.

Thanks for clarifying your position here, it makes a lot of sense that the goal of Redwood is basically to be "Bun + backend," and I think you guys are doing a great job. No need to support Bun, really, just wanted an update on this issue in general, and you provided that 😄

Urigo commented 1 year ago

I would love to bring my personal point of view here if that's ok

I think both pnpm and Bun are awesome!! In fact, @the-guild-org thinks they so awesome that all of our tools use and support them for months, in our GraphQL (which Redwood is already using under the hood) and in our HTTP frameworks.

It was easier for us to support Bun thanks to the fact we are building based on standards and @ardatan's great library

@thedavidprice about the article you shared from Matteo - Its true that Bun is not 100% compatible with Node, but I'm not sure it has to be in order to be used for the use case we want Fastify and Pino for.

The Guild has an HTTP framework that works with Bun for a couple of months now, and we are definitely not the only ones.. Again, its thanks to the fact we use a standards based modern approach when it comes to HTTP frameworks, which you can read here on this blog post by @ardatan

Also, Bun install speeds is really not the only or main benefit for us - we've seen crazy gains on runtime performance of our gateways (both GraphQL and HTTP) when using bun in our famous benchmarks. Also compared to the latest Node 20 with the claims from the blog post you've shared https://github.com/the-guild-org/gateways-benchmark https://github.com/the-guild-org/http-server-benchmarks

By the way, on that actual topic, The Guild has moved all of our libraries to use pnpm a while back and would love to support if you want to go aheads with that transition.

I guess my main point - the developer doesn't need to become a prophet and knowing who wins in the future. Choose a framework that is built on standards that work on both Node and Bun and use whatever works for you. That is a direction frameworks should push for

Lastly - I don't think the defensive position maintainers are taking is looking at the feedback in the right way. We love Node and its maintainers and we would keep using and supporting them. We love Redwood and its maintainers and we would keep using and supporting them, like we've done for years! This is just an opportunity for Redwood to become better for the developer, an opportunity we would love to help take and contribute to. And most importantly - The choice of not investing in that right now is completely understandable and doesn't make us think less of these awesome frameworks and their amazing maintainers!

Looks ilke there are some discussions happening on my Twitter comment about that...

AddictArts commented 3 weeks ago

FWIW: Yarn is a full stop no go, can't use it. I am passing on Redwood due to inflexibility in allowing the user to decide on what to use. I can't even check it out due to the need for yarn.

peterp commented 3 weeks ago

@AddictArts Absolutely hear you. The ability to pick a package manager is something that we'll make happen.

redwoodjs / redwood