Support Yarn v2 - Githubissues

eps1lon commented 5 years ago

While yarn v2 is still in development alphas are already released and documented: https://yarnpkg.github.io/berry/

Since I'm a big fan of dependabot and yarn v2 I would like those two to get along better.

For the default configuration of yarn v2 dependabot already updates versions correctly. However, it does not update the PnP file of yarn. This means that in a fresh clone of a project that cannot enable zero-install you create a diff by simply running yarn because that will update the pnp.js. I don't know any dev history of dependabot with regards to yarn but it might make more sense for dependabot to "just" run yarn up which should cover package.json, yarn.lock and .pnp.js.

In addition to that a nice enhancement would be to run yarn cache clean for those who have the offline mirror checked into version control.

I solved both of these issues for me by letting a github action cleanup after dependabot by running yarn and yarn cache clean but that does mean dependabot can't make changes to the PR anymore.

A hardcoded yarn.lock filename might also be problematic in the future since the lockfile name is configurable in yarn v2.

eps1lon commented 4 years ago

Should I spam "Any updates" to get attention or how else can I assist?

qd-qd commented 4 years ago

Any updates ? 😄 Some projects start to migrate his yarn from v1 to v2, a good compatibility will be more and more requested.

rebelagentm commented 4 years ago

Hi! 👋 We're still pretty swamped integrating Dependabot into GitHub, so we haven't yet gotten to this.

mormahr commented 4 years ago

Now that yarn v2 is out, and pnp is enabled by default, maybe the prioritization of this issue should be revisited. We’re in the process of switching to yarn v2 / pnp, but will for the time being not include the pnp file in the repo, because it’ll cause problems with dependabot. We want to change this to use zero-installs, though.

anthonator commented 4 years ago

Agreed that prioritization on this should be revisited. Dependabot is pretty much broken for anyone using Yarn v2 / pnp.

anthonator commented 4 years ago

Is there anything the community can do to help push this along?

eps1lon commented 4 years ago

I push uncommited changes after yarn install on dependabot branches. This means that you have to recreate PRs instead of letting dependabot rebase them. But this isn't much of an issue for me personally.

Using azure pipelines:

trigger:
  - master

pool:
  name: 'Hosted Ubuntu 1604'
  vmImage: 'ubuntu-latest'

steps:
  - checkout: self
    clean: true
    persistCredentials: true

  - task: NodeTool@0
    inputs:
      versionSpec: '10.16.x'
    displayName: 'Install Node.js'

  - script: |
      yarn install
    displayName: 'Install packages'
  - script: |
      git config --global user.email "silbermann.sebastian@gmail.com"
      git config --global user.name "eps1lon[bot]"
      git add -A
      git status
      git diff-index --quiet HEAD || (git commit  --message 'yarn autofix' && git push -u origin HEAD:$(System.PullRequest.SourceBranch))
    # should test the actor but Build.RequestedFor does not point to dependabot but Microsoft.VisualStudio-something
    condition: and(succeeded(), startsWith(variables['System.PullRequest.SourceBranch'], 'dependabot/'))
    displayName: 'Autofix yarn for dependabot'

You could also include dependency deduplication or other autofixes in here.

trulysinclair commented 4 years ago

Same problem here. Working on Yarnberry Cookbook and dependabot breaks yarn.lock. Opening the generated PRs and running yarn throws YAMLException: end of the stream or a document separator is expected at ...:. So I'm guessing D'bot needs to know Yarn 2. As @eps1lon said,

A hardcoded yarn.lock filename might also be problematic in the future since the lockfile name is configurable in yarn v2.

My guess, maybe there should be a lockfile option or detect the configuration from .yarnrc.yml which might be smarter. Another issue is workspaces, it's also breaking my templates. I'm working on a similar approach as @eps1lon with Actions.

koistya commented 4 years ago

I see that Dependabot can resolve updates for Yarn v2 repo, but chokes on local dependencies such as "workspace:*".

See Node.js API Starter Kit (Yarn v2 based monorepo), kriasoft/nodejs-api-starter#215.

AArnott commented 4 years ago

Since Dependabot apparently doesn't work at all in our yarn v2 "Zero Install" repo, I wrote a GitHub Actions workflow that basically does what Dependabot did, but with just one PR for all updates:

name: Update dependencies

on:
  schedule:
  - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - run: yarn up '*'

    - name: Create pull request
      uses: peter-evans/create-pull-request@v3.1.0
      with:
        commit-message: Update all dependencies
        branch: automatic_dependency_updates
        title: Update all dependencies
        body: An updated update of all NPM dependencies.
        #labels: auto-merge
        #reviewers: # optional

cometkim commented 4 years ago

I'm using a similar as @AArnott does, with a simple plugin for using yarn up with more controls

https://github.com/cometkim/yarn-plugin-bump

ylemkimon commented 4 years ago

Similar to https://github.com/dependabot/dependabot-core/issues/1297#issuecomment-621458459, I'm using a GitHub Actions workflow to fix and update PRs created by Dependabot. This way we can still benefit from its version update logic and release notes.

name: Dependabot

on:
  push:
    branches: [ dependabot/npm_and_yarn/** ]

jobs:
  build:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'

    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 2
        persist-credentials: false # minimize exposure
    - name: Use Node.js 12.x
      uses: actions/setup-node@v1
      with:
        node-version: '12'
    - name: Autofix lockfile
      run: |
        # change directory
        # assuming Angular commit style (build: bump XXX from AAA to BBB in YYY)
        # use $8 for default commit message style (Bump XXX from AAA to BBB in YYY)
        cd .`git log -1 --pretty=%s | awk '{ print $9 }'`

        # restore yarn.lock from the previous commit
        git checkout HEAD^ -- yarn.lock

        # install yarn-plugin-deduplicate
        yarn plugin import https://raw.githubusercontent.com/eps1lon/yarn-plugin-deduplicate/latest/bin/%40yarnpkg/plugin-deduplicate.js

        # if package.json was not updated, upgrade the dependency
        # assuming Angular commit style (build: bump XXX from ...)
        # use $2 for default commit message style (Bump XXX from ...)
        git diff --name-only HEAD^ HEAD | grep -q 'package.json' || yarn up `git log -1 --pretty=%s | awk '{ print $3 }'`

        # restore package.json from the last commit
        git checkout HEAD -- package.json

        yarn install

        # deduplicate lockfile
        yarn deduplicate
      env:
        YARN_ENABLE_SCRIPTS: 0 # disable postinstall scripts
    - name: Config Git
      run: |
        # use personal access token to allow triggering new workflow
        BASIC_AUTH=$(echo -n "x-access-token:${{ secrets.GH_TOKEN }}" | base64)
        echo "::add-mask::$BASIC_AUTH"
        git config --global user.name '${{ github.event.commits[0].author.name }}'
        git config --global user.email '${{ github.event.commits[0].author.email }}'
        git config --local http.$GITHUB_SERVER_URL/.extraheader "AUTHORIZATION: basic $BASIC_AUTH"
    - name: Commit changes
      run: |
        cd .`git log -1 --pretty=%s | awk '{ print $9 }'` # ditto
        git add yarn.lock .yarn/cache .pnp.* # only add yarn.lock if not using zero-installs
        git commit -m "Dependabot autofix"
        git push

EDIT (Aug. 13): Updated to support subdirectories. The test repo is available at https://github.com/ylemkimon/berry-dependabot-test. EDIT (Aug. 19): Updated to use personal access token to allow triggering new workflow. Moved Git credentials setup after dependencies install to limit (not remove) exposure.

paul-soporan commented 4 years ago

Since Dependabot apparently doesn't work at all in our yarn v2 "Zero Install" repo, I wrote a GitHub Actions workflow that basically does what Dependabot did, but with just one PR for all updates:

name: Update dependencies

on:
  schedule:
  - cron: '0 2 * * *'
  workflow_dispatch:

jobs:
  update:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2

    - run: yarn up '*'

    - name: Create pull request
      uses: peter-evans/create-pull-request@v3.1.0
      with:
        commit-message: Update all dependencies
        branch: automatic_dependency_updates
        title: Update all dependencies
        body: An updated update of all NPM dependencies.
        #labels: auto-merge
        #reviewers: # optional

FYI: I'd recommend not using this in production (at least for now). yarn up '*' doesn't really do what most people think it does. The way it currently works is that * matches all of the dependencies stored inside the project, and each one of them is then forwarded with range unknown to our suggestUtils.getSuggestedDescriptors function that resolves the unknown range into the latest npm version of that dependency. This means that yarn up '*' will turn all non-npm dependencies into npm dependencies, even when it shouldn't (just like yarn up utils will turn utils into an npm dependency, no matter if it is a git dependency / a portal dependency / whatever else).

Issue to track inside the Yarn 2 repo: https://github.com/yarnpkg/berry/issues/1492.

AArnott commented 4 years ago

Thanks, @paul-soporan. In my case, I only ever use npm dependencies (honestly, I didn't even know git dependencies existed). In one repo I also use a mono-repo where my package.json includes dependencies on other directories in the repo. I guess I should test how those behave in the yarn up '*' world.

bartocc commented 3 years ago

@rebelagentm Would it be possible to have an update on this issue?

rebelagentm commented 3 years ago

@bartocc 👋 I'm no longer working on Dependabot. @feelepxyz, is there an update to provide on this?

feelepxyz commented 3 years ago

@bartocc we started looking into supporting this recently but paused it as it turned out to be a significant change for how Dependabot works. We'll need to start cloning the entire repo instead of just fetching the main manifest files to support the new Plug'n'play/offline cache features. It's going to be a few months out at least as we're focusing on migrating Dependabot Preview features to the GitHub-native version.

bartocc commented 3 years ago

Ok, this is bad news 😢, but at least we know what to expect in the next coming month.

Thx @rebelagentm for forwarding to @feelepxyz 👍

@feelepxyz would you mind posting here some updates as the work on this advances 🚀 or stays idle 🥶 ?

Milo123459 commented 3 years ago

I saw that it'd be a couple months at least before it came out, is this feature on a roadmap? Is there a roadmap?

feelepxyz commented 3 years ago

I saw that it'd be a couple months at least before it came out, is this feature on a roadmap? Is there a roadmap?

Yes! We're focusing the next few months on upgrading all ecosystems in dependabot to the latest version.

juanpicado commented 3 years ago

Does it also include #1736 ? (pnpm support)?

Milo123459 commented 3 years ago

Wow! Can't wait! Also, is there a way for me to unignore a dependency, before I knew you could disable it I removed them all 😂

jurre commented 3 years ago

Does it also include #1736 ? (pnpm support)?

We've currently paused adding new ecosystems, so not in the foreseeable future at least. More context is here: https://github.com/dependabot/dependabot-core/blob/main/CONTRIBUTING.md#contributing-new-ecosystems

jurre commented 3 years ago

Wow! Can't wait! Also, is there a way for me to unignore a dependency, before I knew you could disable it I removed them all 😂

I think this should cover it: https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/configuration-options-for-dependency-updates#ignore

Milo123459 commented 3 years ago

Wow! Can't wait! Also, is there a way for me to unignore a dependency, before I knew you could disable it I removed them all 😂

I think this should cover it: https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/configuration-options-for-dependency-updates#ignore

Thank you, found out you had to reopen the PRs, maybe a way configure ignored dependencies in the dependabot section in insights?

kiancross commented 3 years ago

@feelepxyz Is there anything the community can do to help speed this up?

Milo123459 commented 3 years ago

At this rate, probably just a PR to be honest. Some features require quite a bit of work, also, custom lockfiles is going to be a pain, not to mention identifying it's yarn v2 (actually, you can just check for .yarn/cache directory), but probably the team will get it done sooner or later.

merceyz commented 3 years ago

actually, you can just check for .yarn/cache directory

No, don't do that as it can be custimized and might not exist, check the yarn.lock file for __metadata: instead

Milo123459 commented 3 years ago

Oh, didn't know that.

trulysinclair commented 3 years ago

Really hoping this gets sped up

jdelStrother commented 3 years ago

For the default configuration of yarn v2 dependabot already updates versions correctly

I got the impression from this that if I'm not using PnP, dependabot+yarn2 would work ok, but having tried it this morning I'm pretty sure that's not the case. Dependabot didn't open any new version update PRs, and existing ones got updated with:

Dependabot tried to update this pull request, but something went wrong. We're looking into it, but in the meantime you can retry the update by commenting @dependabot rebase.

Is anyone successfully using dependabot with yarn 2, even if it's not officially supported?

levymetal commented 3 years ago

@jdelStrother Sadly that comment is dated, or perhaps it never worked. Dependabot isn't capable of updating berry lockfiles properly as it uses the yarn 1 cli: https://github.com/dependabot/dependabot-core/blob/main/npm_and_yarn/helpers/lib/yarn/helpers.js

There's PR in progress to add berry support here: https://github.com/dependabot/dependabot-core/pull/2427

amacneil commented 3 years ago

Workaround:

A recent github change made it more difficult to append a simple commit to dependabot PRs updating the yarn.lock file for yarn V2 (breaking the solution by @ylemkimon above).

I updated it and shared the resulting workflow in this gist (live version at https://github.com/foxglove/studio/blob/main/.github/workflows/dependabot-fix.yml). Long story short, it is still possible to "fix" dependabot PRs, as long as you safely use the pull_request_target event.

Note that this workflow must exist on your primary branch before it will work on any PRs. Once this workflow file exists in your primary branch, you can simply comment @dependabot rebase on existing dependabot PRs and they will be updated with a new lockfile.

@github team please consider one more vote for native yarn 2 support in dependabot 🙂

reimer-atb commented 3 years ago

I have a simple react project (created with create-react-app) and upgraded the project to use yarn2.

Everything is working fine, but whenever dependabot opens a PR, my github action workflow fails during the yarn install step - see error below.

The workflow works fine in all other cases (PRs I create myself, etc.). The error only happens when dependabot opens a PR.

Run yarn install
YAMLException: bad indentation of a mapping entry at line 4, column 15:
      dependencies:
                  ^
      dependencies:
                  ^
    at D (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:49:12425)
    at b (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:49:12523)
    at /home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:49:18698
    at U (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:49:18851)
    at T (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:49:25641)
    at j (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:49:26292)
    at G (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:49:26442)
    at e.exports.safeLoad (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:49:26772)
    at f (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:2:526307)
    at I (/home/runner/work/<project>/.yarn/releases/yarn-2.4.1.cjs:2:526645)
Error: Process completed with exit code 1.

asciimike commented 3 years ago

Hey folks,

The short answer is that yarn v2 usage is extremely low (compared to both yarn 1 and npm's various supported versions), so we've put off implementing this feature for the time being. Looks like there are a few workarounds via Actions, as shown above (again, note the use of pull_request_target per the recent security fixes).

To the folks that have asked "what can be done to get this built?", here's what would bump the priority:

significantly increased usage of yarn v2
provide data of significant yarn v2 usage or growth
the deprecation of yarn v1 (I believe it's currently in "maintenance mode" but given current usage, it's far from "deprecated")

Also I want to be clear that this isn't a "we don't like yarn" or "we think npm is better" thing, it's based off npm usage data (which packages are being pulled by which tools), and the data show that yarn v2 just isn't getting the uptake we'd need to see to justify implementation and ongoing maintenance. If/when it gets to the levels seen by other supported versions (or is trending that way), we're happy to implement it.

dbanty commented 3 years ago

yarn v2 just isn't getting the uptake we'd need to see to justify implementation and ongoing maintenance.

I think it's worth pointing out that there's a bit of a chicken & egg problem here. I love Yarn 2 and use it in personal projects, but my company likely won't switch to it without Dependabot support. It's possible that the larger community will move forward with Yarn 2 eventually, but it seems likely that Dependabot users are, like me, waiting for support here before migrating most projects.

I don't have any info on how Dependabot users compare to the larger JS ecosystem but my guess is that we're more likely to want to update to new tech (since clearly we're interested in being notified of available updates).

kiancross commented 3 years ago

@asciimike

The short answer is that yarn v2 usage is extremely low (compared to both yarn 1 and npm's various supported versions)...

Also I want to be clear that this isn't a "we don't like yarn" or "we think npm is better" thing, it's based off npm usage data (which packages are being pulled by which tools)

From what I understand, Yarn 2 is bootstrapped by Yarn 1 and not downloaded from npm (the binary is downloaded straight from GitHub [also seen in the output of yarn set version berry]). So if you are basing this off the 523 weekly downloads for berry shown here, that is probably an underestimate.

Apparently the maintainers collect usage telemetry - I've pinged them on the Yarn Discord - hopefully one of them with access to the numbers can chime in.

fenetic commented 3 years ago

I've had my eye on this thread for a while. We've had to come up with our own way of upgrading our Yarn 2 project using Dependabot and a custom Github action to deal with the lockfile. I know this is a down-votable comment to make but a big reason teams aren't jumping on Yarn 2/PnP is because third parties aren't moving to support it (I'm subbed to a few of these threads across various tooling.) It would make such a huge difference, especially to big monorepo projects.... I appreciate this is going to be a smaller number of users/teams than your primary audience.

ahnpnl commented 3 years ago

@fenetic I came up with the same approach like yours, but I couldn't make it work. Do you mind sharing the way how you made your approach working?

fenetic commented 3 years ago

Hey @ahnpnl -- we have an issue with it currently (and about 10 Dependabot PRs waiting until it's fixed) -- as soon as we can deal with it I'll see if I can link to a gist or something.

-- Edit -- Looks like a workaround was posted above, sorry, missed it -- this is similar to what we're doing, probably worth a go.

kiancross commented 3 years ago

Thanks to @arcanis (Yarn maintainer) for providing additional info in the Yarn Discord. Summarised below:

~4k weekly 2.4.0 users on a fairly linear curve trending up.
This is not comparable to npm downloads, as both CI and automated systems count into npm downloads (but not included in the Yarn telemetry). These probably count for far more than "human" installs.
Some users will have disabled telemetry and therefore are not included in the above figure - at least one large company is known to have done this.

This issue is still in the top 10 upvotes for this repository - so it's not like it's a fringe need.

andreialecu commented 3 years ago

@asciimike note that the metrics you are using to gauge yarn v2 adoption are not comparable to v1 or npm.

which packages are being pulled by which tools

Yarn v2 has a thing called local cache, which is enabled by default.

What that means is that a full .zip snapshot of every package in the dependency tree is part of each project, and users are encouraged to commit it in order to speed up things like CI builds.

More on that here: https://yarnpkg.com/features/zero-installs

Basically, by having the full cache readily available, an install does not need to hit the network at all. So no requests will be made to npm for packages already in the local cache (stored in <proj>/.yarn/cache by default)

This makes the difference between thousands of NPM requests per CI build (with npm or yarn v1), to practically 0 with yarn v2.

Additionally, developers in large organizations will not need to download anything from npm either, if the cache is kept in the repo.

This feature of storing the cache with the repo is one of the biggest selling points of yarn v2, and it dramatically improves performance of cloning repositories, switching branches, etc.

asciimike commented 3 years ago

@kiancross, it's not based on downloads of the tool itself, it's based on "npm package downloads via a tool".

@andreialecu: what you said is super interesting, since it could explain the orders of magnitude decrease in yarn v2 usage compared to other package managers (at least, in the metrics we're tracking).

@arcanis (or @kiancross, since you grabbed numbers last): what's the breakdown and growth of 1.x users vs 2.x users from your telemetry?

merceyz commented 3 years ago

@kiancross, it's not based on downloads of the tool itself, it's based on "npm package downloads via a tool".

@asciimike How are you checking that? Yarn v2 doesn't set any specific header to identify itself

kiancross commented 3 years ago

@arcanis (or @kiancross, since you grabbed numbers last): what's the breakdown and growth of 1.x users vs 2.x users from your telemetry?

There's no equivalent telemetry for 1.x users, unfortunately - it was implemented in 2.x.

jacobwgillespie commented 3 years ago

@ahnpnl / @fenetic I have a currently working workflow adapted from the example that @amacneil posted above (thank you so much for this!) which you can see here:

https://github.com/styled-icons/styled-icons/blob/main/.github/workflows/dependabot.yml

Note that if you wanted to use this, you would need to replace the two instances of BOT_GITHUB_TOKEN with your personal access token secret name.

Edit: Also one note about this line. You will want it to include yarn.lock always, .yarn if you commit the cache to your repo, and .pnp.js if you use PnP (mix and match).

So if you commit the cache and use PnP:

git add .yarn .pnp.js yarn.lock

If you don't commit the cache, and only use Yarn 2 in node-modules mode without PnP:

git add yarn.lock

And so on...

I also set the condition if: github.actor != 'dependabot[bot]' on the regular CI workflow so that it skips running tests for Dependabot's first commit, which always fail with the yarn.lock error anyways. Although this workflow commits the fix using Dependabot's name/email, github.actor on the resulting workflow will be the username of your personal access token, so tests continue running on any subsequent commit.

It would be really great if Dependabot could support Yarn 2 natively, the CI workflow workarounds are roughly functional for my open-source work and for my employer's projects, but it's hackish and has nontrivial security concerns around the handling of access tokens, personal access tokens, postinstall scripts, build scripts, etc. We've considered switching to alternatives like Renovate but don't really want to leave the integrated experience of GitHub/Dependabot (for similar UX and security reasons).

Not sure how relevant a measure of activity this is, but it appears that there are around 3400 open-source repositories that are (1) in GitHub's search index and (2) contain a .yarnrc.yml Yarn 2 config file: https://github.com/search?l=&o=desc&q=filename%3A.yarnrc.yml&s=indexed&type=Code

asciimike commented 3 years ago

@merceyz I believe the User-Agent header has version info. This is the only reference I can find in the berry source (which seems like it will have yarn version info), and I'm working to grab some raw UA strings to confirm.

@jacobwgillespie I did a similar analysis, though I used slightly different criteria:

v1 ~998k files in repos: https://github.com/search?q=%22yarn+lockfile+v1%22+extension%3Alock&type=code
v2 ~2k files in repos: https://github.com/search?q=__metadata+extension%3Alock&type=code

Do we agree that ~3k is a reasonable number of active yarn v2 repos/users (and that it's likely there are ~100k-1M active repos/users on v1)?

merceyz commented 3 years ago

This is the only reference I can find in the berry source (which seems like it will have yarn version info)

@asciimike Indeed but that isn't sent to the registry, it's only set when running scripts

Permalink version so people in the future can still find it: https://github.com/yarnpkg/berry/blob/8e02ba281331c03da8b34843663839fbe4c0febe/packages/yarnpkg-core/sources/scriptUtils.ts#L124-L128

asciimike commented 3 years ago

That's what I was worried about :/

ylemkimon commented 3 years ago

@asciimike

v2 ~2k files in repos: https://github.com/search?q=__metadata+extension%3Alock&type=code

It seems GitHub doesn't index files over 500 KB, at least for lockfiles. For example, https://github.com/search?q=__metadata+extension%3Alock+user%3Ayarnpkg&type=Code doesn't show yarn.lock at yarnpkg/berry.

dependabot / dependabot-core

Support Yarn v2 #1297