[Bug]: Unable to install packages from CodeSandbox

merceyz commented 3 years ago

Self-service

[X] I'd be willing to implement a fix

Describe the bug

Package URLs from CodeSandbox doesn't include tgz so Yarn is unable to install them

Originally reported by @phryneas

To reproduce

const installPromise = packageJsonAndInstall({
    dependencies: {
        [`@reduxjs/toolkit`]: `https://pkg.csb.dev/reduxjs/redux-toolkit/commit/7491320d/@reduxjs/toolkit`,
    },
});

await expect(installPromise).resolves.toBeTruthy();

Environment

  System:
    OS: Windows 10 10.0.19042
    CPU: (16) x64 AMD Ryzen 7 3700X 8-Core Processor
  Binaries:
    Node: 16.7.0 - ~\AppData\Local\Temp\xfs-d3717859\node.CMD
    Yarn: 3.0.1-git.20210819.hash-32881f98b - ~\AppData\Local\Temp\xfs-d3717859\yarn.CMD
    npm: 7.20.1 - C:\nodejs\npm.CMD

Additional context

No response

yarnbot commented 3 years ago

This issue reproduces on master:

Error: expect(received).resolves.toBeTruthy()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js install

➤ YN0000: ┌ Resolution step
::group::Resolution step
➤ YN0001: │ Error: @reduxjs/toolkit@https://pkg.csb.dev/reduxjs/redux-toolkit/commit/7491320d/@reduxjs/toolkit isn't supported by any available resolver
    at MultiResolver.getResolverByDescriptor (/github/workspace/packages/yarnpkg-core/sources/MultiResolver.ts:73:13)
    at MultiResolver.bindDescriptor (/github/workspace/packages/yarnpkg-core/sources/MultiResolver.ts:31:27)
    at startPackageResolution (/github/workspace/packages/yarnpkg-core/sources/Project.ts:719:32)
    at async Promise.all (index 0)
    at Project.resolveEverything (/github/workspace/packages/yarnpkg-core/sources/Project.ts:797:7)
    at /github/workspace/packages/yarnpkg-core/sources/Project.ts:1402:7
    at StreamReport.startTimerPromise (/github/workspace/packages/yarnpkg-core/sources/StreamReport.ts:309:14)
    at Project.install (/github/workspace/packages/yarnpkg-core/sources/Project.ts:1401:5)
    at /github/workspace/packages/plugin-essentials/sources/commands/install.ts:318:7
    at Function.start (/github/workspace/packages/yarnpkg-core/sources/StreamReport.ts:131:7)
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 103ms
]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-0c0da74930.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:8:7)
    at /github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:19
    at executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:22)
    at Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:18)
    at ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:59)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)

phryneas commented 3 years ago

For context: this is an example CodeSandbox CI build: https://ci.codesandbox.io/status/reduxjs/redux-toolkit/pr/1277/builds/161299

ylemkimon commented 3 years ago

A workaround would be to use a url hash: yarn add @reduxjs/toolkit@https://pkg.csb.dev/reduxjs/redux-toolkit/commit/7491320d/@reduxjs/toolkit#.tgz.

yarnbot commented 3 years ago

This issue reproduces on master:

Error: expect(received).resolves.toBeTruthy()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js install

➤ YN0000: ┌ Resolution step
::group::Resolution step
➤ YN0001: │ Error: @reduxjs/toolkit@https://pkg.csb.dev/reduxjs/redux-toolkit/commit/7491320d/@reduxjs/toolkit isn't supported by any available resolver
    at MultiResolver.getResolverByDescriptor (/github/workspace/packages/yarnpkg-core/sources/MultiResolver.ts:73:13)
    at MultiResolver.bindDescriptor (/github/workspace/packages/yarnpkg-core/sources/MultiResolver.ts:31:27)
    at startPackageResolution (/github/workspace/packages/yarnpkg-core/sources/Project.ts:719:32)
    at async Promise.all (index 0)
    at Project.resolveEverything (/github/workspace/packages/yarnpkg-core/sources/Project.ts:797:7)
    at /github/workspace/packages/yarnpkg-core/sources/Project.ts:1402:7
    at StreamReport.startTimerPromise (/github/workspace/packages/yarnpkg-core/sources/StreamReport.ts:309:14)
    at Project.install (/github/workspace/packages/yarnpkg-core/sources/Project.ts:1401:5)
    at /github/workspace/packages/plugin-essentials/sources/commands/install.ts:318:7
    at Function.start (/github/workspace/packages/yarnpkg-core/sources/StreamReport.ts:131:7)
::endgroup::
➤ YN0000: └ Completed
➤ YN0000: Failed with errors in 0s 99ms
]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-0c0da74930.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:8:7)
    at /github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:19
    at executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:22)
    at Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:18)
    at ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:59)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)

yarnbot commented 3 years ago

This issue reproduces on master:

Error: expect(received).resolves.toBeTruthy()

Received promise rejected instead of resolved
Rejected to value: [Error: Command failed: /usr/bin/node /github/workspace/scripts/actions/../run-yarn.js install

➤ YN0000: ┌ Resolution step
::group::Resolution step
➤ YN0001: │ Error: @reduxjs/toolkit@https://pkg.csb.dev/reduxjs/redux-toolkit/commit/7491320d/@reduxjs/toolkit isn't supported by any available resolver
    at MultiResolver.getResolverByDescriptor (/github/workspace/packages/yarnpkg-core/sources/MultiResolver.ts:73:13)
    at MultiResolver.bindDescriptor (/github/workspace/packages/yarnpkg-core/sources/MultiResolver.ts:31:27)
    at startPackageResolution (/github/workspace/packages/yarnpkg-core/sources/Project.ts:719:32)
    at async Promise.all (index 0)
    at Project.resolveEverything (/github/workspace/packages/yarnpkg-core/sources/Project.ts:797:7)
    at /github/workspace/packages/yarnpkg-core/sources/Project.ts:1402:7
    at StreamReport.startTimerPromise (/github/workspace/packages/yarnpkg-core/sources/StreamReport.ts:309:14)
    at Project.install (/github/workspace/packages/yarnpkg-core/sources/Project.ts:1401:5)
    at /github/workspace/packages/plugin-essentials/sources/commands/install.ts:318:7
    at Function.start (/github/workspace/packages/yarnpkg-core/sources/StreamReport.ts:131:7)
::endgroup::
➤ YN0000: └ Completed in 0s 203ms
➤ YN0000: Failed with errors in 0s 226ms
]
    at expect (/github/workspace/.yarn/cache/expect-npm-24.8.0-8c7640c562-0c0da74930.zip/node_modules/expect/build/index.js:138:15)
    at module.exports (evalmachine.<anonymous>:8:7)
    at /github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:56:19
    at executeInTempDirectory (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:17:22)
    at Object.executeRepro (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/executeRepro.js:24:18)
    at ExecCommand.execute (/github/workspace/.yarn/cache/@arcanis-sherlock-npm-2.0.2-91650a2501-627bee24a7.zip/node_modules/@arcanis/sherlock/lib/commands/exec.js:25:59)
    at async ExecCommand.validateAndExecute (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Command.js:161:26)
    at async Cli.run (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:74:24)
    at async Cli.runExit (/github/workspace/.yarn/cache/clipanion-npm-2.0.0-rc.16-b9444aaf89-91cf93ba72.zip/node_modules/clipanion/lib/advanced/Cli.js:83:28)

Andarist commented 2 years ago

Until CSB fixes how they are generating package.json manifests, this plugin can be used as a workaround:

module.exports = {
  name: 'fix-csb-deps',
  factory: (require) => {
    const crypto = require('crypto')

    // https://github.com/yarnpkg/berry/blob/5411f76bcd89d1d6f430f4bd0e16146ce9fdd370/packages/yarnpkg-core/sources/hashUtils.ts#L5-L26
    function makeHash(...args) {
      const hash = crypto.createHash(`sha512`)

      let acc = ``
      for (const arg of args) {
        if (typeof arg === `string`) {
          acc += arg
        } else if (arg) {
          if (acc) {
            hash.update(acc)
            acc = ``
          }

          hash.update(arg)
        }
      }

      if (acc) hash.update(acc)

      return hash.digest(`hex`)
    }

    return {
      hooks: {
        reduceDependency(dependency, project, locator, initialDependency, { resolver, resolveOptions }) {
          if (dependency.range.startsWith('https://pkg.csb.dev/') && !dependency.range.endsWith('/_pkg.tgz')) {
            const newRange = `${dependency.range}/_pkg.tgz`
            const fixedDescriptor = {
              identHash: dependency.identHash,
              scope: dependency.scope,
              name: dependency.name,
              descriptorHash: makeHash(dependency.identHash, newRange),
              range: newRange,
            }
            return fixedDescriptor
          }
          return dependency
        },
      },
    }
  },
}

yarnpkg / berry