Add pnpm support? - Githubissues

GermainBergeron commented 3 years ago

We recently switched our package manager from npm to pnpm since it reduce our install time by multiple minutes in our monorepo. Since then our Snyk scans are failing, as we should have expected. We hacked something to generate a package-lock from the pnpm-lock.yaml but we'd like to have a more robust solution.

Can we start working on a PR to add pnpm support? I saw that you already have a parser for yarn that uses yaml and I think we could reuse some logic.

Thanks!

abdulhannanali commented 3 years ago

I would like to work on this.

GermainBergeron commented 3 years ago

I'm sorry to ping you directly @lirantal but the issue was open 12 days ago and I'm still waiting on an answer

GermainBergeron commented 3 years ago

Again I'm sorry and notifying random contributors but can you have a look at this issue @lili2311 ?

abdulhannanali commented 3 years ago

@GermainBergeron :cry:

lili2311 commented 3 years ago

hi @GermainBergeron apologies for slow response I am not longer directly involved in this plugin, I have however shared this with the relevant team. Is the ask here that you would like to collaborate on the support for the pnpm lockfile?

steph-herd-snyk-pm commented 3 years ago

Hi @GermainBergeron, I am Steph and I am a Product Manager at Snyk. Thank you for your request! Whilst we think this would be a good addition to our current offering due to other priorities we won't be able to work on this anytime soon, sorry about that. I have noted your request down and I will make sure to keep you posted If anything changes.

Kind Regards, Steph

GermainBergeron commented 3 years ago

If I or @abdulhannanali manage to get it working in this repository, would it be something that you would consider @steph-herd-snyk-pm? Is there any other big piece missing to integrate it in the Snyk CLI?

Thanks

lili2311 commented 3 years ago

👋 hi @GermainBergeron @abdulhannanali while we do welcome contributions, in this particular case adding pnpn support would require setting up a new project type in the backend and lots of associated work that comes with it: settings, icons, project type, vulnerabilities, filters. This is why this request would need to be addressed by the team before the any work in the parser can be utilised.

So even if you were to raise the changes needed and they are merged the team would need to complete all the backend work first before the parser is even called for a new project type pnpm.

Could you help us gather some more requirements on this as well while we have you here:

are you using pnpm workspaces at all?
do you need the support in CLI or are scanning it in a different way?

I also wanted to share a current Github action we have that was a great contribution snyk-tech-services/github-actions-pnpm-snyk it can convert the pnpm lockfile to an npm one and then run snyk cli. It might help you in the mean time.

GermainBergeron commented 3 years ago

Hey @lili2311, thanks for your response!

This is why I asked, I figured it could be a lot more complex than just adding the parser in here. To answer your questions:

Yes, we are using pnpm workspaces. We have an open source repository using Snyk and pnpm that you can check if you want but we'd also like to scan some private ones
Yes, we use the CLI, an example would be in the previously mentioned repository

I'll have a look at the Github action, it's probably better than our current workaround 🙏

steph-herd-snyk-pm commented 2 years ago

Thanks, @lili2311! @GermainBergeron I am curious, what is your current workaround? if you don't mind sharing of course. Thank you.

lili2311 commented 2 years ago

@steph-herd-snyk-pm https://github.com/snyk-tech-services/github-actions-pnpm-snyk

GermainBergeron commented 2 years ago

We have a custom script based on the previously mentioned Github action but it's not working currently since we are in a workspace and the package-lock doesn't match the package.json at the root of our workspace (0 dependencies detected).

We're thinking about different improvements but they all seem hackish:

generating a package.json with the dependencies of all our packages while we generate the package-lock.json
adding a postinstall script to generate the package-lock (but dev shouldn't use it)
converting to a yarn lock file while waiting for support of NPM workspaces in Snyk

mathild3r commented 2 years ago

Hi @GermainBergeron,

I am Mathilde, I am working with @lili2311 on this issue. I have a question on your project. So you have a pnpm-lock, a package.json and a pnpm-workspace.yaml at the base of the project, is that right? Do you have pnpm-lock.yml in each packages too?

GermainBergeron commented 2 years ago

Hey Mathilde,

We have a single pnpm-lock.yaml at the root folder along with a pnpm-workspace.yaml and a package.json. Our different packages all have a package.json but no pnpm-lock.yaml. It looks like this:


project
├── packages
│   ├── package-one
│   │   └── package.json
│   ├── package-two
│       └── package.json
├── pnpm-lock.yaml
├── pnpm-workspace.yaml
└── package.json

mathild3r commented 2 years ago

Hi @GermainBergeron

Ok thank you. We are looking at a solution supporting workspaces project atm. I will let you know

mathild3r commented 2 years ago

Hi @GermainBergeron,

We should be releasing a solution supporting workspaces in the next couple of days. The solution we have at the moment doesn't use the CLI but the API, we will release a tool to that use the nodejs-parser to produce a depTree convert it into a depGraph and send it to the API.

Thank you, Mathilde

mathild3r commented 2 years ago

Hi @GermainBergeron

An update on pnpm parser: Last Friday we released nodes-lockfile-parser-ts-fork and snyk-pnpm-depTree-api-tool. The first one build depTree from lock file and the second tool convert deptree into a depGraph send it to the Snyk api and return at the result of the scan.

We will also release a GitHub action later on.

If you wan to try the snyk-pnpm-depTree-api-tool you can install it using :

npm i -g snyk-pnpm-depTree-api-tool

Then to run it with

npm snyk-pnpm-deptree-api-tool -—root ‘yourProjectPath’ —-orgId ‘yourOrganisation’ —-snykToken ‘yoursnykToken’ —-includeDev ‘false’

https://www.npmjs.com/package/snyk-pnpm-deptree-api-tool

Thank you

mathild3r commented 2 years ago

hi,

Here is the github action: https://github.com/snyk-tech-services/snyk-pnpm-github-action

Thank you

GermainBergeron commented 2 years ago

Thanks a lot @mathild3r,

We'll have a look as soon as possible, it seems to fit our use case quite well 🎉

mathild3r commented 2 years ago

Hello @GermainBergeron,

Did you had a chance to try the tool?

Thank you Mathilde

GermainBergeron commented 2 years ago

I have a few issues with the tool, I tested it in two different packages:

Here's the stack trace for the first repository when I use the full path to the repository:

Error: Error: Fail to generate a depGraph RangeError: Maximum call stack size exceeded
    at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:255:11)

The repository doesn't seem that big to me, so I'm a bit confused about why it would break.

If I change the --root argument to . it changes the error to this one, but the file exists. I see that the path is missing the first U:

Error: Error: package.json not found at location: ./sers/germainbergeron/react-vapor/packages/demo/package.json
    at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:255:11)
    at Object.<anonymous> (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:260:3)
    at Module._compile (node:internal/modules/cjs/loader:1092:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1121:10)
    at Module.load (node:internal/modules/cjs/loader:972:32)
    at Function.Module._load (node:internal/modules/cjs/loader:813:14)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:76:12)
    at node:internal/main/run_main_module:17:47

The second one is a private repository which is a bit more complex (we have packages inside a packages folder, but also inside a build and core folders). There are more packages than in the first one with more dependencies

Error: Error: Fail to generate a depGraph TypeError: Cannot read property 'dependencies' of undefined
    at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:255:11

Side notes:

The readme of the repo is using snyk-pnpm-dephtree-api-tool instead of snyk-pnpm-deptree-api-tool
Also, the --help command does not work
Is the repository for snyk-pnpm-deptree-api-tool open source?

mathild3r commented 2 years ago

Hi Germain,

Thank you for your email.

I will have a look at the issues shortly. In the meantime can you tell me what you put in the --root option to get the this error: Error: Error: Fail to generate a depGraph RangeError: Maximum call stack size exceeded at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:255:11)

For the second repo, is look like it's a workspace project, would you be able to share the pnpm-workspace.yaml? the tool probably haven't found one of the files.

The tool is not open source at the moment, I am sorry, but I will discuss it with the team.

Thank you, Mathilde

GermainBergeron commented 2 years ago

For the root option I tried multiple things, but I got the call stack size exceeded with --root /Users/germainbergeron/react-vapor which is the folder of my Github repository

The pnpm-workspace.yaml look like this for the second repository:

packages:
  - 'build/*'
  - 'packages/*'
  - 'core/*'
  - 'packages/package-a/cypress/'
  - 'packages/package-b/cypress/'

mathild3r commented 2 years ago

Hi Germain,

Apologies for the late answer, I had to look at something else.

I look at your issue a bit closer today and it looks like we are stuck in a loop: webpack: 4.41.6 -> /@webassemblyjs/ast/1.8.5 -> /@webassemblyjs/helper-module-context/1.8.5 -> /@webassemblyjs/ast/1.8.5

I will let you know when I have a fix.

Thank you Mathilde

mathild3r commented 2 years ago

Hi Germain,

Apologies for the delay. I have released a new version of the parser and of the api tool for you to try : https://www.npmjs.com/package/snyk-pnpm-deptree-api-tool

Let me know,

Thank you, Mathilde

GermainBergeron commented 2 years ago

Hey Mathilde,

I tried running the tool with different configurations of my repository but I always get

Error: Error: Fail to generate a depGraph TypeError: Cannot read property 'devDependencies' of undefined
    at main (/path/to/node_modules/snyk-pnpm-deptree-api-tool/src/lib/index.ts:291:11)

Since the repository is open source, can you try to run the tool on it?

mathild3r commented 2 years ago

Hi,

yes I will and let you know mid next week.

Thank you Mathilde

mathild3r commented 2 years ago

Hi Germain,

I would like to apologies for the delay to resolve your problems. The problem here is that our solution was not able to find the linked package in other packages: ie: packages : -demo -vapor helpers: -enzyme-redux

it would find the ones linked that are under packages (ie: demo linked to vapor) but if vapor linked to helpers/enzym-redux it not capable of finding it.

I am working on a solution and will update you before the end of the week.

Thank you, Mathilde

mathild3r commented 2 years ago

Hello,

We have just released a new version of the tool (tested on your full repo): https://www.npmjs.com/package/snyk-pnpm-deptree-api-tool

Let me know what you think. Thank you, Mathilde

GermainBergeron commented 2 years ago

Hello @mathild3r,

Sorry for the late reply, I had a lot of other things on my plate. I tried the latest version (and previous) of the tool and it seems to run but it take ages (100% of CPU for more multiple hours) which is a big blocker for us. Did you have similar performances on your side?

When it finally ends, I get a 404 error on the endpoint /api/v1/test/dep-graph endpoint. Is that private?

I added a bit of debugging info and it seems to try to build a dependency graph of all the dependencies, including those in the node_modules folder of all my packages which I think it is unnecessary since they are dependencies of my package and are already in the dependency graph

mathild3r commented 2 years ago

Hello,

Thank you for trying the tool.

It does take ages yes and unfornately this is the best I can do. This solution is a workaround and not is not fully embedded in the product. But please raise this with the CSM, so we can add 'full pnpm support' in the product roadmap.

For the 404 error, this is probably because you need to ask your CSM to enable the Feature Flag ‘ExperimentalDepGraph’ to enable access to that endpoint.

Thank you, Mathilde

On 14 Feb 2022, at 16:52, Germain Bergeron @.***> wrote:

Hello @mathild3r https://github.com/mathild3r,

Sorry for the late reply, I had a lot of other things on my plate. I tried the latest version (and previous) of the tool and it seems to run but it take ages (100% of CPU for more multiple hours) which is a big blocker for us. Did you have similar performances on your side?

When it finally ends, I get a 404 error on the endpoint /api/v1/test/dep-graph endpoint. Is that private?

I added a bit of debugging info and it seems to try to build a dependency graph of all the dependencies, including those in the node_modules folder of all my packages which I think it is unnecessary since they are dependencies of my package and are already in the dependency graph

— Reply to this email directly, view it on GitHub https://github.com/snyk/nodejs-lockfile-parser/issues/111#issuecomment-1039316124, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATUR2EZNC4BZXSMLGIQIUXDU3EXLJANCNFSM442U75SA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you were mentioned.

christoffee commented 2 years ago

Hey, i just wanted to follow up...

was there any update on this ?
Are we still meant to use the workaround ?
has 'full pnpm support' been added to the road map ?

Thanks

Chris

amirgalor commented 2 years ago

@GermainBergeron @christoffee I've also just encountered this unexpected constraint, and I have a workaround, not a beautiful one, but a working one - make sure both lock files (npm & pnpm) exist and are synced

When can they get out of sync, when a new package is installed - for that i've created a script (add-package.js) that installs the package and keeps the sync.

const execSync = require('child_process').execSync;

const arg = process.argv[2];
execSync('echo "- Adding package to npm (no install)"', { stdio: [0, 1, 2] });
execSync(`npm install ${arg} --package-lock-only --audit=false --save-exact`, { stdio: [0, 1, 2] });
execSync('echo "Importing changes to pnpm"', { stdio: [0, 1, 2] });
execSync(`pnpm import`, { stdio: [0, 1, 2] });
execSync('echo "Task Done"');

adding an npm script: "add": "node ./add-package.js"

and to call: npm run add react-query or pnpm add react-query

mathild3r commented 2 years ago

Hey, i just wanted to follow up...

was there any update on this ?

Are we still meant to use the workaround ?

has 'full pnpm support' been added to the road map ?

Thanks

Chris

Hello @christoffee,

I am Mathilde from the Technical Service Team at Snyk, Snyk doesn't pnpm support at the moment and it is not in the product roadmap for now so you still need to use this workaround for pnpm support with Snyk. We will keep you updated when this changes.

Thank you, Mathilde

milahu commented 2 years ago

possible alternative to snyk-nodejs-lockfile-parser:

my parse-package-lock is a generic lockfile parser for npm, yarn, pnpm to provide a generic interface to @npmcli/arborist, @yarnpkg/parsers, @pnpm/lockfile-file

it does not return a tree of dependencies its a stream parser that calls

eventHandlers.enterPackage(packageData);
// recurse into dependencies of this package
eventHandlers.leavePackage(packageData);

i hope to replace snyk-nodejs-lockfile-parser with parse-package-lock in my pnpm-install-only which is a generic package installer for npm, yarn, (not yet pnpm) using the filesystem layout of pnpm to create a deep tree

similar project: antongolub/lockfile

edit: rename from parselock to parse-package-lock, add description

urbanisierung commented 1 year ago

@mathild3r Is there any update to the product roadmap planned? Since monorepos and pnpm are quite popular it would be very nice to see pnpm on the roadmap.

urbanisierung commented 1 year ago

@GermainBergeron did you find another solution? I'm trying the workaround tool, already running since ages (unfortunately without any feedback about progress). How long does/did it take for your project?

GermainBergeron commented 1 year ago

@urbanisierung Sorry for the late reply. On my machine IIRC it ran for about 3h before I canceled the job. I did try to optimize the file by skipping building the tree of my dependencies but when I ran that version it then failed when poking the API since we didn't have the flag active on our organization.

Our workaround still consist of generating a package-lock.json from our pnpm lockfile

traviscrist commented 1 year ago

Is the current solution to this to use https://github.com/snyk-tech-services/snyk-pnpm-github-action which is archived or is there another workaround?

damiangreen commented 1 year ago

What is the workaround exactly? We use gitlab not github

milahu commented 1 year ago

funny how nobody has solved this simple issue

let collaborate?

my plan is ...

add lockfile types to pnpm
started in https://github.com/pnpm/pnpm/pull/4521
- validate these typess with a sample of pnpm lockfiles
  github search: pnpm-lock.yaml shrinkwrap.yaml
from these types, derive validators
parse lockfile with some yaml parser
validate lockfile data, based on the lockfile version
normalize the lockfile data to the latest lockfile version
maybe normalize the lockfile data to the internal data format of snyk/nodejs-lockfile-parser

most work are steps 5 and 6

step 5 is optional, a minimal solution only needs support for the latest lockfile version

my draft for step 1 is in https://github.com/milahu/nodejs-lockfile-parser/tree/parse-pnpm-lockfiles

ideally, we would use some "semantic stage code generator" to translate between different schemas (schema migration, AST transformer) but realistically, we will write the semantic stage by hand

my use case: a generic npm installer that can consume many types of lockfiles (npm, yarn, pnpm, ...)

weyert commented 1 year ago

What is the workaround exactly? We use gitlab not github

Your not using Gemnasium for the vulnerability scanning?

nVitius commented 1 year ago

@mathild3r Do you know why this repo was archived? snyk-tech-services/snyk-pnpm-github-action

sanderkooger commented 1 year ago

@mathild3r I just wanted you to know that we have decided to drop snyk in our clients projects because of the lack of monorepo support. If Snyk stops innovating, thats fine, but we can't stop innovating for our clients.

enrique-ramirez commented 1 year ago

I am here adding my two cents. The lack of pnpm support and monorepos is quite dissapointing. I hope voicing our concerns can help push the priorities a bit further top, @mathild3r ! Thanks for the hard work!

vasuneet commented 1 year ago

I agree. We recently migrated from yarn to pnpm and our vulnerability scanner is not supporting pnpm lock file. So I am looking to write a script to generate yarn.lock from pnpm-lock file. So I need pnpm parser to do that.

sanderkooger commented 1 year ago

The issue is do we really want to build and maintain things ourselves, while this should just be a plug and play thing? I mean the fact that it's no rocket science, is the primary reason this should be a basic feature right?

I believe Snyk should support all node package managers, as their bread and butter is to sell a product that scans our code. But I might be wrong here :D

To try to put this basic functionality on their roadmap, could you all be so kind as to give a :thumbsup:

@enrique-ramirez, @vasuneet, @nVitius, @weyert, @GermainBergeron, @abdulhannanali, @lili2311, @steph-herd-snyk-pm, @mathild3r, @amirgalor, @milahu, @urbanisierung, @traviscrist, @damiangreen

milahu commented 1 year ago

it's no rocket science

yes. parse yaml, loop objects, parse strings, generate dependency graph

Snyk should

"burger king should sell big macs"...

if you need it, then you "should" write it. some pointers:

or, you know, pay someone else on a freelance website like Upwork, Guru, Freelancer, Fiverr ... i guess this feature should be doable for 50 to 100 usd

weyert commented 1 year ago

I think it's easier for Snyk to support SPDX or SBOM [1] and then let it in ingest one of those standards. In such manner any tool that can generate those files will be supported.

@milahu Yups, implemented three times now but I don't use Snyk myself so not incentive for me. I made PRs for LicenseFinder, PNPM (for license command), Gemnasium [2] to add support for PNPM which allows to generate vulnerability scan report and generate the SBOM format

[1] https://cyclonedx.org/use-cases/#license-compliance [2] https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/merge_requests/455

weyert commented 1 year ago

Looks like you can give Snyk a SBOM file here at https://snyk.io/code-checker/sbom-security/ and then it reports the vulnerabilities. I think that would solve most of the people's problems here.

If anyone is interested in sponsoring (or donate to an Alzheimer's charity) this work at PNPM. Let use now, happy to implement SBOM export format as part of like the pnpm licenses list-command

snyk / nodejs-lockfile-parser

Add pnpm support? #111