oxsecurity / megalinter

🦙 MegaLinter analyzes 50 languages, 22 formats, 21 tooling formats, excessive copy-pastes, spelling mistakes and security issues in your repository sources with a GitHub Action, other CI tools or locally.
https://megalinter.io
GNU Affero General Public License v3.0
1.81k stars 215 forks source link

npm install not work in gitlab ci with mega-linter V7 #3111

Open sharh opened 7 months ago

sharh commented 7 months ago

Describe the bug when use eslint, need to install the dependencies first, because in the soure code, some lib needs: in nextjs layout.tsx

import type { Metadata } from "next";
export const metadata: Metadata = {
  title: "Next.js",
  description: "The React Framework for the Web",
};
export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <meta
        name="viewport"
        content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no"
      />
      <body>{children}</body>
    </html>
  );
}

when run eslint, it needs not just eslint plugins, but also lib deps. Thus I need to install the full deps in package.json

To Reproduce Here is the .mega-linter.yml

---
APPLY_FIXES: all
ENABLE_LINTERS:
  - TSX_ESLINT
  - JSON_PRETTIER
  - HTML_HTMLHINT
  - GRAPHQL_GRAPHQL_SCHEMA_LINTER
  - ENV_DOTENV_LINTER
  - CSS_STYLELINT
  - CSS_SCSS_LINT
  - YAML_PRETTIER
  - YAML_YAMLLINT
  - DOCKERFILE_HADOLINT
  - EDITORCONFIG_EDITORCONFIG_CHECKER
  - COPYPASTE_JSCPD
  - REPOSITORY_GIT_DIFF
PRE_COMMANDS:
  - command: npm install
    cwd: workspace
    secured_env: true
TSX_ESLINT_PRE_COMMANDS:
  - command: pwd
    cwd: workspace
  - command: npm install
    cwd: workspace
SHOW_ELAPSED_TIME: true
FILEIO_REPORTER: false
FILTER_REGEX_EXCLUDE: '(\.git/|\.husky/|\.vscode/|node_modules/|tsconfig\.json|\.d\.ts)'
CSS_STYLELINT_FILE_EXTENSIONS:
  - .css
  - .scss
  - .less
  - .sass
  - .postcss
TSX_ESLINT_FILE_EXTENSIONS:
  - .js
  - .cjs
  - .mjs
  - .jsx
  - .ts
  - .mts
  - .cts
  - .tsx
EDITORCONFIG_EDITORCONFIG_CHECKER_FILTER_REGEX_EXCLUDE: (\.json)
TEXT_REPORTER: true
TEXT_REPORTER_SUB_FOLDER: linters_logs
CONSOLE_REPORTER: true
CONSOLE_REPORTER_SECTIONS: true
OUTPUT_DETAIL: detailed
GITLAB_COMMENT_REPORTER: true
GITHUB_STATUS_REPORTER: true
EDITORCONFIG_EDITORCONFIG_CHECKER_DISABLE_ERRORS: true
TSX_ESLINT_CLI_LINT_MODE: project

As you can see, here is the PRE_COMMANDS

PRE_COMMANDS:
  - command: npm install
    cwd: workspace
    secured_env: true
TSX_ESLINT_PRE_COMMANDS:
  - command: pwd
    cwd: workspace
  - command: npm install
    cwd: workspace

It's failed: image image here is the package.json file in project

{
  "name": "next-template",
  "version": "1.0.0",
  "scripts": {
    "dev": "cross-env BUILD_ENV=development next dev --turbo",
    "build-test": "cross-env BUILD_ENV=test next build",
    "build-prod": "cross-env BUILD_ENV=prod next build",
    "start": "cross-env NODE_ENV=production next start -p 8888",
    "lint": "eslint --fix . "
  },
  "author": "",
  "license": "MIT",
  "devDependencies": {
    "@commitlint/cli": "^18.4.0",
    "@commitlint/config-conventional": "^18.4.0",
    "@next/eslint-plugin-next": "^14.0.2",
    "@styled/typescript-styled-plugin": "^1.0.0",
    "@types/node": "^20.9.0",
    "@types/react": "^18.2.37",
    "@types/react-dom": "^18.2.15",
    "@typescript-eslint/eslint-plugin": "^6.10.0",
    "@typescript-eslint/parser": "^6.10.0",
    "cross-env": "^7.0.3",
    "eslint": "^8.53.0",
    "eslint-config-prettier": "^9.0.0",
    "eslint-import-resolver-typescript": "^3.6.1",
    "eslint-plugin-import": "^2.29.0",
    "eslint-plugin-react": "^7.33.2",
    "eslint-plugin-react-hooks": "^4.6.0",
    "eslint-plugin-yml": "^1.10.0",
    "husky": "^8.0.3",
    "is-ci": "^3.0.1",
    "jscpd": "^3.5.10",
    "lint-staged": "^15.0.2",
    "postcss": "^8.4.31",
    "postcss-html": "^1.5.0",
    "postcss-less": "^6.0.0",
    "prettier": "^3.1.0",
    "stylelint": "^15.11.0",
    "stylelint-config-rational-order": "^0.1.2",
    "stylelint-config-sass-guidelines": "^10.0.0",
    "stylelint-config-standard": "^34.0.0",
    "stylelint-config-standard-scss": "^11.1.0",
    "typescript": "^5.2.2",
    "typescript-eslint-language-service": "^5.0.5",
    "validate-branch-name": "^1.3.0",
    "yaml-eslint-parser": "^1.2.2"
  },
  "config": {
    "commitizen": {
      "path": "@commitlint/cz-commitlint"
    }
  },
  "dependencies": {
    "next": "^14.0.2",
    "react": "^18.2.0",
    "react-dom": "^18.2.0"
  }
}

All the deps are declared and worked well in local. Expected behavior

After npm install the linter can work well when use eslint

Additional context When i use the sample config to install single plugin, it works.

PRE_COMMANDS:
  - command: npm install @next/eslint-plugin-next
    cwd: "root"        # Will be run at the root of MegaLinter docker image

It seems that ESLINT works in root, not work well in workspace.

How ever, there is a lot of deps in package.json, if all list in PRE_COMMANDS like above, that is silly. And when you add new deps in package,json, you need to update PRE_COMMANDS,

log info

Running with gitlab-runner 16.3.0 (8ec04662)
   on 
Preparing the "docker" executor
 Using Docker executor with image oxsecurity/megalinter-javascript:v7 ...
 Pulling docker image oxsecurity/megalinter-javascript:v7 ...
 Using docker image sha256:870e90b8ed799e8c82c266dfd80b8fda862577f9085ddc1dbb9540fc8922[3]

Preparing environment
00:01
 Running on runner-43ylhtju-project-913-concurrent-0 via it-sonarqube...
Getting source from Git repository
00:00
 Fetching changes with git depth set to 50...
 Reinitialized existing Git repository in /builds/fe/template/next/.git/
 Checking out 326f41b7 as detached HEAD (ref is master)...
 Removing megalinter-reports/
 Removing node_modules/
 Skipping Git submodules setup
Executing "step_script" stage of the job script
 Using docker image sha256:870e90b8ed799e8c82c266dfd80b8fda862577f9085ddc1dbb9540fc89223abe for oxsecurity/megalinter-javascript:v7 with digest oxsecurity/megalinter-javascript@sha256:f5758a6eb9f439ea974258f35741d6f977b5ed7ccd31f805c6490d3177f864a5 ...
 Setting git safe.directory DEFAULT_WORKSPACE: /builds/fe/template/next ...
 Setting git safe.directory default: /github/workspace ...
 Setting git safe.directory to /tmp/lint ...
 [MegaLinter init] ONE-SHOT RUN
 [config] /builds/fe/template/next/.mega-linter.yml + Environment variables
     .:oool'                                  ,looo;                           
     .xNXNXl                                 .dXNNXo.                          
      lXXXX0c.                              'oKXXN0;                           
      .oKNXNX0kxdddddddoc,.    .;lodddddddxk0XXXX0c                            
       .:kKXXXXXXXXXXXXNXX0dllx0XXXXXXXXXXXXXXXKd,                             
         .,cdkOOOOOOOO0KXXXXXXXXXXK0OOOOOOOkxo:'                               
                       'ckKXNNNXkc'                                            
               ':::::;.  .c0XX0l.  .;::::;.                                    
               'xXXXXXx'   :kx:   ;OXXXXKd.                                    
                .dKNNXXO;   ..   :0XXXXKl.                                     
                 .lKXXXX0:     .lKXXXX0:                                       
                   :0XXXXKl.  .dXXXXXk,                                        
                    ;kXXXXKd:cxXXXXXx'                                         
                     'xXNXXXXXXXXXKo.                                          
                      .oKXXXXNXXX0l.                                           
                       .lKNNXNNXO:                                             
                         ,looool'                                              
 ==========================================================
 =============   MegaLinter, by OX.security   =============
 =========  https://ox.security?ref=megalinter  ===========
 ==========================================================
 ----------------------------------------------------------------------------------------------------
 ------------------------------------ MegaLinter, by OX Security ------------------------------------
 ----------------------------------------------------------------------------------------------------
  - Image Creation Date: 2023-10-28T23:01:51Z
  - Image Revision: b48455a119cc28045eee8f1e9d0a542a85e[7]
  - Image Version: v7.5.0
 ----------------------------------------------------------------------------------------------------
 The MegaLinter documentation can be found at:
  - https://megalinter.io/7.5.0
 ----------------------------------------------------------------------------------------------------
 MegaLinter initialization (expand for details)
 MegaLinter will analyze workspace [/builds/fe/template/next]
 [Pre] run: [npm install] in cwd [/builds/fe/template/next]
 [Pre] result:
 added 23 packages, and audited 24 packages in 1m
 3 packages are looking for funding
   run `npm fund` for details
 found 0 vulnerabilities
...
...
...

✅ Linted [COPYPASTE] files with [jscpd] successfully - (0.97s) (expand for details)
 - Using [jscpd v3.5.10] https://megalinter.io/7.5.0/descriptors/copypaste_jscpd
 - MegaLinter key: [COPYPASTE_JSCPD]
 - Rules config: [/builds/fe/template/next/.jscpd.json]
 --Log detail:
 ┌────────────┬────────────────┬─────────────┬──────────────┬──────────────┬──────────────────┬───────────────────┐
 │ Format     │ Files analyzed │ Total lines │ Total tokens │ Clones found │ Duplicated lines │ Duplicated tokens │
 ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
 │ javascript │ 6              │ 251         │ 1151         │ 0            │ 0 (0%)           │ 0 (0%)            │
 ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
 │ tsx        │ 1              │ 26          │ 157          │ 0            │ 0 (0%)           │ 0 (0%)            │
 ├────────────┼────────────────┼─────────────┼──────────────┼──────────────┼──────────────────┼───────────────────┤
 │ Total:     │ 7              │ 277         │ 1308         │ 0            │ 0 (0%)           │ 0 (0%)            │
 └────────────┴────────────────┴─────────────┴──────────────┴──────────────┴──────────────────┴───────────────────┘
 Found 0 clones.
 HTML report saved to /builds/fe/template/next/megalinter-reports/copy-paste/html/
 Detection time:: 245.622ms
 Unable to get number of errors with regex_sum and ✖ ([0-9]+) problem
 ❌ Linted [TSX] files with [eslint]: Found 1 error(s) - (4.08s) (expand for details)
 - Using [eslint v8.52.0] https://megalinter.io/7.5.0/descriptors/tsx_eslint
 - MegaLinter key: [TSX_ESLINT]
 - Rules config: identified by [eslint]
 [Pre][TSX_ESLINT] run: [pwd] in cwd [/builds/fe/template/next]
 [Pre][TSX_ESLINT] result:
 /builds/fe/template/next
 [Pre][TSX_ESLINT] run: [npm install] in cwd [/builds/fe/template/next]
 [Pre][TSX_ESLINT] result:
 up to date, audited 24 packages in 2s
 3 packages are looking for funding
   run `npm fund` for details
 found 0 vulnerabilities
 --Error detail:
 Oops! Something went wrong! :(
 ESLint: 8.52.0
 ESLint couldn't find the plugin "@next/eslint-plugin-next".
 (The package "@next/eslint-plugin-next" was not found when loaded as a Node module from the directory "/builds/fe/template/next".)
 It's likely that the plugin isn't installed correctly. Try reinstalling by running the following:
     npm install @next/eslint-plugin-next@latest --save-dev
 The plugin "@next/eslint-plugin-next" was referenced from the config file in ".eslintrc.js".
 If you still can't figure out the problem, please stop by https://eslint.org/chat/help to chat with the team.
 +----SUMMARY------+----------------------+---------------+-------+-------+--------+--------------+
 | Descriptor      | Linter               | Mode          | Files | Fixed | Errors | Elapsed time |
 +-----------------+----------------------+---------------+-------+-------+--------+--------------+
 | ✅ COPYPASTE    | jscpd                | project       |   n/a |       |      0 |        0.97s |
 | ✅ CSS          | stylelint            | list_of_files |     1 |     0 |      0 |        0.71s |
 | ⚠️ EDITORCONFIG | editorconfig-checker | list_of_files |    16 |       |      1 |        0.06s |
 | ✅ JSON         | prettier             | list_of_files |     3 |     0 |      0 |        0.84s |
 | ✅ REPOSITORY   | git_diff             | project       |   n/a |       |      0 |         0.0s |
 | ❌ TSX          | eslint               | project       |   n/a |   yes |      1 |        4.08s |
 | ✅ YAML         | prettier             | list_of_files |     3 |     0 |      0 |        0.34s |
 | ✅ YAML         | yamllint             | list_of_files |     3 |       |      0 |        0.19s |
 +-----------------+----------------------+---------------+-------+-------+--------+--------------+
 [Gitlab Comment Reporter] No merge request has been found, so no comment has been posted
 [Updated Sources Reporter] No source file has been formatted or fixed
 ❌ Error(s) have been found during linting
 To disable linters or customize their checks, you can use a .mega-linter.yml file at the root of your repository
 More info at https://megalinter.io/7.5.0/configuration/
Uploading artifacts for failed job
00:00
 Uploading artifacts...
 WARNING: report: no matching files. Ensure that the artifact path is relative to the working directory (/builds/fe/template/next) 
 ERROR: No files to upload                          
 ERROR: Job failed: exit code 1
sharh commented 7 months ago

Here is the ESLint config file .eslintrc.js

/** @type {import('eslint').Linter.Config} */
module.exports = {
  extends: [
    // npm install --save-dev @typescript-eslint/eslint-plugin
    "plugin:@typescript-eslint/recommended",
    // npm install --save-dev eslint-config-prettier
    "prettier",
    // npm install --save-dev eslint-plugin-react
    "plugin:react/jsx-runtime",
    // npm install --save-dev @next/eslint-plugin-next
    "plugin:@next/next/recommended",
    // npm install --save-dev eslint-plugin-react-hooks
    "plugin:react-hooks/recommended",
    // npm install eslint-plugin-import --save-dev
    "plugin:import/recommended",
    // 支持ts
    "plugin:import/typescript",
    // npm install eslint-plugin-yml --save-dev
    "plugin:yml/recommended",
  ],
  globals: {
    wx: "readonly",
  },
  settings: {
    react: {
      version: "detect",
    },
    next: {
      rootDir: ".",
    },
    "import/parsers": {
      "@typescript-eslint/parser": [".ts", ".tsx", ".mdx"],
    },
    "import/resolver": {
      typescript: true,
      node: true,
    },
  },
  // npm install --save-dev @typescript-eslint/parser
  parser: "@typescript-eslint/parser",
  plugins: ["@typescript-eslint"],
  ignorePatterns: [
    "megalinter-reports/**/*",
    "!.lintstagedrc.cjs",
    "!.stylelintrc.cjs",
    "!.commitlintrc.cjs",
    "!.eslintrc.js",
  ],
  overrides: [
    {
      files: [".eslintrc.{js,cjs}", "**/*.tsx", "**/*.cjs", "**/*.ts", "**/*.mjs", "**/*.jsx", "**/*.mdx"],
      rules: {
        quotes: "off",
        "@typescript-eslint/no-explicit-any": "warn",
        "import/first": 2,
      },
    },
    {
      files: ["*.yaml", "*.yml"],
      parser: "yaml-eslint-parser",
      // Options used with yaml-eslint-parser.
      parserOptions: {
        defaultYAMLVersion: "1.2",
      },
    },
  ],
  noInlineConfig: true,
};
nvuillam commented 7 months ago

@sharh as eslint is installed globally, I think you don't have other choice than installing plugins using the full command in PRE_COMMANDS, like npm install @next/eslint-plugin-next

But you remark is very valid... we could add an option to parse package.json and automatically build & run the dependencies commands to install :)

sharh commented 7 months ago

Thanks for repy.If that becomes true, that would be very nice. And when will it be true?

nvuillam commented 7 months ago

@sharh when someone makes a Pull Request to make it available :) Maybe you ? ^^ I my side I already have a big backlog ^^

I imagine some config like JAVASCRIPT_ESLINT_USE_NPM=true, and if found, it would run npm install then call eslint via npm run eslint :)

echoix commented 7 months ago

Wouldn't it make it really hard to support, on our side? Npm is pretty intertwined. Unless there is a kind of environment possible.

nvuillam commented 7 months ago

@echoix in that case would run the eslint installed by npm install and not the one contained in MegaLinter image... it would be something experimental and disabled by default, as like you i'm not 100% sure it wouldn't generate regressions

echoix commented 7 months ago

I was more thinking on the side that we can't officially offer support for something that we can't control. It's not really testable on our side, and their setup would have to work with what node versions we ship. Does the install of node_modules would like "uninstall" our linters (or downgrade them) to satisfy dependencies? Does an "virtualenv" concept exist in node? What happens if we change the way we package/install our linters? As always, precommands can be used to do anything users want..

nvuillam commented 7 months ago

@echoix our node_modules are installed in /node-deps , i assume what I propose would install them in working directory/node_modules, so it wouldn't mess with our installs ^^ But maybe I missed something ? :/

huczas commented 5 months ago

So basicaly I have similar problem. I'm trying to use TYPESCRIPT_ES with my dependencies to install in

PRE_COMMANDS:
  - command: npm install @angular-eslint/eslint-plugin --save-dev -g
    cwd: "root" 

output is like this:

[Pre] run: [cd /node-deps && npm install @angular-eslint/eslint-plugin --save-dev -g] in cwd [/]
[Pre] result:

added 136 packages in 3s

31 packages are looking for funding
  run `npm fund` for details

But later I get:

- Using [eslint v8.54.0] https://megalinter.io/7.6.0/descriptors/typescript_eslint
- MegaLinter key: [TYPESCRIPT_ES]
- Rules config: [/.eslintrc.json]
- Number of files analyzed: [797]
--Error detail:

Oops! Something went wrong! :(

ESLint: 8.54.0

ESLint couldn't find the plugin "@angular-eslint/eslint-plugin".

(The package "@angular-eslint/eslint-plugin" was not found when loaded as a Node module from the directory "/tmp/lint".)

It's likely that the plugin isn't installed correctly. Try reinstalling by running the following:

    npm install @angular-eslint/eslint-plugin@latest --save-dev

The plugin "@angular-eslint/eslint-plugin" was referenced from the config file in "--config#overrides[0]".

I've tried in PRE_COMMANDS also cwd: workspace but with te same results. Where I should install npm package to make it discoverable and work?

EDIT: for future reference, I managed to make it work with command: npm install --production=false

lars-reimann commented 4 months ago

Regarding prettier plugins: They can only be discovered by MegaLinter's prettier if you point to the exact location in .prettierrc.js using require.resolve:

module.exports = {
    plugins: ['prettier-plugin-svelte'], // does not work
}
module.exports = {
    plugins: [require.resolve('prettier-plugin-svelte')], // works
}

Afterward, plugins can be installed using PRE_COMMANDS, as usual:

PRE_COMMANDS:
    - command: npm install prettier-plugin-svelte

This might be relevant, if MegaLinter tries to install the required plugins based on the package.json or for anyone using prettier plugins in the meantime.