Mega-Linter Runner Documentation Outdated

[Kurt-von-Laven]

Describe the bug The counts of supported tools in Mega-Linter Runner's README have fallen out of date with respect to the counts in Mega-Linter's README.

Expected behavior I expected both counts to match. I would either delete the counts from Mega-Linter Runner or copy and paste the entire duplicated portion from the main README, overwriting the Runner's README.

Additional context I haven't counted to verify whether the main README gives correct counts.

[llaville]

And also

[Kurt-von-Laven]

Good catch; that should say 19 tooling formats if the main README is correct, which I am increasingly skeptical of ha ha. It also has an extra space before two of the commas.

[llaville]

And @nvuillam, look your profile too with

[nvuillam]

I didn't find a way to automate that except for readme, so I mnually do it from time to time ^^ Maybe there is ^^

[Kurt-von-Laven]

It's definitely not a perfect fit for the job, but I found markdown-autodocs. Maybe someone else has a better suggestion.

[nvuillam]

Not sure the repo description is in markdown :/ I browse github to check if there is something via their api...

[nvuillam]

Not sure the repo description is in markdown :/ I browse github to check if there is something via their api...

[llaville]

I browse github to check if there is something via their api...

Perharps https://docs.github.com/en/rest/reference/repos#update-a-repository

[nvuillam]

True ! But nothing in github CLI :( https://cli.github.com/manual/

If you feel motivated , please be my guest to implement auto-update , else it will go at the end of the backlog ... and i'll continue to update manually the repo description from time to time ^^

[Kurt-von-Laven]

FYI, GitHub also has a GraphQL API, which offers equivalent functionality in this case and superior functionality in some other cases.

[nvuillam]

@Kurt-von-Laven as this is only cosmetic update, I won't take care of it until V5 - megalinter organization, but i'll be happy to valiate a PR if you submit it before that ^^

[Kurt-von-Laven]

I agree that this isn't our biggest fish to fry.

[Kurt-von-Laven]

It is worth noting that we actually "only" support 29 languages, 11 formats, and 13 tooling formats. 48 is the count of the total number of language linters, 22 the count of the total number of format linters, and 19 the count of the total number of tooling format linters. I just confirmed that these counts are correct relative to the documentation. We should also add some mention of Secretlint while we are at it.

[nvuillam]

@Kurt-von-Laven I plead guilty haha , congrats you're the first to have noticed my little arrangement with reality 🤣

[llaville]

I should be able to propose tomorrow a solution to automate update process. Stats count is possible with yq command https://github.com/mikefarah/yq (current version is 4.14.2)

Here are bash commands to get current stat from megalinter repository

type="language" yq e '{.descriptor_id: .descriptor_type | select(. == env(type))}' megalinter/descriptors/*.megalinter-descriptor.yml | wc -l
> 29

type="format" yq e '{.descriptor_id: .descriptor_type | select(. == env(type))}' megalinter/descriptors/*.megalinter-descriptor.yml | wc -l
> 11

type="tooling_format" yq e '{.descriptor_id: .descriptor_type | select(. == env(type))}' megalinter/descriptors/*.megalinter-descriptor.yml | wc -l
> 13

[llaville]

To get the current full list of descriptors language, format and tooling_format, launch command

yq e '{.descriptor_id: .descriptor_type}' megalinter/descriptors/*.megalinter-descriptor.yml

Output

ACTION: tooling_format
ANSIBLE: tooling_format
ARM: tooling_format
BASH: language
C: language
CLOJURE: language
CLOUDFORMATION: tooling_format
COFFEE: language
COPYPASTE: other
CPP: language
CREDENTIALS: other
CSHARP: language
CSS: format
DART: language
DOCKERFILE: tooling_format
EDITORCONFIG: tooling_format
ENV: format
GHERKIN: tooling_format
GIT: other
GO: language
GRAPHQL: format
GROOVY: language
HTML: format
JAVA: language
JAVASCRIPT: language
JSON: format
JSX: language
KOTLIN: language
KUBERNETES: tooling_format
LATEX: format
LUA: language
MARKDOWN: format
OPENAPI: tooling_format
PERL: language
PHP: language
POWERSHELL: language
PROTOBUF: format
PUPPET: tooling_format
PYTHON: language
R: language
RAKU: language
RST: format
RUBY: language
RUST: language
SALESFORCE: language
SCALA: language
SNAKEMAKE: tooling_format
SPELL: other
SQL: language
SWIFT: language
TEKTON: tooling_format
TERRAFORM: tooling_format
TSX: language
TYPESCRIPT: language
VBDOTNET: language
XML: format
YAML: format

[llaville]

I've propose a PR in draft mode because :

1. it solved only official Mega-Linter Markdown documentation files with a little (WARNING) exception on welcome phrase for Mega-Linter-Runner (that is a bit different from first one). Perharps we should fix it !?
2. Main Dockerfile contains hard coded yq command installation
3. The project description about field could not be solve by this PR. A call to github api is necessary: I will propose another solution for that too: that may be run with github cli gh api (https://cli.github.com/manual/gh_api) or httpie (https://github.com/httpie/httpie) by using the GraphQL API

[llaville]

4. I forget allow to specify that there is anoter use case that we will take care : https://raw.githubusercontent.com/megalinter/megalinter/main/docs/assets/images/archi.png @nvuillam I don't know how and by which tool your generate it, but I suggest to use tool that generate image from textual description. I propose a tool I know well and use it https://github.com/yuzutech/kroki Have a look on examples page: https://kroki.io/examples.html Try it yourself with playground page: https://kroki.io/#try

[nvuillam]

@llaville good old PowerPoint in 10 minutes 😅 But i'd be happy if we had a way to automate it ;)

[llaville]

A quick preview (basic example) what we can do with kroki (automation will come later, with graphql gh api too ; I'll have not time today)

Go on playground https://kroki.io/#try, select BlockDiag diagram, and put this text diagram description

blockdiag {
  code_change -> run_linters -> run_reporters;
  code_change -> languages;
  code_change -> formatters
  code_change -> tooling_linters;
  run_linters -> console;
  run_linters -> text;
  run_linters -> json;
  run_linters -> ide_config;
  run_linters -> email;
  run_linters -> file_io;
  run_linters -> gh_pr;
  run_linters -> gh_status;
  run_linters -> tap;
  run_linters -> updated_sources;

  code_change [label = "Code Change"];
  run_linters [label = "Run linters"];
  run_reporters [label = "Run reporters"];
  languages [label = "29 languages", color="yellow"];
  formatters [label = "11 formatters", color="yellow"];
  tooling_linters [label = "13 tooling linters", color="yellow"];
  console [label = "Console", color="greenyellow"];
  text [label = "Text", color="greenyellow"];
  json [label = "JSON", color="greenyellow"];
  ide_config [label = "IDE Configuration", color="greenyellow"];
  email [label = "E-Mail", color="greenyellow"];
  file_io [label = "file.io", color="greenyellow"];
  gh_pr [label = "Github Pull Request", color="greenyellow"];
  gh_status [label = "Github Status", color="greenyellow"];
  tap [label = "TAP", color="greenyellow"];
  updated_sources [label = "Updated sources", color="greenyellow"];
}

[llaville]

Last but not least, kroki is a french project :)

[llaville]

@nvuillam my PR (still in draft mode) is ready for review except for a CI fail job that seems related to contributor env broken https://github.com/megalinter/megalinter/runs/4272059799?check_suite_focus=true ; can you confirm this last point ?

[llaville]

For architecture diagram automation, I propose this bash script (.automation/kroki.sh for demo) but may be integrate in CI automation.

#!/usr/bin/env bash

################################################################################
############# Generate architecture diagram of Mega-Linter #####################
#############  See how to do it at https://kroki.io/#how   #####################
################################################################################
shopt -s lastpipe

# Get live stats from YAML descriptor files
docker run --rm -v "${PWD}":/workdir mikefarah/yq eval '{.descriptor_id: .descriptor_type}' megalinter/descriptors/*.megalinter-descriptor.yml | descriptors=$(</dev/stdin)

grep -o "language" <<< "$descriptors" | wc -l | language=$(</dev/stdin)
grep -o " format" <<< "$descriptors" | wc -l | format=$(</dev/stdin)
grep -o "tooling_format" <<< "$descriptors" | wc -l | tooling_format=$(</dev/stdin)

cp docs/assets/archi.blockdiag.txt /tmp/ml_archi_diagram

sed -i "s/{language}/$language/" /tmp/ml_archi_diagram
sed -i "s/{format}/$format/" /tmp/ml_archi_diagram
sed -i "s/{tooling_format}/$tooling_format/" /tmp/ml_archi_diagram

# Build diagram in blockdiag format and encode diagram using deflate + base64
cat /tmp/ml_archi_diagram | python -c "import sys; import base64; import zlib; print(base64.urlsafe_b64encode(zlib.compress(sys.stdin.read(), 9)))" | diagram=$(</dev/stdin)

echo "https://kroki.io/blockdiag/svg/$diagram"

Here is sample of what we can get. Follows link generated to see SVG version online

https://kroki.io/blockdiag/svg/eNp9k19LwzAUxd_3KUKfnTB9Epkg2xAFdWz6JFKy9i6LZrk1TVARv7tJ_2ZdzFvzO-fc297cbgRm7zmnjPyMCMkwhzTbUcmAjK-IMjIVXGpQZXtUUKBy4PLYLuyDoQxC2hbVnmoXPNY0ou3C2lYuPeicoSxRQEDR8KUD-K1EGcDc9UW55Swgwp5yEeBbLiDlGFDYLi1UmJeaahP6FE2LADVFTjXkaYlGZW6Agym9CLoBQaYkmVlKZhVNXoeVetvKSNLQ3tZd3sDY8dra3aNnO7vocXJi302gmibfIAR-1qn-hr3YZOLxcG5w-374vBW7TwlWaLbjYEgV6e1MAciDrnZvvMCTPUbcbp0899368SHi7rfMy9zOF2RWQaOo5igjBapN9LKL8b0FkUCzol7EkVOOkUy1vF7ihuud2ZClEYKs4MNAqePpesWPK6wrHps9LfzRXy8j3sGP4eWea4U0yr81fkd_UYGk8w==

[llaville]

Source file docs/assets/archi.blockdiag.txt become a template with 3 placeholders : {language}, {format}, {tooling_format}, (see sed operations)

blockdiag {
  code_change -> run_linters -> run_reporters;
  code_change -> languages;
  code_change -> formatters
  code_change -> tooling_linters;
  run_linters -> console;
  run_linters -> text;
  run_linters -> json;
  run_linters -> ide_config;
  run_linters -> email;
  run_linters -> file_io;
  run_linters -> gh_pr;
  run_linters -> gh_status;
  run_linters -> tap;
  run_linters -> updated_sources;

  code_change [label = "Code Change"];
  run_linters [label = "Run linters"];
  run_reporters [label = "Run reporters"];
  languages [label = "{language} languages", color="yellow"];
  formatters [label = "{format} formatters", color="yellow"];
  tooling_linters [label = "{tooling_format} tooling linters", color="yellow"];
  console [label = "Console", color="greenyellow"];
  text [label = "Text", color="greenyellow"];
  json [label = "JSON", color="greenyellow"];
  ide_config [label = "IDE Configuration", color="greenyellow"];
  email [label = "E-Mail", color="greenyellow"];
  file_io [label = "file.io", color="greenyellow"];
  gh_pr [label = "Github Pull Request", color="greenyellow"];
  gh_status [label = "Github Status", color="greenyellow"];
  tap [label = "TAP", color="greenyellow"];
  updated_sources [label = "Updated sources", color="greenyellow"];
}

[llaville]

Just included now in PR #1007 with commit https://github.com/megalinter/megalinter/pull/1007/commits/8923d0bab226814982f1abdae51fd57338178030

[llaville]

GraphQL Python script is ready, but I don't think we should include it in PR 1007.

[llaville]

Use case : update my mega-linter v4 version fork still available at https://github.com/llaville/mega-linter

Before running script

After running script

Steps are :

• generate a github personal access token (PAT)
• change in graphql_gh_repo.py

      variables = {
            "owner": "llaville",
            "name": "mega-linter"
      } 

  Must identify github repository to update

• change in graphql_gh_repo.py value of client mutation

      variables = {
          "clientMutationId": "llaville",

• call the following python script with PAT and data coming from stats retrieved from yq command (see .automation/kroki.sh script in commit https://github.com/megalinter/megalinter/pull/1007/commits/8923d0bab226814982f1abdae51fd57338178030)

Source code of graphql_gh_repo.py is

# !/usr/bin/env python3
"""
Update Github Mega-Linter repository about description
"""
# pylint: disable=import-error
import requests

class graphQL:
    BASE_URL = "https://api.github.com/graphql"

    """docstring for graphQL"""
    def __init__(
        self,
        descriptor_language_count,
        descriptor_format_count,
        descriptor_tooling_format_count,
        github_token,
        additional_headers=None
    ):
        super(graphQL, self).__init__()

        self.descriptor_language_count = descriptor_language_count
        self.descriptor_format_count = descriptor_format_count
        self.descriptor_tooling_format_count = descriptor_tooling_format_count

        self.github_token = github_token
        self.additional_headers = additional_headers or dict()

        self.getRepositoryId()
        self.updateRepositoryDescription()

    @property
    def headers(self):
        default_headers = dict(
            Authorization=f"token {self.github_token}",
        )
        return {
            **default_headers,
            **self.additional_headers
        }

    def run_query(self, query, variables):
        request = requests.post(self.BASE_URL, json={'query': query, 'variables': variables}, headers=self.headers)
        if request.status_code == 200:
            return request.json()
        else:
            raise Exception("Query failed to run by returning code of {}. {}".format(request.status_code, query))

    def getRepositoryId(self):
        query = '''
            query($owner: String!, $name: String!) {
                repository(owner: $owner, name: $name) {
                    id
                }
            }'''

        variables = {
              "owner": "llaville",
              "name": "mega-linter"
        }

        result = self.run_query(query, variables)
        self.repositoryId = result["data"]["repository"]["id"]

    def updateRepositoryDescription(self):
        mutation = '''
            mutation($clientMutationId: String!, $repositoryId: String!, $description: String!) {
                updateRepository(input: {clientMutationId: $clientMutationId, repositoryId: $repositoryId, description: $description}) {
                    clientMutationId
                    repository {
                        updatedAt
                    }
                }
            }'''

        variables = {
            "clientMutationId": "llaville",
            "repositoryId": self.repositoryId,
            "description": "🦙 Mega-Linter analyzes " +
                str(self.descriptor_language_count) + " languages, " +
                str(self.descriptor_format_count) + " formats, "  +
                str(self.descriptor_tooling_format_count) + " tooling formats, " +
                "abusive copy-pastes and spelling mistakes in your repository sources, " +
                "with a GitHub Action, other CI tools or locally"
        }
        result = self.run_query(mutation, variables)
        print(result)

Example of python call

graphql = graphQL(29, 11, 13, PAT)

Will print something like (for demo debug purpose only)

{'data': {'updateRepository': {'clientMutationId': 'llaville', 'repository': {'updatedAt': '2021-11-20T11:10:21Z'}}}}

[Kurt-von-Laven]

@nvuillam my PR (still in draft mode) is ready for review except for a CI fail job that seems related to contributor env broken https://github.com/megalinter/megalinter/runs/4272059799?check_suite_focus=true ; can you confirm this last point ?

I don't know what this is about, unfortunately. My unhelpful suggestion for you is to rebase before pushing your next iteration, and check if this issue still manifests before we debug it too hard.

[Kurt-von-Laven]

@llaville, thank you for including such helpful context on this thread, especially the sample product of kroki.sh. It was very helpful for code review. I agree with you that the GraphQL would be best to leave for a separate PR.

[nvuillam]

kroki seems to be indeed a great tool,I hope that your PR "with less bash" will be ok soon so we can benefit from it :)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

If you think this issue should stay open, please remove the O: stale 🤖 label or comment on the issue.