search

priv-kweihmann / oelint-adv

Advanced oelint
BSD 2-Clause "Simplified" License
56 stars 27 forks source link
bitbake openembedded qatools yocto

readme

oelint-adv

Build status PyPI version Python version Downloads

Advanced oelint

Purpose

Based on the OpenEmbedded Styleguide and work done by oe-stylize-tool this module offers a (nearly) complete linter for bitbake-recipes.

The tool should help anyone working with YOCTO/OpenEmbedded to write more clean, less 'magical' recipes, without the need to know all the internals of your used poky/OpenEmbedded version.

It could also be used as part of a CI to avoid hard to debug issues slipping to your code base - be sure to checkout rulefile for that use case.

As every linter this tool is sometimes extra picky, but for the good of it, not just to bug people. Especially for novice users it might be a help to avoid the most common pitfalls of bitbake recipes.

The tool does handle includes/requires automatically, so you don't have to pass them via CLI.

NOTE: .bbappend-files have to be passed via CLI - these are NOT gathered automatically.

You can also pass distro, machine and layer config files. Those will be automatically handled in the correct order.

Install

With pip (recommended)

pip3 install oelint_adv

from source

git clone https://github.com/priv-kweihmann/oelint-adv
cd oelint-adv
python3 setup.py install # might require sudo/root permissions

NOTE if you install from source, you'll have to provide all matching required python libraries on your own. See requirements.txt for details

Usage

usage: oelint-adv [-h] [--suppress SUPPRESS] [--output OUTPUT] [--fix] [--nobackup] [--addrules ADDRULES [ADDRULES ...]]
                  [--customrules CUSTOMRULES [CUSTOMRULES ...]] [--rulefile RULEFILE] [--jobs JOBS] [--color] [--quiet]
                  [--hide {info,warning,error}]
                  [--mode {fast,all}] [--relpaths] [--messageformat MESSAGEFORMAT] [--constantmods CONSTANTMODS [CONSTANTMODS ...]] [--print-rulefile]
                  [--exit-zero]
                  [--release {...}]
                  [--version]
                  [files ...]

Advanced OELint - Check bitbake recipes against OECore styleguide

positional arguments:
  files                 File to parse

options:
  -h, --help            show this help message and exit
  --suppress SUPPRESS   Rules to suppress
  --output OUTPUT       Where to flush the findings (default: stderr)
  --fix                 Automatically try to fix the issues
  --nobackup            Don't create backup file when auto fixing
  --addrules ADDRULES [ADDRULES ...]
                        Additional non-default rulessets to add
  --customrules CUSTOMRULES [CUSTOMRULES ...]
                        Additional directories to parse for rulessets
  --rulefile RULEFILE   Rulefile
  --jobs JOBS           Number of jobs to run (default all cores)
  --color               Add color to the output based on the severity
  --quiet               Print findings only
  --hide {info,warning,error}
                        Hide mesesages of specified severity
  --mode {fast,all}     Level of testing (default: fast)
  --relpaths            Show relative paths instead of absolute paths in results
  --messageformat MESSAGEFORMAT
                        Format of message output
  --constantmods CONSTANTMODS [CONSTANTMODS ...]
                        Modifications to the constant db. prefix with: + - to add to DB, - - to remove from DB, None - to override DB
  --print-rulefile      Print loaded rules as a rulefile and exit
  --exit-zero           Always return a 0 (non-error) status code, even if lint errors are found
  --release {...}
                        Run against a specific Yocto release
  --cached              Use caches (default: off)
  --cachedir CACHEDIR   Cache directory (default $HOME/.oelint/caches)
  --clear-caches        Clear cache directory and exit
  --version             show program's version number and exit

Output

Will be [file]:[line]:[severity]:[id]:[message]. To change the default message format, please see Output message format section.

Example:

/disk/meta-some/cppcheck-native/cppcheck.inc:26:error:oelint.task.nomkdir:'mkdir' shall not be used in do_install. Use 'install' [branch:true]
/disk/meta-some/cppcheck-native/cppcheck-native_1.87.bb:0:error:oelint.var.mandatoryvar.SECTION:Variable 'SECTION' should be set [branch:true]
/disk/meta-some/cppcheck-native/cppcheck.inc:1:warning:oelint.vars.summary80chars:'SUMMARY' should not be longer than 80 characters [branch:true]
/disk/meta-some/cppcheck-native/cppcheck.inc:4:warning:oelint.vars.homepageprefix:'HOMEPAGE' should start with 'http://' or 'https://' [branch:true]
/disk/meta-some/cppcheck-native/cppcheck.inc:28:warning:oelint.spaces.lineend:Line shall not end with a space [branch:true]
/disk/meta-some/cppcheck-native/cppcheck-native_1.87.bb:0:error:oelint.var.mandatoryvar.AUTHOR:Variable 'AUTHOR' should be set [branch:true]
/disk/meta-some/cppcheck-native/cppcheck.inc:26:error:oelint.task.nocopy:'cp' shall not be used in do_install. Use 'install' [branch:true]
/disk/meta-some/cppcheck-native/cppcheck.inc:12:warning:oelint.var.order.DEPENDS:'DEPENDS' should be placed before 'inherit' [branch:true]

NOTE: as the tool checks against permutations of the input files, the used permutation matrix is appended to the rules message. E.g.

This should help to better spot the issue found by the linter.

Apply automatic fixing

Some of the rules are capable of fixing the issues found automatically. This will be done if you pass --fix as a startup parameter.

As long as you don't pass --nobackup a backup copy (filename + .bak) will be created for all files fixed.

Available rules

Rules marked with [F] are able to perform automatic fixing Rules marked with [S] can have multiple sub-IDs Rules marked with [I] need to be activated through a rule file first

Non-default rulesets

To enable rulesets that are not part of the standard ruleset pass --addrules \<ruleset-name> to CLI.

These rules are sometimes contrary to OE-style-guide, so use them with caution.

jetm ruleset

To enable pass --addrules jetm to CLI.

Rules marked with [F] are able to perform automatic fixing.

Writing your own additional rules

By passing --customrules via CLI you could add additional rules to be checked. The argument should point to a directory - every class derived from Rule will be automatically loaded. Please use the following as a template for your own:

from oelint_adv.cls_rule import Rule

class FooMagicRule(Rule):
    def __init__(self):
        super().__init__(id="foocorp.foo.magic",
                         severity="error",
                         message="Too much foo happening here")

    def check(self, _file, stash):
        res = []
        items = stash.GetItemsFor(filename=_file)
        for i in items:
            if "Foo" in i.Raw:
                res += self.finding(i.Origin, i.InFileLine)
        return res

    # To provide automatic fixing capability
    # add the following optional function
    def fix(self, _file, stash):
        res = []
        items = stash.GetItemsFor(filename=_file)
        for i in items:
            if 'Foo' in i.Raw:
                # you have to replace the content of `RealRaw` and `Raw`
                # with the fixed content
                # `Raw` is the raw block with expanded inlines blocks
                # `RealRaw` is the raw block without any modifications
                #           this is what will be actually written to the file
                i.RealRaw = i.RealRaw.replace('Foo', 'Bar')
                i.Raw = i.Raw.replace('Foo', 'Bar')
                # Return the file name to signalize that fixes have been
                # applied
                res.append(_file)
        return res

For more details please see the function docstrings of the API.

You can find this example also in the development source tree

Additional real-life examples can be found in e.g. meta-rubygems

Release constrained rules

Rules can be also configured to work on specific releases of YP only, if determined not to be applicable, the rules will be skipped during the loading process and therefore won't show up if e.g. --print-rulefile is used

Rules working up to a certain release

from oelint_adv.cls_rule import Rule

class FooMagicRule(Rule):
    def __init__(self):
        super().__init__(id="foocorp.foo.magic",
                         severity="error",
                         message="Too much foo happening here",
                         valid_till_release="kirkstone")

Would enable the rule, but only if --release is set to a YP release earlier than kirkstone

Rules working from a certain release

from oelint_adv.cls_rule import Rule

class FooMagicRule(Rule):
    def __init__(self):
        super().__init__(id="foocorp.foo.magic",
                         severity="error",
                         message="Too much foo happening here",
                         valid_from_release="kirkstone")

Would enable the rule, but only if --release is set to a YP release later than kirkstone (including kirkstone)

Enable special settings per release

from oelint_adv.cls_rule import Rule

class FooMagicRule(Rule):
    def __init__(self):
        super().__init__(id="foocorp.foo.magic",
                         severity="error",
                         message="Too much foo happening here")

    def check_release_range(self, release_range: List[str]) -> bool:
        if 'kirkstone' in release_range:
            self._we_are_running_on_kirkstone = True
            self.Appendix.append('kirkstone')
        return super().check_release_range(release_range)

Enables the special Appendix kirkstone, if kirkstone is part of the calculated --release list.

It also sets the variable self._we_are_running_on_kirkstone, which can be used as part of check() to code special code paths.

Defining a ruleset

If you pass the option --rulefile you could define the rules to be checked and their severity via a simple json file.

The rule file could look like this:

{
  "<rule>": "<severity>"
}

to override the severity, or

{
  "<rule>": ""
}

to keep the original severity.

Example

{
  "oelint.file.includenotfound": "",
  "oelint.file.requirenotfound": "warning"
}

would enable the two rules oelint.file.includenotfound and oelint.file.requirenotfound. The severity of oelint.file.includenotfound will be the default of the tool, while oelint.file.requirenotfound will report warning instead of the original suggested severity.

Adding additional constants

Please see oelint-parser for further details, how to add your own constants to the parser.

Output message format

You can freely define a custom output format. The following placeholder symbols will be automatically replaced

name replaced by
{path} path of the file
{line} line of the finding
{severity} severity of the finding
{id} error-ID of the finding
{msg} description of the finding
{wikiurl} a link to the online wiki

Configuration file

You can define your own global or project wide defaults for all CLI parameters with an ini-style configuration file.

In the following order files are probed

Explicitly passed options to CLI are always chosen over the defaults defined by the configuration file.

To skip the loading of any configuration file, set OELINT_SKIP_CONFIG to a non empty value in your environment.

File format

[oelint]
# this will set the --hide warning parameter automatically
hide = warning
# this will set A + B as suppress item
# use indent (tab) and line breaks for multiple items
suppress = 
  A
  B
# this will set messageformat parameter
messageformat = {severity}:{id}:{msg}
# will configure --release to dunfell
release=dunfell

You can find an example file here

Inline suppression

You can suppress one or more checks on a line by line basis

# nooelint: <id>[,<id>,...]

suppresses all the specified IDs for the next line. Multiple IDs can be separated by commas.

Example

# nooelint: oelint.vars.insaneskip
INSANE_SKIP:${PN} = "foo"

will not warn about the usage of INSANE_SKIP.

Configure your custom machine or distro settings

You can let oelint-adv know about your custom MACHINE and DISTRO overrides, so they won't produce any findings, when used in your setup.

Create a layer specific oelint configuration file

Copy over the template this file to the root of your layer

LAYER_PATH=<path to your layer>
cp docs/.oelint.cfg.custom-layer $LAYER_PATH/.oelint.cfg

Create the known MACHINE and DISTRO settings

Source your bitbake build directory and run the following commands (**) (***)

LAYER_PATH=<path to your layer>
bitbake-getvar --quiet --value MACHINEOVERRIDES | tr ':' '\n' | jq -Rn  '{replacements:{ machines: [inputs]}}' > $LAYER_PATH/.oelint-custom-machines.json
bitbake-getvar --quiet --value DISTROOVERRIDES | tr ':' '\n' | jq -Rn  '{replacements:{ distros: [inputs]}}' > $LAYER_PATH/.oelint-custom-distros.json

(**) you'll need to have jq installed on your local machine.

(***) bitbake-getvar command is available since kirkstone release. For older release you can use bitbake core-image-minimal -e | grep ^MACHINEOVERRIDES resp. bitbake core-image-minimal -e | grep ^DISTROOVERRIDES and pass them into the rest of the pipe.

cached mode

When run with --cached the tool will store resuls into a local caching directory and reuse the results, if nothing has changed in the input tree and configuration.

By default caches are stored to OELINT_CACHE_DIR environment variable (or ~/.oelint/caches if not set).

To clear the local caches run --clear-caches

vscode extension

Find the extension in the marketplace, or search for oelint-vscode.

Vim / NeoVim integration

Integration for Vim / NeoVim is provided by ale or nvim-lint.

Jenkins integration

Jenkins integration is provided by warnings-ng.

Use as a library

To use the linter as part of calling application do

from oelint_adv.core import create_lib_arguments, run

args = create_lib_arguments(['file to check', 'another file to check']])

# check the docstring of create_lib_arguments for more options

results = run(args)

# the results will be a List[Tuple[Tuple[str, int], str]]
# each item is
#  [0] - 'path to the finding', 'line of the finding'
#  [1] - 'message'

The caller is responsible for appropriate exception handling

Missing anything?

You think there's something missing, wrong, 'improvable'... Just file an issue to get in contact.

Contribute

Any sort of ideas, pull requests, bug reports, reports of false positives are welcome. This project is open to anyone - no pitfalls or legal inconveniences.