Closed zephyrbot closed 6 years ago
zephyrbot
commented
8 years ago by Inaky Perez-Gonzalez:
.known-issues is not tests/ specific; it is used to be able to filter out any output of any process for building, testing, doc building, etc in the Zephyr tree to remove things that are already known so they are not displayed or flagged by CI.
defaults.tc can be moved to tests/, but then a copy has to be placed in samples/
zephyrbot
commented
8 years ago by Anas Nashif:
{quote}.known-issues is not tests/ specific; it is used to be able to filter out any output of any process for building, testing, doc building, etc in the Zephyr tree to remove things that are already known so they are not displayed or flagged by CI.{quote}
What you describe above is test-related. If something is not working or is a known issue, I would like it to be documented in some big KNOWN-ISSUEs file and in some readable format. To get into the known-issues it must be really huge, otherwise we should
what kind of known-issue is this?
#
# Make blurbs
#
^make: (Entering|Leaving) directory .*Looking at the content of the directory, it all points to testing, we are creating a path here for developers to avoid fixing issues or for testers to avoid fixing their test cases or test tools by adding stuff like filters into a hidden directory and in a very cryptic format.
{quote}defaults.tc can be moved to tests/, but then a copy has to be placed in samples/ {quote}
Let me say this once again, samples are not tests and should not be treated as such.
zephyrbot
commented
8 years ago by Inaky Perez-Gonzalez:
bq. What you describe above is test-related.
Building the documentation is not test related. Compiling is not necessarily test related. Checkpatch is not test related.
This tool is meant to filter the output of all the processes we do to remove the things that are known and tracked so we can display to the user who invoked it what he needs to focus on, producing quickly usable and actionable output logs.
bq. If something is not working or is a known issue, I would like it to be documented in some big KNOWN-ISSUEs file and in some readable format.
You can place all the expressions in a single file if you want; that kills readability though.
Readability has to be for humans and for the tool.
For that, the format has a part for the humans (the command block before each regular expression) and the regular expression (for the tool to know what to filter out). Furthermore, they can be spread on one or multiple files per policy preference. It is clearly documented in .known-issues/README https://gerrit.zephyrproject.org/r/gitweb?p=zephyr.git;a=blob;f=.known-issues/README; if you have issues with the lack of documentation inside the files, then bring it up with whoever created/named those files.
bq. To get into the known-issues it must be really huge, otherwise we should
Certainly
bq. try to fix it
bq. exclude it where it happens (sanitychecks supports exclusion)
Both sanitycheck and TCF allow to skip or not, but does not allow you to skip by architecture or by board type or by board instance, which is the case in many failing test cases. So by using skip you'd be excluding a test case everywhere.
In this case, you want to filter by CONFIG options. You can do that if you want not to even try to execute the test case.
This methodology, however, introduces an escalation issue, as you are spreading the exclusion information around a myriad of files; .known-issues allows you to keep it in a single directory (in one or multiple files) that can be controlled with the MAINTAINERS mechanism.
bq. document why it is excluded.
Certainly; the format forces a documentation block for each regular expression that has to be used to explain to humans what it is and why (eg: https://gerrit.zephyrproject.org/r/#/c/3400/10/.known-issues/testcases/quark_se_c1000.conf) (this case is anotherone that should be renamed after the JIRA entry it is tracking). Note that particular example will be resolved once the https://gerrit.zephyrproject.org/r/4533 series is merged.
bq. what kind of known-issue is this? (make stuff)
This filters lines by the build process we want to ignore, as they provide no useful information--we haven't agreed on a good name for said file and probably it can use a better documentation block to explain what it is removing. Suggestions?
bq. Looking at the content of the directory, it all points to testing,
Nope, .know-issues/doc/ is purely issues steaming from building the documentation (in fact, this is how this system started)
bq. we are creating a path here for developers to avoid fixing issues or for testers to avoid fixing their test cases or test tools by adding stuff like filters into a hidden directory and in a very cryptic format.
It is a path for people to allow ignoring issues, hence why it is heavily centralized in a single place that is easy to monitor via MAINTAINERS and by inspecting what is in .known-issues.
Cryptic: As said before, the format is intended for both humans and tool parsing and It is clearly documented in .known-issues/README https://gerrit.zephyrproject.org/r/gitweb?p=zephyr.git;a=blob;f=.known-issues/README. Or propose a better format that fits all the situations (note the documentation process generates multiline error and warning messages)
bq. Let me say this once again, samples are not tests and should not be treated as such.
They still have to be built and run. For that, it needs said file.
nashif
commented
6 years ago detauls.tc remove, we are good
Reported by Anas Nashif:
There is no reason why we should those test related files in top directory. Please keep all test related files under tests.
(Imported from Jira ZEP-768)