taiki-e / cargo-llvm-cov

Cargo subcommand to easily use LLVM source-based code coverage (-C instrument-coverage).
Apache License 2.0
933 stars 57 forks source link

--codecov reports a line with no branches as only partially covered #252

Closed taiki-e closed 1 year ago

taiki-e commented 1 year ago

--lcov (full report):

lcov

--codecov (full report):

codecov

--html:

html

I believe this line should be marked as 100% covered in both line and region coverage because there are no branches.

cc @andrewgazelka: Could you take a look at this?

andrewgazelka commented 1 year ago

Will do. By the way, C-* means something is less important, whereas B-* is more important and A-* is most important, correct? I see tags like this often.

Also—did you run tests in release or debug? When I did tests in debug, I seemed never to be able to get full coverage for any line.

taiki-e commented 1 year ago

Also—did you run tests in release or debug? When I did tests in debug, I seemed never to be able to get full coverage for any line.

Ah, I ran it in debug mode. I will try the release mode later.

By the way, C-* means something is less important, whereas B-* is more important and A-* is most important, correct? I see tags like this often.

In rust-lang/rust, C-* means "Category" of the issue (e.g., bug report, feature request, etc.), and A-* means "Area" related to the issue (e.g., macros, lifetime, etc.). Many of my projects have adopted that convention as well.

taiki-e commented 1 year ago

Hmm. I ran it in release mode, but that did not fix this issue.

taiki-e commented 1 year ago

I checked the generated json file.

Generate coverage:

git clone https://github.com/openrr/openrr
cd openrr
cargo llvm-cov --release -p arci --codecov --output-path codecov.json

Get coverage of the file that ran into this issue:

jq '.coverage | to_entries[] | select(.key | endswith("utils.rs"))' codecov.json
output ```json { "key": "/Users/taiki/projects/sources/smilerobotics/openrr/arci/src/utils.rs", "value": { "8": "2/4", "9": "2/4", "10": "2/4", "11": "2/4", "12": "2/4", "13": "2/4", "14": "2/4", "15": "6/12", "16": "5/12", "17": "2/4", "20": "2/4", "21": "2/4", "22": "2/4", "23": "2/4", "24": "2/4", "25": "2/4", "26": "2/4", "27": "2/4", "28": "2/4", "29": "2/4", "30": "2/4", "31": "2/4", "32": "2/4", "33": "5/12", "34": "4/12", "35": "2/4", "36": "2/4", "37": "2/4", "38": "2/4", "39": "2/4", "40": "2/8", "42": "2/4", "43": "2/4", "44": "2/4", "45": "2/4", "48": "6/16", "49": "2/4", "50": "0/4", "51": "0/12", "52": "0/4", "53": "4/8", "54": "2/4", "55": "3/8", "56": "1/4", "57": "1/4", "58": "2/4", "59": "0/12", "60": "2/4", "61": "2/4", "62": "2/4", "63": "4/8", "65": "6/16", "66": "2/4", "67": "2/4", "68": "2/8", "69": "2/4", "70": "0/4", "71": "0/8", "72": "2/4", "73": "2/4" } } ```

Problematic line:

    "45": "2/4",

(full codecov.json: https://gist.github.com/taiki-e/8eb5339633c2634a0d1ea29c932cb509) (I wanted to post the output file from the --json flag, but it was too large to upload to gist.)

andrewgazelka commented 1 year ago

will look into this later today :)

andrewgazelka commented 1 year ago

Ran --json --release and got

       // ...
        {
          "branches": [],
          "count": 0,
          "filenames": [
            "/Users/andrewgazelka/Projects/fork/openrr/arci/src/utils.rs"
          ],
          "name": "_RNCINvNtCsAfKivQEvLC_4arci5utils21move_joint_until_stoppE0B6_",
          "regions": [
            [
              33,
              9,
              33,
              20,
              0,
              0,
              0,
              0
            ],
            [
              33,
              23,
              33,
              75,
              0,
              0,
              0,
              0
            ],
            [
              33,
              75,
              33,
              76,
              0,
              0,
              0,
              0
            ],
            [
              34,
              9,
              34,
              22,
              0,
              0,
              0,
              0
            ],
            [
              34,
              25,
              34,
              74,
              0,
              0,
              0,
              0
            ],
            [
              34,
              74,
              34,
              75,
              0,
              0,
              0,
              0
            ],
            [
              35,
              9,
              40,
              91,
              0,
              0,
              0,
              0
            ],
            [
              40,
              91,
              40,
              92,
              0,
              0,
              0,
              0
            ],
            [
              42,
              9,
              45,
              55,
              0,
              0,
              0,
              0
            ],
            [
              48,
              13,
              48,
              29,
              0,
              0,
              0,
              0
            ],
            [
              48,
              32,
              48,
              81,
              0,
              0,
              0,
              0
            ],
            [
              48,
              81,
              48,
              82,
              0,
              0,
              0,
              0
            ],
            [
              48,
              83,
              49,
              77,
              0,
              0,
              0,
              0
            ],
            [
              51,
              13,
              51,
              112,
              0,
              0,
              0,
              0
            ],
            [
              51,
              13,
              51,
              112,
              0,
              0,
              0,
              0
            ],
            [
              53,
              12,
              53,
              51,
              0,
              0,
              0,
              0
            ],
            [
              53,
              52,
              55,
              10,
              0,
              0,
              0,
              0
            ],
            [
              55,
              16,
              57,
              10,
              0,
              0,
              0,
              0
            ],
            [
              58,
              12,
              58,
              51,
              0,
              0,
              0,
              0
            ],
            [
              59,
              19,
              59,
              29,
              0,
              0,
              0,
              0
            ],
            [
              59,
              19,
              59,
              29,
              0,
              0,
              0,
              0
            ],
            [
              60,
              13,
              60,
              18,
              0,
              0,
              0,
              0
            ],
            [
              61,
              10,
              63,
              24,
              0,
              0,
              0,
              0
            ],
            [
              63,
              24,
              63,
              30,
              0,
              0,
              0,
              0
            ],
            [
              65,
              9,
              65,
              25,
              0,
              0,
              0,
              0
            ],
            [
              65,
              28,
              65,
              77,
              0,
              0,
              0,
              0
            ],
            [
              65,
              77,
              65,
              78,
              0,
              0,
              0,
              0
            ],
            [
              65,
              79,
              68,
              62,
              0,
              0,
              0,
              0
            ],
            [
              68,
              62,
              68,
              63,
              0,
              0,
              0,
              0
            ],
            [
              69,
              13,
              69,
              31,
              0,
              0,
              0,
              0
            ],
            [
              70,
              10,
              70,
              11,
              0,
              0,
              0,
              0
            ],
            [
              71,
              9,
              71,
              15,
              0,
              0,
              0,
              0
            ],
            [
              71,
              15,
              71,
              16,
              0,
              0,
              0,
              0
            ],
            [
              72,
              5,
              72,
              25,
              0,
              0,
              0,
              0
            ],
            [
              73,
              1,
              73,
              2,
              0,
              0,
              0,
              0
            ]
          ]
        },
        // ...

note:

            [
              42,
              9,
              45,
              55,
              0,
              0,
              0,
              0
            ],

and

#[derive(Copy, Clone, Serialize, Deserialize)]
#[cfg_attr(test, serde(deny_unknown_fields))]
pub(crate) struct Region(
    /* LineStart */ pub(crate) u64,
    /* ColumnStart */ pub(crate) u64,
    /* LineEnd */ pub(crate) u64,
    /* ColumnEnd */ pub(crate) u64,
    /* ExecutionCount */ pub(crate) u64,
    /* FileID */ pub(crate) u64,
    /* ExpandedFileID */ pub(crate) u64,
    /* Kind */ pub(crate) u64,
);

impl Region {
    pub(crate) fn line_start(&self) -> u64 {
        self.0
    }

    pub(crate) fn column_start(&self) -> u64 {
        self.1
    }

    pub(crate) fn line_end(&self) -> u64 {
        self.2
    }

    pub(crate) fn column_end(&self) -> u64 {
        self.3
    }

    pub(crate) fn execution_count(&self) -> u64 {
        self.4
    }

    pub(crate) fn file_id(&self) -> u64 {
        self.5
    }

    pub(crate) fn expanded_file_id(&self) -> u64 {
        self.6
    }

    pub(crate) fn kind(&self) -> u64 {
        self.7
    }
}

so this is equivalent to a Region with line 42-45 with 0 execution count. I think this is an issue with --json not --codecov unless I understand incorrectly.

andrewgazelka commented 1 year ago

It is weird that .html does not have it highlighted though...

andrewgazelka commented 1 year ago

Ah I think I found the issue.. let me test