search

codeclimate / test-reporter

Code Climate Test Reporter
MIT License
156 stars 76 forks source link

Error finding jacoco.xml even though it's there #361

Closed toymachiner62 closed 6 years ago

toymachiner62 commented 6 years ago

Here's some output from jenkins running a Docker file. The jacoco.xml is clearly there as you can seen from step 30

Step 30/33 : RUN ls -al /home/gradle/project/build/reports/jacoco/test
 ---> Running in ca6a589fe85c
total 892
drwxr-sr-x    1 gradle   root            52 Sep 18 03:02 .
drwxr-sr-x    1 gradle   root            18 Sep 18 03:02 ..
drwxr-sr-x   13 gradle   root          4096 Sep 18 03:02 html
-rw-r--r--    1 gradle   root        905276 Sep 18 03:02 jacoco.xml
Removing intermediate container ca6a589fe85c
 ---> f74cdc021006
Step 31/33 : RUN ./cc-test-reporter after-build -d -t jacoco -p /home/gradle/project/build/reports/jacoco/test
 ---> Running in 7b1f1e826d65
time="2018-09-18T03:03:01Z" level=debug msg="about to run format-coverage" 
time="2018-09-18T03:03:01Z" level=debug msg="using formatter jacoco" 
time="2018-09-18T03:03:01Z" level=debug msg="checking search path  for jacoco formatter" 
time="2018-09-18T03:03:01Z" level=debug msg="checking search path jacoco.xml for jacoco formatter" 
time="2018-09-18T03:03:01Z" level=error msg="could not find coverage file \ncould not find any files in search paths for jacoco. search paths were: , jacoco.xml" 
Error: could not find any files in search paths for jacoco. search paths were: , jacoco.xml
Usage:
  cc-test-reporter after-build [flags]

Flags:
  -s, --batch-size int               batch size for source files (default 500)
  -e, --coverage-endpoint string     endpoint to upload coverage information to (default "https://api.codeclimate.com/v1/test_reports")
  -t, --coverage-input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov]
      --exit-code int                exit code of the test run
  -r, --id string                    reporter identifier (default "<my_id>")
      --insecure                     send coverage insecurely (without HTTPS)
  -p, --prefix string                the root directory where the coverage analysis was performed (default "/home/gradle/project")

Global Flags:
  -d, --debug   run in debug mode

The command '/bin/sh -c ./cc-test-reporter after-build -d -t jacoco -p /home/gradle/project/build/reports/jacoco/test' returned a non-zero code: 255
make: *** [build] Error 255

Is there a bug in the jacoco formatter or do I have a configuration incorrect?

toymachiner62 commented 6 years ago

Let me know if you need me to test out a snapshot version for you. Hoping to get this working today before my trial expires.

wfleming commented 6 years ago

I commented on the wrong tab & hit the wrong key, that's embarrassing for me, sorry for the noise.

@toymachiner62 The help text explaining the -p/--prefix option could be clearer. That option isn't a directory to search for the coverage file (the test reporter always looks relative to the current working directory), it's a path to strip off of the filenames listed in the coverage file so they'll be relative to the repository root. In some CI setups, a coverage file might end up containing a report for /home/ci/project/src/foo.groovy for a file called src/foo.groovy: our test-reporter needs to be able to match up entries to files in the repository relative to the repository root, so the prefix option is to allow manipulating those paths when necessary so they'll match up.

I see you already commented on https://github.com/codeclimate/test-reporter/issues/243. For Jacoco users, we suggest using the lower-level test-reporter commands, e.g.

./cc-test-reporter format-coverage -d -t jacoco build/reports/jacoco/test/jacoco.xml
./cc-test-reporter upload-coverage -d

You may still need to use the --add-prefix flag: it seems like jacoco usually reports paths that have some of the repository's filepath cut off. Also, the test reporter should be run from the root of the repository's checked out source code, it's not clear from your snippet if it is.

I'm also curious why it appears you're reporting test coverage as part of a docker build. I think this will work if everything is configured correctly, but it is an unusual choice.

toymachiner62 commented 6 years ago

Docker Multi Stage Builds - maybe a little uncommon right now since it's a little bleeding edge, but more and more people are using multi stage builds https://docs.docker.com/develop/develop-images/multistage-build/

I'll try the low level commands and report back

toymachiner62 commented 6 years ago

Still didn't work using those low level commands. Any idea why it's not seeing that file when it exists right where it's looking?

Step 30/34 : RUN ls -al /home/gradle/project/src/main/java/com/ciitizen/code/test
 ---> Running in daff30869274
total 72
drwxr-xr-x    2 gradle   root           246 Sep 12 19:13 .
drwxr-xr-x    3 gradle   root            18 Jun  3 19:51 ..
-rw-r--r--    1 gradle   root          1230 Jun  3 19:51 CollectionToCsvBridge.java
-rw-r--r--    1 gradle   root          5922 Jun 18 17:41 CopyConstructorTester.java
-rw-r--r--    1 gradle   root          7954 Jul 19 23:49 EqualsHashcodeTester.java
-rw-r--r--    1 gradle   root          6113 Jul 19 23:49 GetterSetterTester.java
-rw-r--r--    1 gradle   root         13661 Jul 19 23:49 ProxyTester.java
-rw-r--r--    1 gradle   root         13551 Sep 12 19:13 TestUtility.java
-rw-r--r--    1 gradle   root          4098 Jun  3 19:51 XmlSerializationTester.java
-rw-r--r--    1 gradle   root            69 Jun  3 19:51 package-info.java
Removing intermediate container daff30869274
 ---> 4322ad44e2c8
Step 31/34 : RUN ./cc-test-reporter format-coverage -d -t jacoco /home/gradle/project/build/reports/jacoco/test/jacocoTestReport.xml
 ---> Running in 72ab7ac100a0
time="2018-09-18T17:58:38Z" level=debug msg="coverage path /home/gradle/project/build/reports/jacoco/test/jacocoTestReport.xml" 
time="2018-09-18T17:58:38Z" level=debug msg="using formatter jacoco" 
time="2018-09-18T17:58:38Z" level=debug msg="checking search path /home/gradle/project/build/reports/jacoco/test/jacocoTestReport.xml for jacoco formatter" 
time="2018-09-18T17:58:38Z" level=info msg="trimming with prefix /home/gradle/project/" 
time="2018-09-18T17:58:38Z" level=debug msg="getting fallback blob_id for source file com/ciitizen/code/test/CopyConstructorTester.java" 
time="2018-09-18T17:58:38Z" level=error msg="failed to read file com/ciitizen/code/test/CopyConstructorTester.java\nopen com/ciitizen/code/test/CopyConstructorTester.java: no such file or directory" 
Error: open com/ciitizen/code/test/CopyConstructorTester.java: no such file or directory
Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/gradle/project")

Global Flags:
  -d, --debug   run in debug mode

The command '/bin/sh -c ./cc-test-reporter format-coverage -d -t jacoco /home/gradle/project/build/reports/jacoco/test/jacocoTestReport.xml' returned a non-zero code: 255
make: *** [build] Error 255
wfleming commented 6 years ago

What is the current working directory when you run cc-test-reporter format-coverage? The test-reporter should be run from the root of the checked out repository so it can read git data if necessary. The fact that you're providing an absolute path makes me think that's not the working directory.

Error: open com/ciitizen/code/test/CopyConstructorTester.java: no such file or directory

What's your directory structure under the root of your repository? I suspect that file actually lives at something like src/com/ciitizen/code/test/CopyConstructorTester.java: this is what you'd want to use the --add-prefix flag for mentioned above and in https://github.com/codeclimate/test-reporter/issues/243#issuecomment-339375322. The jacoco coverage format reports files from the root of the source directory, not the root of the repository, which is what our test-reporter needs, so this flag is used to fix those paths so they'll be relative to the root of the repository. E.g. if your com/ directory was under a src/ directory at the root of your repository, you'd want to use --add-prefix src.

toymachiner62 commented 6 years ago

There is no .git folder b/c we are not copying that to our Dockerfile when we do our build (which is where we run our tests). We pass in the all the git environment variables that the cc-test-reporter needs (I had to dig into the source code to find them).

I suspect that file actually lives at something like src/com/ciitizen/code/test/CopyConstructorTester.java

Actually from the project root it's src/main/java/com/ciitizen/code/test/CopyConstructorTester.java. The project root is in /home/gradle/project

I'm not sure I followed the rest of your comment. Are you suggesting that I use the --add-prefix param with either the format-coverage or upload-coverage scripts?

For reference this is the file system hierarchy of where CopyConstructorTester.java lives

/home
    /gradle
        /project <-- Project root
            /build
                /reports
                    /jacoco
                        /test
                            /jacocoTestReport.xml
            /src
                /main
                    /java
                        /com
                            /ciitizen
                                /code
                                    /test
                                        /CopyConstructorTester.java
toymachiner62 commented 6 years ago

Here's my docker commands it's it's respective output

...
RUN pwd
RUN ./cc-test-reporter after-build -d -t jacoco -p /home/gradle/project/build/reports/jacoco/test
...
...
Step 31/34 : RUN pwd
 ---> Running in 0cb38dc7c6f2
/home/gradle/project
Removing intermediate container 0cb38dc7c6f2
 ---> 46cf7539315a
Step 32/34 : RUN ./cc-test-reporter after-build -d -t jacoco -p /home/gradle/project/build/reports/jacoco/test
 ---> Running in 244b5841eb79
time="2018-09-18T20:41:57Z" level=debug msg="about to run format-coverage" 
time="2018-09-18T20:41:57Z" level=debug msg="using formatter jacoco" 
time="2018-09-18T20:41:57Z" level=debug msg="checking search path  for jacoco formatter" 
time="2018-09-18T20:41:57Z" level=debug msg="checking search path jacoco.xml for jacoco formatter" 
time="2018-09-18T20:41:57Z" level=error msg="could not find coverage file \ncould not find any files in search paths for jacoco. search paths were: , jacoco.xml" 
Error: could not find any files in search paths for jacoco. search paths were: , jacoco.xml
Usage:
  cc-test-reporter after-build [flags]

Flags:
  -s, --batch-size int               batch size for source files (default 500)
  -e, --coverage-endpoint string     endpoint to upload coverage information to (default "https://api.codeclimate.com/v1/test_reports")
  -t, --coverage-input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov]
      --exit-code int                exit code of the test run
  -r, --id string                    reporter identifier (default "a98632fb4ae9e9eb1c2da9455df1d655b21e5d10771aaa87d0aafd801f579690")
      --insecure                     send coverage insecurely (without HTTPS)
  -p, --prefix string                the root directory where the coverage analysis was performed (default "/home/gradle/project")

Global Flags:
  -d, --debug   run in debug mode

The command '/bin/sh -c ./cc-test-reporter after-build -d -t jacoco -p /home/gradle/project/build/reports/jacoco/test' returned a non-zero code: 255
make: *** [build] Error 255
toymachiner62 commented 6 years ago

Did you mean to add the --add-prefix flag to the format-coverage command like this?

RUN ./cc-test-reporter format-coverage -d -t jacoco --add-prefix /home/gradle/project/build/reports/jacoco/test/jacocoTestReport.xml
wfleming commented 6 years ago

Actually from the project root it's src/main/java/com/ciitizen/code/test/CopyConstructorTester.java. The project root is in /home/gradle/project

Then you want the current working directory when you run cc-test-reporter to be /home/gradle/project, and I believe you'd need to pass --add-prefix src/main/java/ to format-coverage

Your format coverage invocation should probably be

./cc-test-reporter format-coverage -d -t jacoco --add-prefix src/main/java/ build/reports/jacoco/test/jacocoTestReport.xml
toymachiner62 commented 6 years ago

How do I set the current working directory with the format-coverage command?

wfleming commented 6 years ago

How do I set the current working directory with the format-coverage command?

Since you're doing this with a docker build, I think a WORKDIR command would be the most direct way.

toymachiner62 commented 6 years ago

I'm already doing that. WORKDIR /home/gradle/project is at the top of my Dockerfile

toymachiner62 commented 6 years ago

What exactly is the --add-prefix flag supposed to be, the dir where the report file lives or the path the the file itself? The documented help is so vague and unclear

I'm very confused on what --add-prefix is supposed to be vs --prefix

Step 32/36 : RUN ./cc-test-reporter format-coverage --help
 ---> Running in 2b90c0eac6a1
Locate, parse, and re-format supported coverage sources.

Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/gradle/project")
toymachiner62 commented 6 years ago

Here's the output when running RUN ./cc-test-reporter format-coverage -d -t jacoco --add-prefix /home/gradle/project/build/reports/jacoco/test/jacoco.xml

Step 33/36 : RUN ./cc-test-reporter format-coverage -d -t jacoco --add-prefix /home/gradle/project/build/reports/jacoco/test/jacoco.xml
 ---> Running in 040b12eb8043
time="2018-09-18T21:05:20Z" level=debug msg="using formatter jacoco" 
time="2018-09-18T21:05:20Z" level=debug msg="checking search path  for jacoco formatter" 
time="2018-09-18T21:05:20Z" level=debug msg="checking search path jacoco.xml for jacoco formatter" 
time="2018-09-18T21:05:20Z" level=error msg="could not find coverage file \ncould not find any files in search paths for jacoco. search paths were: , jacoco.xml" 
Error: could not find any files in search paths for jacoco. search paths were: , jacoco.xml
Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/gradle/project")

Global Flags:
  -d, --debug   run in debug mode

The command '/bin/sh -c ./cc-test-reporter format-coverage -d -t jacoco --add-prefix /home/gradle/project/build/reports/jacoco/test/jacoco.xml' returned a non-zero code: 255
make: *** [build] Error 255
wfleming commented 6 years ago

What exactly is the --add-prefix flag supposed to be, the dir where the report file lives or the path the the file itself? The documented help is so vague and unclear

I'm very confused on what --add-prefix is supposed to be vs --prefix

--prefix should perhaps have been called --strip-prefix, but for backwards compatibility reasons I think we're stuck with --prefix for now.

The two --*-prefix flags exist for cases where the file names as reported in the coverage report (jacoco.xml, in your case) are not relative to the project root. Commonly either because the coverage report is reporting relative to a subdirectory (your case), so a prefix needs to be added to the file paths to make them relative to the project root, or sometimes because the coverage report's filenames are absolute paths, in which case some of the reported path should be deleted to correct the paths (what --prefix is for). These options do not have anything to do with where the coverage report is found. They are for helping the test-reporter "fix" source paths within the coverage report.

Can you please try the invocation suggested earlier in https://github.com/codeclimate/test-reporter/issues/361#issuecomment-422551355 ?

toymachiner62 commented 6 years ago

Also not sure if this is a gradle specific thing, but the output file is jacocoTestReport.xml and not jacoco.xml which is what format-coverage seems to be looking for. Is there a way to explicitly say what file to look for?

toymachiner62 commented 6 years ago

Giving your suggestion from #361 a go now..

wfleming commented 6 years ago

the output file is jacocoTestReport.xml and not jacoco.xml which is what format-coverage seems to be looking for. Is there a way to explicitly say what file to look for?

Yes, there is. The argument to format-coverage is an explicit path to the coverage file. I suggested the command in https://github.com/codeclimate/test-reporter/issues/361#issuecomment-422551355 based on you referring to jacoco.xml earlier in this issue and the ls output showing the file with that name in your initial report. But if the file is now named something else, you'd want to adjust the path passed as the argument to format-coverage.

toymachiner62 commented 6 years ago

Sorry for the confusion. I was trying to hack some magic within gradle to rename jacocoTestReport.xml to jacoco.xml, but that was a bad failed attempt.

Ok with your suggestion of RUN ./cc-test-reporter format-coverage -d -t jacoco --add-prefix src/main/java/ build/reports/jacoco/test/jacocoTestReport.xml

I get 

Step 33/36 : RUN ./cc-test-reporter format-coverage -d -t jacoco --add-prefix src/main/java/ build/reports/jacoco/test/jacocoTestReport.xml
 ---> Running in 073bcf9bf1ab
time="2018-09-18T21:21:55Z" level=debug msg="coverage path build/reports/jacoco/test/jacocoTestReport.xml" 
time="2018-09-18T21:21:55Z" level=debug msg="using formatter jacoco" 
time="2018-09-18T21:21:55Z" level=debug msg="checking search path build/reports/jacoco/test/jacocoTestReport.xml for jacoco formatter" 
time="2018-09-18T21:21:55Z" level=info msg="trimming with prefix /home/gradle/project/" 
time="2018-09-18T21:21:55Z" level=debug msg="getting fallback blob_id for source file com/ciitizen/code/test/CopyConstructorTester.java" 
time="2018-09-18T21:21:55Z" level=error msg="failed to read file com/ciitizen/code/test/CopyConstructorTester.java\nopen com/ciitizen/code/test/CopyConstructorTester.java: no such file or directory" 
Error: open com/ciitizen/code/test/CopyConstructorTester.java: no such file or directory
Usage:
  cc-test-reporter format-coverage [coverage file] [flags]

Flags:
      --add-prefix string   add this prefix to file paths
  -t, --input-type string   type of input source to use [clover, cobertura, coverage.py, excoveralls, gcov, gocov, jacoco, lcov, simplecov]
  -o, --output string       output path (default "coverage/codeclimate.json")
  -p, --prefix string       the root directory where the coverage analysis was performed (default "/home/gradle/project")

Global Flags:
  -d, --debug   run in debug mode

The command '/bin/sh -c ./cc-test-reporter format-coverage -d -t jacoco --add-prefix src/main/java/ build/reports/jacoco/test/jacocoTestReport.xml' returned a non-zero code: 255
make: *** [build] Error 255
wfleming commented 6 years ago

Ok, there is a confusing piece of behavior regarding --add-prefix that I misunderstood, my apologies, @toymachiner62. My colleague @ale7714 has corrected me.

The confusing behavior is that --add-prefix does alter the path of the file reported to our API, which is necessary in your case, but it does not change where the test-reporter attempts to read the file from locally (which it does to calculate git blob IDs). So the invocation is mostly correct, but you want the WORKDIR to be /home/gradle/project/src/main/java, and then the path to the XML file needs to be adjusted accordingly. E.g.

WORKDIR /home/gradle/project/src/main/java
RUN ./cc-test-reporter format-coverage -d -t jacoco --add-prefix src/main/java/ ../../../build/reports/jacoco/test/jacocoTestReport.xml
toymachiner62 commented 6 years ago

Ok this is working now with your suggestion of

WORKDIR /home/gradle/project/src/main/java
RUN ./cc-test-reporter format-coverage -d -t jacoco --add-prefix src/main/java/ ../../../build/reports/jacoco/test/jacocoTestReport.xml

Is there any way we can make this easier for others and/or my future self when trying to setup jacoco test coverage?

Also i would recommend documentation all the commands and options available along with examples for the cc-test-reporter tool.