Coverage XML sometimes checked too early

[timgates42]

I could have missed something so my apologies if there is an easier fix but I have noticed that depending on the order I install pytest-azurepipelines or pytest-cov via pip that sometimes the pytest11 plugin hook invoke coverage XML production after the pytest-azurepipeline session finish hook, meaning it thinks the coverage XML is not present. This is less a problem when using pip but if using pipenv it can be more difficult to control install order.

Output observed - skip upload appears before coverage report:

##[warning]Coverage XML was not created, skipping upload.

---------- coverage: platform linux2, python 2.7.15-final-0 ----------

[timgates42]

Oh and thanks for such a great library :)

[basgas]

I run in to the same/similar problem. I run pytest in a docker container.

output

[error]Unable to process command '##vso[codecoverage.publish codecoveragetool=Cobertura;summaryfile=/gherkin/coverage.xml;reportdirectory=/gherkin/htmlcov;]' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)

[error]File '/gherkin/coverage.xml' does not exist or is not accessible.

My pipeline step:

  - task: Bash@3
    displayName: Run Unittests
    inputs:
      targetType: 'inline'
      script: |
        # Write your commands here

        echo 'Testing docker'
        docker run --rm -v $(System.DefaultWorkingDirectory):/staging \
        $(containerRegistry)/$(imageRepository):$(branchTag) \
        pytest ./utils --cov utils --cov-report html --junitxml=test-gherkin.xml

I changed the order of the modules in my requirements file, but that did not change the behaviour.

[PyDataBlog]

I run in to the same/similar problem. I run pytest in a docker container.

output

[error]Unable to process command '##vso[codecoverage.publish codecoveragetool=Cobertura;summaryfile=/gherkin/coverage.xml;reportdirectory=/gherkin/htmlcov;]' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)

[error]File '/gherkin/coverage.xml' does not exist or is not accessible.

My pipeline step:

  - task: Bash@3
    displayName: Run Unittests
    inputs:
      targetType: 'inline'
      script: |
        # Write your commands here

        echo 'Testing docker'
        docker run --rm -v $(System.DefaultWorkingDirectory):/staging \
        $(containerRegistry)/$(imageRepository):$(branchTag) \
        pytest ./utils --cov utils --cov-report html --junitxml=test-gherkin.xml

I changed the order of the modules in my requirements file, but that did not change the behaviour.

Did you ever solve this? I have the exact issue.

[PyDataBlog]

I could have missed something so my apologies if there is an easier fix but I have noticed that depending on the order I install pytest-azurepipelines or pytest-cov via pip that sometimes the pytest11 plugin hook invoke coverage XML production after the pytest-azurepipeline session finish hook, meaning it thinks the coverage XML is not present. This is less a problem when using pip but if using pipenv it can be more difficult to control install order.

Output observed - skip upload appears before coverage report:

##[warning]Coverage XML was not created, skipping upload.

---------- coverage: platform linux2, python 2.7.15-final-0 ----------

@timgates42 is there any way of getting this to work? It checks out too early and then fails to find the generated coverage files

[timgates42]

I had a go at fixing it in #34 but the PR failed verification due to what looked like CI credentials not being setup in the automation correctly

[Kabir-Khan]

I am running into same issue running from docker image in Azure Pipeline Build. Will this be fixed soon ? As you can see on line 1 from the below code coverage check is at 88% when the error occurs. It does not happen when running outside the docker or locally. Any help will be appreciated. Thanks.

[exec] #11 6.915 src/tests/unit/test_job_name.py ... [ 88%] Result Attachments will be stored in LogStore Run Attachments will be stored in LogStore Failed to parse result files: System.IO.DirectoryNotFoundException: Could not find a part of the path '/app/test-output.xml'. at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func'2 errorRewriter) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode) at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy) at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn) at System.Xml.XmlTextReaderImpl.FinishInitUriString() at System.Xml.XmlTextReaderImpl..ctor(String uriStr, XmlReaderSettings settings, XmlParserContext context, XmlResolver uriResolver) at System.Xml.XmlReaderSettings.CreateReader(String inputUri, XmlParserContext inputContext) at System.Xml.XmlReader.Create(String inputUri, XmlReaderSettings settings) at Microsoft.TeamFoundation.TestClient.PublishTestResults.NUnitResultParser.ParseTestResultFile(TestRunContext runContext, String filePath) at Microsoft.TeamFoundation.TestClient.PublishTestResults.NUnitResultParser.<>c__DisplayClass1_0.b__0(String file) at System.Linq.Enumerable.SelectListIterator'2.MoveNext() at System.Linq.Enumerable.WhereEnumerableIterator'1.ToList() at Microsoft.TeamFoundation.TestClient.PublishTestResults.NUnitResultParser.ParseTestResultFiles(TestRunContext runContext, IList'1 resultFilePaths) at Microsoft.VisualStudio.Services.Agent.Worker.TestResults.Parser.ParseFiles(IExecutionContext executionContext, TestRunContext testRunContext, List'1 testResultsFiles, ITestResultParser testResultParser) Reading code coverage summary from '/app/coverage.xml'

[error]Unable to process command ' [exec] #11 6.915 ##vso[codecoverage.publish codecoveragetool=Cobertura;summaryfile=/app/coverage.xml;reportdirectory=/app/htmlcov;]' successfully. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=817296)

[error]File '/app/coverage.xml' does not exist or is not accessible.

[exec] #11 6.915 [exec] #11 6.915 [exec] #11 6.915 ---------------- generated Nunit xml file: /app/test-output.xml ---------------- [exec] #11 6.915 ------------ generated xml file: /app/src/tests/unit/unit_tests.xml ------------

[Kabir-Khan]

I believe I have a work around. You can add --no-coverage-upload directive to your pytest command and the error will go away as it will stop the vso publish altogether to Azure Pipeline. You will have to copy the htmlcov and coverage.xml from docker container to $(System.DefaultWorkingDirectory) or somewhere accessible by the pipeline for publishing task PublishCodeCoverageResults@1.

[leonids2005]

Looks like we have the same problem :( ... any idea when it could be fixed?

[enrique-ayala]

Same here , receiving DirectoryNotFoundException

2022-06-16T23:14:52.4135713Z [command]/bin/bash --noprofile --norc /agent/_work/_temp/d309be40-198f-4557-8742-2ee3b7e18b4b.sh
2022-06-16T23:14:53.3949384Z ============================= test session starts ==============================
2022-06-16T23:14:53.3951057Z platform linux -- Python 3.9.10, pytest-7.1.2, pluggy-1.0.0
2022-06-16T23:14:53.3951749Z rootdir: /dummy-workdir
2022-06-16T23:14:53.3952347Z plugins: azurepipelines-1.0.3, nunit-1.0.0
2022-06-16T23:14:53.3952748Z collected 1 item
2022-06-16T23:14:53.3952894Z 
2022-06-16T23:15:02.6605264Z Result Attachments will be stored in LogStore
2022-06-16T23:15:02.7152146Z Run Attachments will be stored in LogStore
2022-06-16T23:15:02.7808350Z Failed to parse result files: System.IO.DirectoryNotFoundException: Could not find a part of the path

[yardstick17]

The issue still exists:

============================= test session starts ==============================
platform linux -- Python 3.7.13, pytest-7.1.2, pluggy-1.0.0
rootdir: /usr/src/app, configfile: pytest.ini
plugins: anyio-3.6.2, nunit-1.0.3, azurepipelines-1.0.4, cov-4.0.0
collected 13 items

tests/test_main.py ..                                                    [ 15%]
tests/abc****_tests/test_api.py ....                                  [ 46%]
tests/abc****_tests/test_abc.py .                                     [ 53%]
Result Attachments will be stored in LogStore
Run Attachments will be stored in LogStore
Failed to parse result files: System.IO.FileNotFoundException: Could not find file '/usr/src/app/test-output.xml'.
File name: '/usr/src/app/test-output.xml'
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
   at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials, IWebProxy proxy, RequestCachePolicy cachePolicy)
   at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
   at System.Xml.XmlTextReaderImpl.FinishInitUriString()
   at System.Xml.XmlTextReaderImpl..ctor(String uriStr, XmlReaderSettings settings, XmlParserContext context, XmlResolver uriResolver)
   at System.Xml.XmlReaderSettings.CreateReader(String inputUri, XmlParserContext inputContext)
   at System.Xml.XmlReader.Create(String inputUri, XmlReaderSettings settings)
   at Microsoft.TeamFoundation.TestClient.PublishTestResults.NUnitResultParser.ParseTestResultFile(TestRunContext runContext, String filePath)
   at Microsoft.TeamFoundation.TestClient.PublishTestResults.NUnitResultParser.<>c__DisplayClass1_0.<ParseTestResultFiles>b__0(String file)
   at System.Linq.Enumerable.SelectListIterator`2.MoveNext()
   at System.Linq.Enumerable.WhereEnumerableIterator`1.ToList()
   at Microsoft.TeamFoundation.TestClient.PublishTestResults.NUnitResultParser.ParseTestResultFiles(TestRunContext runContext, IList`1 resultFilePaths)
   at Microsoft.VisualStudio.Services.Agent.Worker.TestResults.Parser.ParseFiles(IExecutionContext executionContext, TestRunContext testRunContext, List`1 testResultsFiles, ITestResultParser testResultParser)
##[warning]Coverage XML was not created, skipping upload.

------------ generated Nunit xml file: /usr/src/app/test-output.xml ------------
--------- generated xml file: /usr/src/app/tests/junit/test-output.xml ---------
============================= 13 passed in 10.16s ==============================
Async Command Start: Publish test results
Async Command End: Publish test results
Finishing: Run Python Tests

https://stackoverflow.com/questions/74384658/coverage-report-not-published-using-docker-and-pytest-azurepipelines

Is anyone able to get the pytest results published from the Docker container?

(copying the junit/xml files from the container to runner leads to a broken filepath in the report. You can still open the report but won't see the line-wise coverage)

[wfranceys]

+1 Also seeing this

[NathanEmb]

+1 still exists

[pythonmonty]

+1 issue still persists