pointfreeco / swift-snapshot-testing

📸 Delightful Swift snapshot testing.
https://www.pointfree.co/episodes/ep41-a-tour-of-snapshot-testing
MIT License
3.74k stars 570 forks source link

Azure Devops Tests succeeding while fail expected #712

Closed Mittchel closed 1 year ago

Mittchel commented 1 year ago

Hi,

We've been using swift-snapshot for a while now and locally everything seems to work correctly. Now we host our source on Azure Devops and manage our CI/CD there. We've checked in our Snapshots (isRecording = true) in main. I've modified the source code (changed some texts) so tests are failing.

However, when running Xcodebuild in Azure Devops, all the tests are succeeding when they fail locally and I expect them to fail on the pipeline as well. When extracting the .xcresult with XCParse to get the screenshots I've noticed there are no screenshots in there.

I am wondering if someone knows whats causing this or is able to point me into the right direction to debug this.

Azure Devops task (removed some personal info):

- task: Xcode@5
  inputs:
    actions: 'clean build test archive'
    configuration: '${{ parameters.configuration }}'
    sdk: '$(sdk)'
    scheme: '${{ parameters.scheme }}'
    xcodeVersion: '14'
    packageApp: true
    signingOption: 'manual'
    signingIdentity: '__signingidentity__
    provisioningProfileUuid: ${{ parameters.provisioningUuid }}
    destinationPlatformOption: 'iOS'
    destinationSimulators: 'iPhone 14 Pro,OS=16.2'
    args: '-verbose -derivedDataPath $(Build.BinariesDirectory)/derived'
    publishJUnitResults: true
    useXcpretty: false

Produces the following xcodebuild command (removed placeholders): xcodebuild -sdk iphoneos -configuration TST -workspace ../project.xcworkspace -scheme Release -destination platform=iOS Simulator,name=iPhone 14 Pro,OS=16.2 clean build test archive -verbose -derivedDataPath /Users/runner/work/1/b/derived CODE_SIGN_STYLE=Manual CODE_SIGN_IDENTITY=__signingIdentity__ PROVISIONING_PROFILE=__guid__ PROVISIONING_PROFILE_SPECIFIER=

If more information is needed please let me know. Thank you in advance!

mbrandonw commented 1 year ago

Hi, as this is not an issue with the library and most likely has to do with Azure, I'm going to move this to a discussion.