search

microsoft / sarif-sdk

.NET code and supporting files for working with the 'Static Analysis Results Interchange Format' (SARIF, see https://github.com/oasis-tcs/sarif-spec)
Other
191 stars 88 forks source link

List exact test files which actual/expected sarif results are different in FileDiffing tests #2750

Closed yongyan-gh closed 7 months ago

yongyan-gh commented 7 months ago

Description

Current the End-To-End test output all files names in the TestData folder, after the change, it only list the test files the analysis results don't match the expected results.

E.g. current test failure message:

Expected output.Length to be 0 because there should be no unexpected diffs detected comparing actual results to 'SEC101_001.HttpAuthorizationRequestHeader.ps1, SEC101_002.GoogleOAuthCredentials.ps1, SEC101_003.GoogleApiKey.txt, SEC101_004.FacebookAppCredentials.ps1, SEC101_006.GitHubClassicPat.ps1, SEC101_007.GitHubAppCredentials.ps1, SEC101_010.SquarePat.ps1, SEC101_011.SquareCredentials.ps1, SEC101_012.SlackWebhook.txt, SEC101_013.PemPrivateKey.txt, SEC101_014.FacebookAccessToken.ps1, SEC101_015.AkamaiCredentials.ps1, SEC101_017.NpmLegacyAuthorToken.ps1, SEC101_018.TwilioCredentials.ps1, SEC101_019.PicaticApiKey.ps1, SEC101_020.DropboxAccessToken.ps1, SEC101_021.DropboxAppCredentials.ps1, SEC101_022.PayPalBraintreeAccessToken.ps1, SEC101_024.TwilioApiKey.ps1, SEC101_026.MailgunApiCredentials.ps1, SEC101_027.MailChimpApiKey.ps1, SEC101_028.PlaintextPassword.ps1, SEC101_030.GoogleServiceAccountKey.ps1, SEC101_032.GpgCredentials.ps1, SEC101_033.MongoDbCredentials.ps1, SEC101_034.CredentialObject.ps1, SEC101_035.CloudantCredentials.ps1, SEC101_040.ShopifySharedSecret.ps1, SEC101_041.RabbitMqCredentials.ps1, SEC101_042.DynatraceToken.ps1, SEC101_043.NuGetCredentials.ps1, SEC101_046.DiscordApiCredentials.ps1, SEC101_047.CratesApiKey.ps1, SEC101_048.SlackWorkflowKey.ps1, SEC101_049.TelegramBotToken.ps1, SEC101_052.StripeLiveRestrictedApiKey.txt, SEC101_053.StripeTestRestrictedApiKey.txt, SEC101_054.DerPrivateKey.der, SEC101_055.Pkcs12PrivateKey_P12_NoPasswordSelfSigned.p12, SEC101_055.Pkcs12PrivateKey_PemEncoded.txt, SEC101_055.Pkcs12PrivateKey_Pfx_NoPassword.pfx, SEC101_055.Pkcs12PrivateKey_Pfx_PasswordProtected.pfx, SEC101_056.OpenSshPrivateKey.txt, SEC101_057.PuttyPrivateKey.txt, SEC101_058.PgpPrivateKey.txt, SEC101_059.RsaPrivateKey.txt, SEC101_101.AadClientAppLegacyCredentials.xml, SEC101_102.AzureDevOpsPersonalAccessToken.ps1, SEC101_103.AzureCacheForRedisLegacyKey.ps1, SEC101_104.AzureCosmosDBLegacyKey.ps1, SEC101_105.AzureServiceBusLegacyCredentials.ps1, SEC101_105.AzureServiceBusLegacyCredentialsMultiline.ps1, SEC101_106.AzureStorageAccountLegacyCredentials.ps1, SEC101_107.SharedAccessSignature.ps1, SEC101_109.AzureContainerRegistryLegacyKey.ps1, SEC101_110.AzureDatabrickPat.ps1, SEC101_111.AzureSearchLegacyKey.ps1, SEC101_112.AzureFunctionLegacyKey.ps1, SEC101_113.AzureBatchLegacyKey.ps1, SEC101_114.AzureIotHubLegacyCredentials.ps1, SEC101_115.AzureIotDeviceProvisioningLegacyCredentials.ps1, SEC101_116.AzureMapsKey.ps1, SEC101_117.AzureSignalRKey.ps1, SEC101_118.AzureBlockchainKeyOrPassword.ps1, SEC101_119.AzureEventGridKey.ps1, SEC101_120.AzureWebAppBotCredentials.ps1, SEC101_121.AzureServiceDeploymentCredentials.xml, SEC101_122.AspNetMachineKey.ps1, SEC101_123.AzureMLWebServiceKey.ps1, SEC101_124.VisualStudioAppCenterKey.ps1, SEC101_125.AzureCognitiveServicesKey.ps1, SEC101_126.NetworkCredentials.ps1, SEC101_127.UrlCredentials.ps1, SEC101_128.BingMapsKey.ps1, SEC101_130.AadUserCredentials.ps1, SEC101_131.AzureAppConfigurationCredentials.ps1, SEC101_134.AzureCognitiveServicesTranslatorKey.ps1, SEC101_135.BingSearchKey.ps1, SEC101_136.AzureMixedRealityCredentials.ps1, SEC101_137.AzureCommunicationServicesKey.ps1, SEC101_138.AzureApplicationInsightsCredentials.ps1, SEC101_140.AzureGenomicsKey.ps1, SEC101_141.AzureHDInsightCredentials.ps1, SEC101_142.AzureWebAppBotKey.ps1, SEC101_143.BingApiKey.ps1, SEC101_144.AzureWebPubSubCredentials.ps1, SEC101_145.CiscoLocalAccountCredentials.ps1, SEC101_146.AzureIotDeviceLegacyCredentials.ps1, SEC101_147.OfficeIncomingWebhook.ps1, SEC101_148.AzureDevOpsOAuthTokenValidator.ps1, SEC101_149.AzureStorageSas.ps1, SEC101_150.AzureLogicAppSas.ps1, SEC101_151.AzureCdnSas.ps1'.To compare all difference for this test suite:
C:\Users\sumukherjee\AppData\Roaming\DiffSecurePlaintextSecretsInternal.cmd

after the change:

    Expected output.Length to be 0
    because there are unexpected diffs detected comparing actual results to expected results for following test files:
     - SEC101_006.GitHubClassicPat.ps1
     - SEC101_007.GitHubAppCredentials.ps1

    To compare all difference for this test suite:
    C:\Users\v-yongyan\AppData\Roaming\DiffSecurePlaintextSecretsInternal.cmd

    To rebaseline with current behavior:
    C:\Users\v-yongyan\AppData\Roaming\RebaselineSecurePlaintextSecretsInternal.cmd