cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.16k stars 3.82k forks source link

roachtest: c2c/shutdown/src/coordinator failed #135794

Closed cockroach-teamcity closed 1 day ago

cockroach-teamcity commented 2 days ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.c2c/shutdown/src/coordinator failed with artifacts on release-24.1.7-rc @ 3613a619e327fb412ac62036ce2c7060c25e48db:

VMs preempted during the test run: teamcity-17851186-1732085248-23-n9cpu8-0004 (us-east1-d), teamcity-17851186-1732085248-23-n9cpu8-0005 (us-east1-d), teamcity-17851186-1732085248-23-n9cpu8-0006 (us-east1-d)

**Other Failures:**
(sql_runner.go:94).ExecWithMessage: error executing query="CREATE TENANT \"destination-tenant\" FROM REPLICATION OF \"src-tenant\" ON 'postgres://roachprod:cockroachdb@35.196.175.126:29000?sslcert=-----BEGIN+CERTIFICATE-----%0AMIIDJjCCAg6gAwIBAgIRAI8yeeKw6nPzbxIHJ3FRJb4wDQYJKoZIhvcNAQELBQAw%0AKzESMBAGA1UEChMJQ29ja3JvYWNoMRUwEwYDVQQDEwxDb2Nrcm9hY2ggQ0EwHhcN%0AMjQxMTE5MDcyNDU4WhcNMjkxMTI0MDcyNDU4WjAoMRIwEAYDVQQKEwlDb2Nrcm9h%0AY2gxEjAQBgNVBAMTCXJvYWNocHJvZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC%0AAQoCggEBANJI4T9YCRIeGz9ysof5UjVXYJq%2FLqN48WFdD5joqFxKN75B7IGm3Qg0%0AYIIqqkBiEvms1Fb7UNdWoCqRqqQ%2FAV5oynMkw%2B2HrOWR9g%2FG2k6ml7qh%2BsYKZ0Ak%0AaAfxRwgMSuIgCo012l%2FM%2B1luGEOiYTZLOjCIE1Q2yGQ34qKUvt1i%2FAqiPMYf9ocw%0AdvNzbfJlbILKyZuqJfLaF6NN%2Fr8Vf4K8B8vrviCwtPkdo9dZgIPmKZGTx%2FBetDha%0A6Q8EPnJmPdjDWtLBl37MsUsYA4Qwn9lEW41dXs12GlAL8hAEgGTdkkeTGDqzYSPz%0AkbCZ4FBLX4ExOB%2B7MGgmwrmZPzX0IQ0CAwEAAaNIMEYwDgYDVR0PAQH%2FBAQDAgWg%0AMBMGA1UdJQQMMAoGCCsGAQUFBwMCMB8GA1UdIwQYMBaAFFrrFecRLKOGCFCHzkpP%0AwC0QnEPQMA0GCSqGSIb3DQEBCwUAA4IBAQB1UznKwiaCLdTOQfEOEWv114rzytqI%0A%2BKoSwKXQ3afFZDV4a9bcmYSPs1Q%2BfFP0eUu%2FTcjlF2CohBMiC%2BrDZ%2BrLwHpPuWRv%0AGK%2BtNDlw6bSE2L6FHHs01V36i2c99%2Fs3lYVAZdGCR5Kgb3M%2BaPu9gMjHIn90b4uE%0APqpg9k6XkIX362NNGItgqk7eE5JYKldVlJd%2BMYfj35uM%2BGHMLhT%2FWKTHdLNOuBiv%0A83%2FocGAOWJobzOFX9OwO0xEAiGr0akk0q6SrDt6cWwWJYcqxbpnipiV6PKYXPsao%0AT8O9x%2BYRBo1PKSjW13SpwSBn%2FUMrN4KRRFjQx1AzOT6TD6upLPjjGhxw%0A-----END+CERTIFICATE-----%0A&sslinline=true&sslkey=-----BEGIN+RSA+PRIVATE+KEY-----%0AMIIEowIBAAKCAQEA0kjhP1gJEh4bP3Kyh%2FlSNVdgmr8uo3jxYV0PmOioXEo3vkHs%0AgabdCDRggiqqQGIS%2BazUVvtQ11agKpGqpD8BXmjKcyTD7Yes5ZH2D8baTqaXuqH6%0AxgpnQCRoB%2FFHCAxK4iAKjTXaX8z7WW4YQ6JhNks6MIgTVDbIZDfiopS%2B3WL8CqI8%0Axh%2F2hzB283Nt8mVsgsrJm6ol8toXo03%2BvxV%2FgrwHy%2Bu%2BILC0%2BR2j11mAg%2BYpkZPH%0A8F60OFrpDwQ%2BcmY92MNa0sGXfsyxSxgDhDCf2URbjV1ezXYaUAvyEASAZN2SR5MY%0AOrNhI%2FORsJngUEtfgTE4H7swaCbCuZk%2FNfQhDQIDAQABAoIBAD8%2FrIBg%2Bll4tLlg%0AxlIfq5pfra7d0ilZN0ERvJ9BW2Jm8F1JdtmIVvMgzeZY7txuo0Z1tWh0LtyWMZac%0AObctPDwpSLtZU5UeOeLTBC%2B6n%2B8ILJr2HXdzqZqhWKkK6VKCYAOjn4jZqdUm0nMT%0Aqjitno%2B2EMqII0gxMhTyujgshTwQ%2FbIOy8BZ91AFoeLlcjfrfekfYfvWyxjMz32a%0A9aYWVMh%2FESRsj7dw6rMewWmFpYm4j7fWM5exfUZ%2FugKnSK3V4%2F9gwFUo2n1WyJ79%0Abi2NQ%2B0seLsu5uNwDHcYjAZOo7gyVyz4rEOdNZw6lnonJxtIP%2FZ3iVRuh1cCFMaV%0AvZFrYAECgYEA%2B%2B7RZFkZku4jYl%2FuadCod9so1KLACzPTuMBrIklpULQZf71vKKpM%0AFf58n1dVtGb%2BcPAdNJQri0DCgxfjKMzsArh8h4uiUTj9lfMHhL68W17goiI27MhE%0A%2BtCgIep4QpGn0xwaLMpfJXUgKMD3LoDvr%2BjVPAaihNlNZZrIEACdXxkCgYEA1a3w%0AdVHwDkX5ULmoe3BQNGMYq3nW5cOzj8o7IW7eHp1Wj%2BhIX8zTjeM7w4%2B3IlraF5Yq%0AYBAU9zN6eDhb80tQcOA3CW9QhpucU%2F8M%2F%2BAcWikjO2WmAL4%2BoXudpIZywrkoP5gm%0ATmVmm1pAdzZQzt6JC98eexyG2tzbSeQVSUwUdBUCgYEAn9bJaw1MuHf2YzfSH7iO%0A6lxV73LCvVjIf3M1ha9l7hoWA4HAwOArHdk0YDQc4DRoQmdVYEKRzevsfRfuu7go%0AFSHPgIALH9U0rqpens4U2zO6%2BaMTje8SNL%2BgmWQVoMag38P4p2pOK4nAOZsUOKD9%0AcXIbt9XNztO%2F0zL8s4pwTiECgYBiIfn2x2SqCXTQoBNCbXhINuKTjeclqlUCymuR%0AlSspgDTK7nCenqDh3uQJki0W%2BAaFIo%2FfgkM%2BrEOohqmj%2BIi%2B7hoYmSiwSTLoWJAN%0Ay0z6Ez58%2FtfYK4bz%2FqONdbmquAJGAohFJMiUvEQy%2B5ztBMKH0zZab%2BV3Cksjij5E%0AZU5XaQKBgHFLPsYyz5jVdmLpkMC5EuHgv0ujN2LJFUyhU3X2%2BRJpjTNgY8hOJ2ew%0AT9mumdHfi%2FNE2lnqV%2FPhXqfx5hsrEH31BrGZCfGtie%2Bc6YFTtHw2Sl1%2BP4xVcojm%0AMmojiG8fYcKCvV4nJqyHx%2BDlj34gRiunx%2BoH5RJLaC3aNQbGIXgo%0A-----END+RSA+PRIVATE+KEY-----%0A&sslmode=verify-full&sslrootcert=-----BEGIN+CERTIFICATE-----%0AMIIDJjCCAg6gAwIBAgIRAJphyT41Wuh1QUu81nFuu3QwDQYJKoZIhvcNAQELBQAw%0AKzESMBAGA1UEChMJQ29ja3JvYWNoMRUwEwYDVQQDEwxDb2Nrcm9hY2ggQ0EwHhcN%0AMjQxMTE5MDcyNDU4WhcNMzQxMTI4MDcyNDU4WjArMRIwEAYDVQQKEwlDb2Nrcm9h%0AY2gxFTATBgNVBAMTDENvY2tyb2FjaCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEP%0AADCCAQoCggEBANIXweIkNoG34JAV2DKvnN7XT0qtK6Sm5YXhNTZod1ErEIDLV42q%0AsVDBqB4zbKSbDE26v7GT0e6fzPGDXbjeWo9PfS5Myyh6vbCqnFaVfMYzBWXJ3vZc%0ACX8PeETKslyLhqwIxOiO6aIMyADtcluLrhfUesde7QE9vmZjv67npch36EpQ8VV5%0AuhCu1E71VaAFWXyM8neZsjvKdM1WYjh6ebBE3b1p3bCnRDrEN05Afo%2FobuHeT5m9%0AMIpAaxt0YVDeLS6mK1OsLg4rma7V2mVIdNHmgNQud%2Br9hXCs92wVquv2I5HNSVuM%0ATtsfyk8PiD%2FUORj1u32RPCgS%2FoQ3jM3QxhECAwEAAaNFMEMwDgYDVR0PAQH%2FBAQD%0AAgLkMBIGA1UdEwEB%2FwQIMAYBAf8CAQEwHQYDVR0OBBYEFFrrFecRLKOGCFCHzkpP%0AwC0QnEPQMA0GCSqGSIb3DQEBCwUAA4IBAQAavODFWZs3TMidmP9%2FzPSK7JntA56K%0AQqdnZwcQC0f49e%2FhCeMtaIsH8VS5Npx9xEBk9u5serMf%2FkxG63vgazA9Lf8k2gV5%0AtIfgN0OFclDbLQo9Im9qqJf0N1oOzd3u2EZkrKCuhU5AU7C%2B5njlrFWTNzzGKc8n%0AvLW%2FiTPd3tieAO4yW5JB3MdbNvLnfRFuvxr8yUM3psWM60uI4DWf7mlRpZeOgCXN%0AA4D3ftRlr7oGnytqjP3Zco1Ft6KEM5%2FIPFOPkOtRBRHo95WFZC7gwNyqbJohMeOT%0ArANGN%2B%2BaxoY59rR3AT%2BXbXprz1Yye460yniOQ1590MlV7IVogZdAWDDS%0A-----END+CERTIFICATE-----%0A'" args=[]: dial tcp 35.231.176.96:29000: connect: connection timed out
(test_runner.go:1243).runTest: test timed out (20m0s)
test artifacts and logs in: /artifacts/c2c/shutdown/src/coordinator/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-44713

msbutler commented 1 day ago

vm preemption