search

istio / istio

Connect, secure, control, and observe services.
https://istio.io
Apache License 2.0
35.88k stars 7.74k forks source link

Slow unit tests #37555

Open howardjohn opened 2 years ago

howardjohn commented 2 years ago

The following tests take >1s:

These should be optimized.

It is recommended to run https://github.com/istio/istio/wiki/Test-Flakes#reproducing-test-flakes locally before fixing anything to ensure that changes (such as lowering timeouts, etc) do not make things flaky.

cebernardi commented 1 year ago

Hi @howardjohn, is this issue still open to help from the community? I would be interested to help here

ramaraochavali commented 1 year ago

@cebernardi Sure. Appreciate your help.

cebernardi commented 1 year ago

Awesome! Taking a stab..

howardjohn commented 1 year ago

Great! the list may be out of date though, just FYI

cebernardi commented 1 year ago

I opened a draft PR to check if the direction could be good (for the first 2 enumerated tests) and get some early feedback.

whitneygriffith commented 1 year ago

Hey @howardjohn, I am looking to pick up some of these tests. Is the original list up to date?

howardjohn commented 1 year ago

Hasn't been updated since the original issue.

I believe I found it by

test-times () {
        rg --hidden '\-\-\- PASS' | map 'i.split("PASS: ")[1].replace("(", "").replace(")", "")' | sort -k2,2 -n
}
cat build-log.txt | test-times

You can get the build log by goin to the commit, clicking the green check or red x and going to the prow page: https://prow.istio.io/view/gs/istio-prow/logs/unit-tests_istio_postsubmit/1637825841897410560 for example

simplyatul commented 1 year ago 
test-times () {
        rg --hidden '\-\-\- PASS' | map 'i.split("PASS: ")[1].replace("(", "").replace(")", "")' | sort -k2,2 -n
}

@howardjohn I could not locate map command on my ubuntu (v22.04) box. Which package it belongs to?

simplyatul commented 1 year ago

I see time taken by TestSignCSR varies a lot. On my local system, it is taking around one to one and half sec. Some times it takes less than a sec as well. On Prow CI/CD pipeline, I see two instances with 8s and 3s. Wonder why such deviation? Appreciate if someone can help me to understand.

go test ./security/pkg/pki/ca -count=1 -v -run ^TestSignCSR$
=== RUN   TestSignCSR
--- PASS: TestSignCSR (1.44s)
PASS
ok      istio.io/istio/security/pkg/pki/ca  1.452s

go test ./security/pkg/pki/ca -count=1 -v -run ^TestSignCSR$
=== RUN   TestSignCSR
--- PASS: TestSignCSR (1.12s)
PASS
ok      istio.io/istio/security/pkg/pki/ca  1.130s

go test ./security/pkg/pki/ca -count=1 -v -run ^TestSignCSR$
=== RUN   TestSignCSR
--- PASS: TestSignCSR (0.78s)
PASS
ok      istio.io/istio/security/pkg/pki/ca  0.795s

Btw below links might have expire

https://prow.istio.io/log?job=unit-tests_istio_postsubmit&id=1669581982494887936

=== RUN   TestSignCSR
--- PASS: TestSignCSR (8.21s)

https://prow.istio.io/log?job=unit-tests_istio_postsubmit&id=1669650939406979072

=== RUN   TestSignCSR
--- PASS: TestSignCSR (3.60s)
howardjohn commented 1 year ago

Ci runs with -race, I think it makes the signing take exceptionally long

howardjohn commented 1 year ago

on the map command - sorry it's an alias, I can post the content when I'm at my computer

cebernardi commented 1 year ago

I use this to get a list of the tests with execution time:

cat build-log.txt | grep "\-\-\- PASS" | grep -v "    \-\-\- PASS:" | sort -V -k2 -t'('

I hope this helps

howardjohn commented 1 year ago

I wrote a lot about Go build times which is tightly coupled to test times: https://blog.howardjohn.info/posts/go-build-times/

jessicegao commented 8 months ago

cc

howardjohn commented 2 months ago

New slow list: