k8 upgrade failure in CI

[sehz]

https://github.com/infinyon/fluvio/runs/3583053458?check_suite_focus=true

Using Fluvio binary located @ /home/runner/work/fluvio/fluvio/fluvio
109
installing sys chart
110
starting installing
111
        Ok: ✅ Kubernetes config is loadable
112
        Ok: ✅ Supported helm version is installed
113
        Ok: ✅ Fluvio system charts are installed
114
Upgrading fluvio chart
115
checking fluvio crd attempt: 0
116
fluvio crd installed
117
found SC service addr: 172.18.0.2:32224
118
Trying to connect to sc at: 172.18.0.2:32224, attempt: 0
119
Current Version 0.9.6 is not same as expected: 0.9.7
120
Connection failed.  sleeping 10 seconds
121
Trying to connect to sc at: 172.18.0.2:32224, attempt: 1
122
Got updated SC Version0.9.7
123
Connection to sc suceed!
124
updating profile for: 172.18.0.2:32224
125
checking if spu groups exists
126
spu group: main exists, skipping
127
0 of 1 spu are ready, sleeping 10 seconds...
128
All SPUs(1) are ready
129
Successfully installed Fluvio!
130
~/work/fluvio/fluvio/tests ~/work/fluvio/fluvio
131
Fluvio CLI           : 0.9.7
132
Fluvio CLI SHA256    : 19afdb394fcf72ad69c97896b9eac819038a0328d1fe9c772b1495651c6ad7f0
133
Fluvio Platform      : 0.9.7 (k3d-fluvio)
134
Git Commit           : 3c65ddda9476dd7855e4f2d9798179e87c4e192c
135
OS Details           : Ubuntu 20.04 (kernel 5.8.0-1040-azure)
136
=== Plugin Versions ===
137
Infinyon Cloud CLI (fluvio-cloud) : 0.1.6
138
Fluvio Runner (fluvio-run)     : 0.0.0
139
✅ CLI Version verified: 0.9.7
140
✅ Platform Version verified: 0.9.7
141
Create test topic: prerelease
142
topic "prerelease" created
143
Error: 
144
   0: Fluvio client error
145
   1: Fluvio socket error: Connection refused (os error 111)
146
   2: Connection refused (os error 111)
147
Validate test data w/ v0.9.7 CLI matches expected data AFTER upgrading cluster + CLI to v0.9.7
148
-: FAILED
149
shasum: WARNING: 1 computed checksum did NOT match
150
Got: 
151
make: *** [Makefile:155: upgrade-test] Error 1
152
Expected: 1f591b852352a1c068502393bf5f7490efbe6e7b
153
Error: Process completed with exit code 2.

failure logs:

NAME        NAMESPACE   REVISION    UPDATED                                 STATUS      CHART               APP VERSION
17
fluvio      default     2           2021-09-13 04:24:05.316032416 +0000 UTC deployed    fluvio-app-0.9.0    0.9.7      
18
fluvio-sys  default     2           2021-09-13 04:24:04.070198289 +0000 UTC deployed    fluvio-sys-0.9.1    0.9.7      
19
NAME              READY   AGE
20
fluvio-spg-main   1/1     39s
21
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
22
data-fluvio-spg-main-0   Bound    pvc-c4497a1a-19b0-49ae-9ed0-8ecee2ca4eab   1Gi        RWO            local-path     40s
23
NAME                         READY   STATUS        RESTARTS   AGE
24
fluvio-sc-85dd5d94c8-5q8zv   1/1     Running       0          27s
25
fluvio-sc-bbbf8d644-shvj7    1/1     Terminating   0          51s
26
fluvio-spg-main-0            1/1     Running       0          7s
27
NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
28
kubernetes           ClusterIP   10.43.0.1       <none>        443/TCP             90s
29
fluvio-sc-internal   ClusterIP   10.43.175.170   <none>        9004/TCP            51s
30
fluvio-sc-public     NodePort    10.43.87.34     <none>        9003:32224/TCP      51s
31
fluvio-spg-main      ClusterIP   None            <none>        9005/TCP,9006/TCP   40s
32
fluvio-spu-main-0    NodePort    10.43.157.111   <none>        9005:32138/TCP      40s
33
NAME     ID    STATUS   TYPE      PUBLIC HOST                                                               PRIVATE HOST                        PORT
34
main-0   0     Online   Managed   {"encryption":"PLAINTEXT","ingress":[{"ip":"172.18.0.2"}],"port":32138}   fluvio-spg-main-0.fluvio-spg-main   9006
35
NAME   AGE
36
main   40s
37
Error from server (NotFound): pods "fluvio-spg-main-1" not found

[sehz]

From failure log, SPU pod has not been upgraded

[github-actions[bot]]

Stale issue message

[tjtelan]

This issue hasn't been seen in CI for a while. Will create new issue if this changes.