search

kingdonb / stats-tracker-ghcr

Tracking GHCR download counts for FluxCD
0 stars 0 forks source link

It should publish the results #11

Closed kingdonb closed 1 year ago

kingdonb commented 1 year ago
kingdonb commented 1 year ago

Progress on #12 today, but it comes with a regression

leafer.1  | {"message":"add finalizer to source-watcher (dae7ee3c-05a1-4cb4-b3ab-e523cd60cfcf)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T01:32:10+00:00"}
leafer.1  | /usr/local/bundle/ruby/3.1.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler/selector.rb:190:in `block in select': undefined method `alive?' for 12312:Integer (NoMethodError)
leafer.1  |
leafer.1  |           fiber.transfer if fiber.alive?
leafer.1  |                                  ^^^^^^^
leafer.1  |     from /usr/local/bundle/ruby/3.1.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler/selector.rb:188:in `times'
leafer.1  |     from /usr/local/bundle/ruby/3.1.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler/selector.rb:188:in `select'
leafer.1  |     from /usr/local/bundle/ruby/3.1.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:122:in `run_once'
leafer.1  |     from /usr/local/bundle/ruby/3.1.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:116:in `run'
leafer.1  |     from /usr/local/bundle/ruby/3.1.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:138:in `close'
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
leafer.1  | exited with code 1
system    | sending SIGTERM to all processes
sample.1  | terminated by SIGTERM
measure.1 | terminated by SIGTERM

This is a different error than the SIGIOT from foreman, or from maybe pg or wasmer, but this process fails at least 30% of the time and it does not recover on its own.

This could be much better in so many ways, (but we need it to have a web front-end before we can address any of those!)

kingdonb commented 1 year ago

Here they are, the primary error (SIGIOT):

and

leafer.1  | /usr/local/bundle/ruby/3.1.0/gems/activerecord-7.0.5/lib/active_record/core.rb:310: [BUG] Segmentation fault at 0x0000000000000029
leafer.1  | ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [aarch64-linux]

The full output:

(⎈|chicago:default):~ (macos-homebrew *|u=)$ k get po -w
NAME                                READY   STATUS              RESTARTS   AGE
stats-tracker-ghcr-28113228-ff2nh   0/1     Completed           0          30m
stats-tracker-ghcr-28113232-pkq56   0/1     Completed           1          26m
stats-tracker-ghcr-28113236-8zq2d   0/1     Completed           1          22m
stats-tracker-ghcr-28113256-4rjv4   0/1     ContainerCreating   0          2s
stats-tracker-ghcr-28113256-4rjv4   1/1     Running             0          2s
^C
(⎈|chicago:default):~ (macos-homebrew *|u=)$ k get leaf
NAME                          AGE   READY   STATUS
charts-flagger                11s   False   Reconciling
flagger                       11s   False   Reconciling
flagger-loadtester            11s   False   Reconciling
flagger-manifests             11s   False   Reconciling
flux-cli                      11s   False   Reconciling
flux-manifests                11s   False   Reconciling
golang-with-libgit2           11s   False   Reconciling
golang-with-libgit2-all       11s   False   Reconciling
golang-with-libgit2-only      11s   False   Reconciling
helm-controller               11s   False   Reconciling
image-automation-controller   11s   False   Reconciling
image-reflector-controller    11s   False   Reconciling
kustomize-controller          11s   False   Reconciling
notification-controller       11s   False   Reconciling
source-controller             11s   False   Reconciling
source-watcher                11s   False   Reconciling

(⎈|chicago:default):~ (macos-homebrew *|u=)$ k get po -w
NAME                                READY   STATUS      RESTARTS   AGE
stats-tracker-ghcr-28113232-pkq56   0/1     Completed   1          26m
stats-tracker-ghcr-28113236-8zq2d   0/1     Completed   1          22m
stats-tracker-ghcr-28113256-4rjv4   0/1     Completed   0          21s
^C
(⎈|chicago:default):~ (macos-homebrew *|u=)$ k get leaf
NAME                          AGE   READY   STATUS
charts-flagger                21s   False   Reconciling
flagger                       21s   False   Reconciling
flagger-loadtester            21s   False   Reconciling
flagger-manifests             21s   False   Reconciling
flux-cli                      21s   False   Reconciling
flux-manifests                21s   False   Reconciling
golang-with-libgit2           21s   False   Reconciling
golang-with-libgit2-all       21s   False   Reconciling
golang-with-libgit2-only      21s   False   Reconciling
helm-controller               21s   False   Reconciling
image-automation-controller   21s   False   Reconciling
image-reflector-controller    21s   False   Reconciling
kustomize-controller          21s   False   Reconciling
notification-controller       21s   False   Reconciling
source-controller             21s   False   Reconciling
source-watcher                21s   False   Reconciling

(⎈|chicago:default):~ (macos-homebrew *|u=)$ ^C

(⎈|chicago:default):~ (macos-homebrew *|u=)$ k logs -f stats-tracker-ghcr-28113256-4rjv4
sample.1  | started with pid 8
leafer.1  | started with pid 9
measure.1 | started with pid 10
sample.1  | {"message":"use incluster config","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:32+00:00"}
sample.1  | {"message":"start the operator","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:32+00:00"}
sample.1  | {"message":"trigger add action for fluxcd (78d6e145-1767-4191-9435-69d53ef3d5c2)","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:32+00:00"}
sample.1  | {"message":"create new project {projectName: fluxcd}","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:32+00:00"}
leafer.1  | {"message":"use incluster config","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:32+00:00"}
leafer.1  | {"message":"start the operator","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:32+00:00"}
leafer.1  | {"message":"trigger add action for source-controller (4953e519-96a0-4416-a740-d575a55d9feb)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: source-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to source-controller (4953e519-96a0-4416-a740-d575a55d9feb)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
leafer.1  | {"message":"trigger add action for kustomize-controller (5f233779-0608-425a-b76b-d4b05abe020b)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: kustomize-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to kustomize-controller (5f233779-0608-425a-b76b-d4b05abe020b)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for notification-controller (12d2591c-765c-4efb-b5aa-430ed1c927c1)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: notification-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to notification-controller (12d2591c-765c-4efb-b5aa-430ed1c927c1)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for helm-controller (7c669934-47fa-4560-844c-4623fbab774e)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: helm-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to helm-controller (7c669934-47fa-4560-844c-4623fbab774e)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for image-automation-controller (d9c2c886-af4f-4700-a21a-23ceeba50184)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: image-automation-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to image-automation-controller (d9c2c886-af4f-4700-a21a-23ceeba50184)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for image-reflector-controller (ea09f6dd-e97c-490d-82e2-77a36a09c61c)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: image-reflector-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to image-reflector-controller (ea09f6dd-e97c-490d-82e2-77a36a09c61c)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for flagger (c1da0be9-fbce-429a-afb1-a36dc0ea7429)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flagger}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to flagger (c1da0be9-fbce-429a-afb1-a36dc0ea7429)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for flux-cli (5f52b0f5-d754-406b-919f-32f96dc28a9c)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flux-cli}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to flux-cli (5f52b0f5-d754-406b-919f-32f96dc28a9c)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for flagger-loadtester (ba20a192-e365-42db-b8ad-089414129a88)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flagger-loadtester}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to flagger-loadtester (ba20a192-e365-42db-b8ad-089414129a88)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for flux-manifests (5c7bfc93-ffdf-4366-b7c8-4b724ac3f083)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flux-manifests}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to flux-manifests (5c7bfc93-ffdf-4366-b7c8-4b724ac3f083)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for charts-flagger (6c664c6b-f444-45d3-b6a5-cbaaeb70ddb8)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: charts%2Fflagger}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to charts-flagger (6c664c6b-f444-45d3-b6a5-cbaaeb70ddb8)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for golang-with-libgit2 (00395e6c-acc2-475e-b6cd-c1487b8e63b4)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: golang-with-libgit2}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"add finalizer to golang-with-libgit2 (00395e6c-acc2-475e-b6cd-c1487b8e63b4)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"trigger add action for flagger-manifests (41fa12b9-ab93-4e29-957a-1c291a1509d3)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flagger-manifests}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:33+00:00"}
sample.1  | {"message":"add event registered health check for leaves from project/fluxcd(Normal) to fluxcd(78d6e145-1767-4191-9435-69d53ef3d5c2)","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:34+00:00"}
sample.1  | {"message":"add finalizer to fluxcd (78d6e145-1767-4191-9435-69d53ef3d5c2)","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:34+00:00"}
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
leafer.1  | {"message":"add finalizer to flagger-manifests (41fa12b9-ab93-4e29-957a-1c291a1509d3)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"trigger add action for golang-with-libgit2-only (128b2c61-a28d-43d0-8d54-c9d12c4f3d36)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"upsert called for {packageName: golang-with-libgit2-only}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"add finalizer to golang-with-libgit2-only (128b2c61-a28d-43d0-8d54-c9d12c4f3d36)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"trigger add action for golang-with-libgit2-all (8c9d06c2-537c-4c25-a842-1ab9ddf3efdf)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"upsert called for {packageName: golang-with-libgit2-all}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"add finalizer to golang-with-libgit2-all (8c9d06c2-537c-4c25-a842-1ab9ddf3efdf)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"trigger add action for source-watcher (6553472d-4495-47f7-af7b-1c778324ac9d)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"upsert called for {packageName: source-watcher}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | {"message":"add finalizer to source-watcher (6553472d-4495-47f7-af7b-1c778324ac9d)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:18:38+00:00"}
leafer.1  | /usr/local/bundle/ruby/3.1.0/gems/activerecord-7.0.5/lib/active_record/core.rb:310: [BUG] Segmentation fault at 0x0000000000000029
leafer.1  | ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [aarch64-linux]
leafer.1  |
leafer.1  | -- Control frame information -----------------------------------------------
leafer.1  | c:0015 p:0151 s:0089 e:000086 BLOCK  /usr/local/bundle/ruby/3.1.0/gems/activerecord-7.0.5/lib/active_record/core.rb:310 [FINISH]
leafer.1  | SEGV received in SEGV handler
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
leafer.1  | terminated by SIGIOT
system    | sending SIGTERM to all processes
sample.1  | terminated by SIGTERM
measure.1 | terminated by SIGTERM
kingdonb commented 1 year ago

Sometimes it does this:

$ k logs -f stats-tracker-ghcr-28113260-hvknn
sample.1  | started with pid 7
leafer.1  | started with pid 8
measure.1 | started with pid 9
sample.1  | {"message":"use incluster config","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
sample.1  | {"message":"start the operator","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"use incluster config","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"start the operator","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
sample.1  | {"message":"trigger add action for fluxcd (78d6e145-1767-4191-9435-69d53ef3d5c2)","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
sample.1  | {"message":"create new project {projectName: fluxcd}","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for kustomize-controller (5f233779-0608-425a-b76b-d4b05abe020b)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: kustomize-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to kustomize-controller (5f233779-0608-425a-b76b-d4b05abe020b)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for helm-controller (7c669934-47fa-4560-844c-4623fbab774e)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: helm-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to helm-controller (7c669934-47fa-4560-844c-4623fbab774e)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for image-automation-controller (d9c2c886-af4f-4700-a21a-23ceeba50184)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: image-automation-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to image-automation-controller (d9c2c886-af4f-4700-a21a-23ceeba50184)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for golang-with-libgit2-all (8c9d06c2-537c-4c25-a842-1ab9ddf3efdf)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: golang-with-libgit2-all}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to golang-with-libgit2-all (8c9d06c2-537c-4c25-a842-1ab9ddf3efdf)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for notification-controller (12d2591c-765c-4efb-b5aa-430ed1c927c1)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: notification-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to notification-controller (12d2591c-765c-4efb-b5aa-430ed1c927c1)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for image-reflector-controller (ea09f6dd-e97c-490d-82e2-77a36a09c61c)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: image-reflector-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to image-reflector-controller (ea09f6dd-e97c-490d-82e2-77a36a09c61c)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for flagger-loadtester (ba20a192-e365-42db-b8ad-089414129a88)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flagger-loadtester}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to flagger-loadtester (ba20a192-e365-42db-b8ad-089414129a88)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for golang-with-libgit2 (00395e6c-acc2-475e-b6cd-c1487b8e63b4)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: golang-with-libgit2}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to golang-with-libgit2 (00395e6c-acc2-475e-b6cd-c1487b8e63b4)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for flagger-manifests (41fa12b9-ab93-4e29-957a-1c291a1509d3)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flagger-manifests}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to flagger-manifests (41fa12b9-ab93-4e29-957a-1c291a1509d3)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for golang-with-libgit2-only (128b2c61-a28d-43d0-8d54-c9d12c4f3d36)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: golang-with-libgit2-only}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to golang-with-libgit2-only (128b2c61-a28d-43d0-8d54-c9d12c4f3d36)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for source-controller (4953e519-96a0-4416-a740-d575a55d9feb)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: source-controller}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to source-controller (4953e519-96a0-4416-a740-d575a55d9feb)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for flagger (c1da0be9-fbce-429a-afb1-a36dc0ea7429)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flagger}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to flagger (c1da0be9-fbce-429a-afb1-a36dc0ea7429)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for flux-cli (5f52b0f5-d754-406b-919f-32f96dc28a9c)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flux-cli}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to flux-cli (5f52b0f5-d754-406b-919f-32f96dc28a9c)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for flux-manifests (5c7bfc93-ffdf-4366-b7c8-4b724ac3f083)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: flux-manifests}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to flux-manifests (5c7bfc93-ffdf-4366-b7c8-4b724ac3f083)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for charts-flagger (6c664c6b-f444-45d3-b6a5-cbaaeb70ddb8)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: charts%2Fflagger}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to charts-flagger (6c664c6b-f444-45d3-b6a5-cbaaeb70ddb8)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"trigger add action for source-watcher (6553472d-4495-47f7-af7b-1c778324ac9d)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"upsert called for {packageName: source-watcher}","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
leafer.1  | {"message":"add finalizer to source-watcher (6553472d-4495-47f7-af7b-1c778324ac9d)","log_level":"INFO","log_type":"Log4RTest","log_app":"leaves.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:04+00:00"}
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
sample.1  | {"message":"add event registered health check for leaves from project/fluxcd(Normal) to fluxcd(78d6e145-1767-4191-9435-69d53ef3d5c2)","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:05+00:00"}
sample.1  | {"message":"add finalizer to fluxcd (78d6e145-1767-4191-9435-69d53ef3d5c2)","log_level":"INFO","log_type":"Log4RTest","log_app":"projects.example.com/v1alpha1","log_timestamp":"2023-06-15T02:20:05+00:00"}
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### fresh packages count: 0 (expecting 16) #######
measure.1 | ########### final packages count: 0 (expecting 16) #######
measure.1 | ########### c (0) != package_count (16) #######
measure.1 | ########### g (16) leaves left; still collecting #######
measure.1 | ########### g (16) leaves left; still collecting #######
kingdonb commented 1 year ago

All of those things seem to happen less often than they appeared. I saw it run 20-30x today without interrupting, but it does still occasionally fail like this.

I think these were concurrency issues, even though they prevented things from working at all yesterday, it was a fluke.

kingdonb commented 1 year ago

So there will be a small compromise, in the spirit of textualism

In the requirements spec'd out in the top-level comment, we said we wanted the graph on a web page, served by Rails. Turns out that's not so difficult but it requires a Grafana instance, and if you wanted Grafana Cloud instead of to host your own Grafana, then scraping graphs from the free instance to your web page becomes a problem again.

So we've got a Rails server with access to the cluster, and a cronjob that runs our reconciler, and they are separate. And there is a postgres database that the reconciler outputs to and the Rails server can access, but it doesn't know how to do any graphs. We defer to Grafana for that. And currently there's a gap:

kingdonb commented 1 year ago

Checked off All of this, but on a web page (we'll serve it up at rails root!) as an exception as explained above

kingdonb commented 1 year ago

We have a Rails website, we have a Grafana Cloud, they will hopefully become friends soon

This issue can close in spite of #21 above, when we are writing data points into production as spec'd by In the production database at neon dot tech and we're checking off All of this, but on a web page (we'll serve it up at rails root!) in spite of not quite getting all the way there.

That, and closing of #20 will signify production readiness 👍

kingdonb commented 1 year ago

It isn't quite publishing data points into Neon.tech database yet

I think we need a separate deployment in dev and in prod, since dev is now a substantial dataset and we should probably maintain it at least until prod is as long as it is now (2 weeks +)

Will keep this open until there is a Grafana Cloud or OSS instance pointed at our Neon.tech prod database and it has data points to report. #21 is still separate.

kingdonb commented 1 year ago

I was able to publish some data points on Neon.tech today, with the RAILS_ENV=production setting in a Kubernetes cluster, and I have published a second release now: 0.0.2.

This is still a bit away from being able to say "we have Neon.tech in production" however, we can take the next step which I believe is:

If that works reliably, then we'll call that the production recorder, and then with a GitHub action that runs 6 or 8 times a day, we'll have enough data points to show at least daily trends. We can slap that visualization into a serverless process or KEDA-enabled deployment, and we can even explore using the KEDA cron scheduler to replace GitHub as new-prod later on.

(The actual viz will be done in Grafana, we were aiming to host our own Grafana OSS alongside of the Rails app deployment, as expressed in #21, but we don't need to make this our only deployment. We should have useful output for the DX reporting for example, where it likely doesn't need to be presented in any format that shows more granular information than one single number reported at most daily.)

kingdonb commented 1 year ago

We have 4 hours of continuous data in a database labeled "prod" on neon.tech, and all the boxes in the top-post are checked so I think this can be closed now ✅