seal-io / walrus

Walrus is an open-source application management platform based on IaC tools including OpenTofu, Terraform and others. It helps platform engineers build golden paths for developers and empowers developers with self-service capabilities.
Apache License 2.0
428 stars 42 forks source link

Incorrect catalog status when synchronizing templates timeout #1048

Closed Finenyaco closed 1 year ago

Finenyaco commented 1 year ago

Describe the bug

The catalog status is incorrect when synchronizing templates timeout.

To Reproduce Steps to reproduce the behavior:

  1. Add a catalog.

  2. Wait to synchronize template.

    image
  3. The timeout is too short, and it remains in Initializing state after timeout.

    image
2023-08-11T06:34:45.564Z    E   catalog failed to update sync info: context deadline exceeded
github.com/seal-io/seal/pkg/catalog.Sync.func1
    github.com/seal-io/seal/pkg/catalog/catalog.go:110
github.com/alitto/pond.(*WorkerPool).executeTask
    github.com/alitto/pond@v1.8.3/pond.go:454
github.com/alitto/pond.worker
    github.com/alitto/pond@v1.8.3/worker.go:32
2023-08-11T06:34:45.574Z    E   catalog failed to update catalog cloudposse status: context deadline exceeded
github.com/seal-io/seal/pkg/catalog.Sync.func1
    github.com/seal-io/seal/pkg/catalog/catalog.go:119
github.com/alitto/pond.(*WorkerPool).executeTask
    github.com/alitto/pond@v1.8.3/pond.go:454
github.com/alitto/pond.worker
    github.com/alitto/pond@v1.8.3/worker.go:32

Expected behavior

Environment

Additional context Add any other context about the problem here.

Finenyaco commented 1 year ago

Version: main UI: dev-b37a35c Commit ID: 84930d41256fd033b924aed3c9a2ca36bd180fdb Env: AWS EC2 Ubuntu 22.04 amd64 Test Result: Failed

Synchronization continues for a period of time and then is interrupted without error logs. The catalog remained in Initializing state.

2023-08-15T02:43:30.002Z    D   task    executed task: service-relationship-check
2023-08-15T02:43:33.498Z    I   catalog found 484 repositories in https://github.com/aliyun
2023-08-15T02:43:33.498Z    D   catalog sync template alicloud-android-demo
2023-08-15T02:43:33.498Z    D   catalog sync template aliyun-openapi-php-sdk
2023-08-15T02:43:33.498Z    D   catalog sync template aliyun-spark-deploy-tool
2023-08-15T02:43:33.498Z    D   catalog sync template aliyun-oss-java-sdk
2023-08-15T02:43:33.498Z    D   catalog sync template aliyun-odps-java-sdk
2023-08-15T02:43:33.498Z    D   catalog sync template aliyun-openapi-python-sdk
2023-08-15T02:43:33.498Z    D   catalog sync template aliyun-openapi-java-sdk
2023-08-15T02:43:33.498Z    D   catalog sync template aliyun-cli
2023-08-15T02:43:33.499Z    D   catalog sync template aliyun-emapreduce-datasources
2023-08-15T02:43:33.499Z    D   catalog sync template aliyun-ocs-java-sdk
...
2023-08-15T03:16:47.368Z    W   template    no versions found for aliyun-pai-featurestore-go-sdk
2023-08-15T04:13:41.796Z    I   catalog found 484 repositories in https://github.com/aliyun
2023-08-15T04:13:41.797Z    D   catalog sync template alicloud-android-demo
2023-08-15T04:13:41.797Z    D   catalog sync template aliyun-openapi-php-sdk
2023-08-15T04:13:41.802Z    D   catalog sync template aliyun-openapi-python-sdk
2023-08-15T04:13:41.802Z    D   catalog sync template aliyun-cli
2023-08-15T04:13:41.802Z    D   catalog sync template aliyun-spark-deploy-tool
2023-08-15T04:13:41.802Z    D   catalog sync template aliyun-emapreduce-datasources
2023-08-15T04:13:41.802Z    D   catalog sync template aliyun-oss-java-sdk
2023-08-15T04:13:41.802Z    D   catalog sync template aliyun-openapi-java-sdk
2023-08-15T04:13:41.802Z    D   catalog sync template aliyun-odps-java-sdk
2023-08-15T04:13:41.802Z    D   catalog sync template aliyun-ocs-java-sdk
2023-08-15T04:13:43.777Z    W   template    no versions found for aliyun-spark-deploy-tool
2023-08-15T04:13:43.780Z    D   catalog sync template aliyun-oss-ios-sdk
2023-08-15T04:13:43.980Z    W   template    no versions found for aliyun-ocs-java-sdk
2023-08-15T04:13:43.988Z    D   catalog sync template aliyun-oss-android-sdk
2023-08-15T04:13:45.253Z    W   template    no versions found for aliyun-openapi-php-sdk
2023-08-15T04:13:45.551Z    D   catalog sync template aliyun-openapi-net-sdk
2023-08-15T04:13:49.183Z    W   template    no versions found for alicloud-android-demo
2023-08-15T04:13:49.272Z    D   catalog sync template aliyun-oss-python-sdk
...
2023-08-15T05:11:38.562Z    W   template    no versions found for alibabacloud-bailian-sdk
alexcodelf commented 1 year ago

Currently not reproducible, need to keep watching

Finenyaco commented 1 year ago

Environment

Trigger catalog synchronization via CatalogTemplateSync task. Sometimes it may time out and the catalog stuck in Initializing state.

image
2023-09-07T07:14:00.043Z    E   catalog failed to sync catalog terraform-alicloud-modules templates: Get "https://api.github.com/orgs/terraform-alicloud-modules/repos?per_page=100&visibility=public": context deadline exceeded
github.com/seal-io/walrus/pkg/catalog.UpdateStatusWithSyncErr
    github.com/seal-io/walrus/pkg/catalog/catalog.go:197
github.com/seal-io/walrus/pkg/scheduler/catalog.(*TemplateSyncTask).Process
    github.com/seal-io/walrus/pkg/scheduler/catalog/template_sync_task.go:79
github.com/seal-io/walrus/utils/cron.timeoutTask.Process
    github.com/seal-io/walrus/utils@v0.0.0-00010101000000-000000000000/cron/cron.go:156
reflect.Value.call
    reflect/value.go:584
reflect.Value.Call
    reflect/value.go:368
github.com/go-co-op/gocron.callJobFuncWithParams
    github.com/go-co-op/gocron@v1.29.0/gocron.go:116
github.com/go-co-op/gocron.runJob
    github.com/go-co-op/gocron@v1.29.0/executor.go:76
github.com/go-co-op/gocron.(*executor).runJob
    github.com/go-co-op/gocron@v1.29.0/executor.go:173
github.com/go-co-op/gocron.(*executor).run.func1
    github.com/go-co-op/gocron@v1.29.0/executor.go:239
2023-09-07T07:14:00.046Z    W   task    error executing: error syncing templates of catalog 477357966266204162: failed to update sync info: context deadline exceeded   {"createBy": "CatalogTemplateSyncCronExpr"}
Finenyaco commented 1 year ago
Finenyaco commented 1 year ago

Environment

Test Result: Cannot reproduce in latest version.