Failed to register worker due to API client error code 598 when running orchard dev

[llamgffe]

Following the README to run Orchard dev, the worker is failing to register due to an API client error code "598". Looking for some input on how to get up and running locally. Thanks!

Version: 0.18.0

% orchard dev                        
badger 2024/05/06 12:46:16 INFO: All 0 tables opened in 0s
badger 2024/05/06 12:46:16 INFO: Discard stats nextEmptySlot: 0
badger 2024/05/06 12:46:16 INFO: Set nextTxnTs to 0
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /metrics                  --> github.com/penglongli/gin-metrics/ginmetrics.(*Monitor).Use.func1 (4 handlers)
[GIN-debug] GET    /v1/                      --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func1 (5 handlers)
[GIN-debug] GET    /v1/openapi.yaml          --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func2 (5 handlers)
[GIN-debug] GET    /v1/controller/info       --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func3 (5 handlers)
[GIN-debug] GET    /v1/cluster-settings      --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func4 (5 handlers)
[GIN-debug] PUT    /v1/cluster-settings      --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func5 (5 handlers)
[GIN-debug] POST   /v1/service-accounts      --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func6 (5 handlers)
[GIN-debug] PUT    /v1/service-accounts/:name --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func7 (5 handlers)
[GIN-debug] GET    /v1/service-accounts/:name --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func8 (5 handlers)
[GIN-debug] GET    /v1/service-accounts      --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func9 (5 handlers)
[GIN-debug] DELETE /v1/service-accounts/:name --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func10 (5 handlers)
[GIN-debug] POST   /v1/workers               --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func11 (5 handlers)
[GIN-debug] PUT    /v1/workers/:name         --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func12 (5 handlers)
[GIN-debug] GET    /v1/workers/:name         --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func13 (5 handlers)
[GIN-debug] GET    /v1/workers               --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func14 (5 handlers)
[GIN-debug] GET    /v1/workers/:name/port-forward --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func15 (5 handlers)
[GIN-debug] DELETE /v1/workers/:name         --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func16 (5 handlers)
[GIN-debug] POST   /v1/vms                   --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func17 (5 handlers)
[GIN-debug] PUT    /v1/vms/:name             --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func18 (5 handlers)
[GIN-debug] GET    /v1/vms/:name             --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func19 (5 handlers)
[GIN-debug] GET    /v1/vms                   --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func20 (5 handlers)
[GIN-debug] GET    /v1/vms/:name/port-forward --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func21 (5 handlers)
[GIN-debug] DELETE /v1/vms/:name             --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func22 (5 handlers)
[GIN-debug] GET    /v1/vms/:name/events      --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func23 (5 handlers)
[GIN-debug] POST   /v1/vms/:name/events      --> github.com/cirruslabs/orchard/internal/controller.(*Controller).initAPI.func24 (5 handlers)
2024-05-06T12:46:46.094-0400    WARN    worker/worker.go:105    failed to register worker: API client encountered an API error to make a request: 598 
github.com/cirruslabs/orchard/internal/worker.(*Worker).runNewSession
    /tmp/cirrus-ci-build/internal/worker/worker.go:105
github.com/cirruslabs/orchard/internal/worker.(*Worker).Run
    /tmp/cirrus-ci-build/internal/worker/worker.go:73
github.com/cirruslabs/orchard/internal/command/dev.runDev.func2
    /tmp/cirrus-ci-build/internal/command/dev/dev.go:71

[edigaryev]

1. Do you run the Orchard Controller behind a proxy/load-balancer?
2. Does this happen regularly or occasionally?

[llamgffe]

1. No, it's not being run behind any proxy/load balancer.
2. It has been occurring regularly. Each time I've run orchard dev, it has been the same error while trying to register the worker.

[edigaryev]

Sorry, but I cannot seem to reproduce your issue.

I've tried running orchard dev on my host machine and in a ghcr.io/cirruslabs/macos-sonoma-base:latest VM, both work just fine.

Note that the Orchard itself does not return the HTTP 598 code (network read timeout) anywhere in its codebase. Is it possible that you have some software running on your computer that intercepts local connections?

[edigaryev]

Closing due to lack of reply.

Feel free to reopen this if you'll ever encounter this again.