Add handling to cancelled task on agent + cron spec update fix

Andyz26 commented 4 months ago

Context

Cron spec update is broken due to resetting the cron trigger ID early.
There are cases where workers on agents can "leak" due to a shutdown event failing to get processed on the agent (the agent was in an unhealthy/busy/stuck) state while its resource gateway connection can recover.

The new handling for the leaked worker's issue does the following:

When the control plane retries the failed request to shut down the worker, the worker id is marked as canceled on the agent's state in the resource cluster actor.
When the next heartbeat from the agent still shows the agent is occupied by the same worker ID, an error will be returned by the control plane to the heartbeat request, containing the body of the error with the worker ID.
The agent's heartbeat handling in the resource gateway connection will handle the new error response with status code (not accepted) and invoke cancelTask(workerID) on the task executor to force another worker shutdown.

Checklist

[ ] ./gradlew build compiles code correctly
[ ] Added new tests where applicable
[ ] ./gradlew test passes all tests
[ ] Extended README or added javadocs where applicable

github-actions[bot] commented 4 months ago

Test Results

520 tests +3 514 :white_check_mark: +3 7m 29s :stopwatch: -1s 135 suites +1 6 :zzz: ±0 135 files +1 0 :x: ±0

Results for commit 601a7b88. ± Comparison against base commit 56e0a1c7.

:recycle: This comment has been updated with latest results.

github-actions[bot] commented 4 months ago

Uploaded Artifacts

To use these artifacts in your Gradle project, paste the following lines in your build.gradle.

resolutionStrategy {
    force "io.mantisrx:mantis-common:0.1.0-20240611.234421-539"
    force "io.mantisrx:mantis-discovery-proto:0.1.0-20240611.234421-539"
    force "io.mantisrx:mantis-client:0.1.0-20240611.234421-540"
    force "io.mantisrx:mantis-network:0.1.0-20240611.234421-539"
    force "io.mantisrx:mantis-remote-observable:0.1.0-20240611.234421-540"
    force "io.mantisrx:mantis-runtime-executor:0.1.0-20240611.234421-75"
    force "io.mantisrx:mantis-common-serde:0.1.0-20240611.234421-539"
    force "io.mantisrx:mantis-runtime-loader:0.1.0-20240611.234421-540"
    force "io.mantisrx:mantis-shaded:0.1.0-20240611.234421-538"
    force "io.mantisrx:mantis-rxcontrol:0.1.0-20240611.234421-13"
    force "io.mantisrx:mantis-testcontainers:0.1.0-20240611.234421-209"
    force "io.mantisrx:mantis-runtime:0.1.0-20240611.234421-540"
    force "io.mantisrx:mantis-connector-iceberg:0.1.0-20240611.234421-538"
    force "io.mantisrx:mantis-connector-job:0.1.0-20240611.234421-540"
    force "io.mantisrx:mantis-connector-kafka:0.1.0-20240611.234421-540"
    force "io.mantisrx:mantis-control-plane-core:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-connector-publish:0.1.0-20240611.234421-539"
    force "io.mantisrx:mantis-examples-groupby-sample:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-examples-core:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-control-plane-server:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-control-plane-client:0.1.0-20240611.234421-539"
    force "io.mantisrx:mantis-examples-jobconnector-sample:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-examples-mantis-publish-sample:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-examples-sine-function:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-examples-twitter-sample:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-publish-core:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-examples-synthetic-sourcejob:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-publish-netty-guice:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-publish-netty:0.1.0-20240611.234421-532"
    force "io.mantisrx:mantis-examples-wordcount:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-source-job-kafka:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-source-job-publish:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-server-agent:0.1.0-20240611.234421-533"
    force "io.mantisrx:mantis-control-plane-store-dynamodb:0.1.0-20240611.234421-27"
    force "io.mantisrx:mantis-server-worker-client:0.1.0-20240611.234421-533"
}

Netflix / mantis