Adding DynamoDB High Availability Features

kmg-stripe commented 5 months ago

Context

This is a forked version of the branch for Adding DynamoDB high availability features

The motivation for this PR is to ensure we can build a snapshot using the original branch, which was failing since the branch is in a fork of Mantis. Context is here: https://github.com/Netflix/mantis/pull/675#issuecomment-2173895730

Checklist

[X] ./gradlew build compiles code correctly
[X] Added new tests where applicable
[X] ./gradlew test passes all tests
[X] Extended README or added javadocs where applicable

kmg-stripe commented 5 months ago

@Andyz26 - I pulled @mcowgill-stripe's branch into my forked repo, pushed the branch and created this PR to test building a snapshot. It looks like I need an approval to start the build.

github-actions[bot] commented 5 months ago

Test Results

533 tests +13 527 :white_check_mark: +13 7m 52s :stopwatch: -15s 139 suites + 4 6 :zzz: ± 0 139 files + 4 0 :x: ± 0

Results for commit bbb4b11e. ± Comparison against base commit d773089b.

This pull request removes 1 and adds 14 tests. Note that renamed tests count towards both.

``` io.mantisrx.server.master.store.KeyValueStoreTest ‑ testUpsertOrdered ``` ``` io.mantisrx.extensions.dynamodb.DynamoDBClientSingletonTest ‑ highAvailabilityServices io.mantisrx.extensions.dynamodb.DynamoDBLeaderElectorTest ‑ becomesCurrentLeader io.mantisrx.extensions.dynamodb.DynamoDBLeaderElectorTest ‑ respectsExistingLock io.mantisrx.extensions.dynamodb.DynamoDBLeaderElectorTest ‑ restartsOnFailures io.mantisrx.extensions.dynamodb.DynamoDBLeaderElectorTest ‑ shutdownAsLeader io.mantisrx.extensions.dynamodb.DynamoDBLeaderElectorTest ‑ shutdownNotAsLeader io.mantisrx.extensions.dynamodb.DynamoDBMasterMonitorTest ‑ getCurrentLeader io.mantisrx.extensions.dynamodb.DynamoDBMasterMonitorTest ‑ runShutdown io.mantisrx.extensions.dynamodb.DynamoDBStoreTest ‑ testInsertAndDelete io.mantisrx.extensions.dynamodb.DynamoDBStoreTest ‑ testInsertAndDeleteMoreThan25 … ```

:recycle: This comment has been updated with latest results.

Andyz26 commented 5 months ago

@Andyz26 - I pulled @mcowgill-stripe's branch into my forked repo, pushed the branch and created this PR to test building a snapshot. It looks like I need an approval to start the build.

what did you see when you click the details/approval link on the "publish snapshot to ..." task above? I thought the collab/write permission groups give you permission to approve there. BTW I've approved the build.

github-actions[bot] commented 5 months ago

Uploaded Artifacts

To use these artifacts in your Gradle project, paste the following lines in your build.gradle.

resolutionStrategy {
    force "io.mantisrx:mantis-common-serde:0.1.0-20240703.235625-542"
    force "io.mantisrx:mantis-common:0.1.0-20240703.235625-542"
    force "io.mantisrx:mantis-discovery-proto:0.1.0-20240703.235625-542"
    force "io.mantisrx:mantis-remote-observable:0.1.0-20240703.235625-543"
    force "io.mantisrx:mantis-network:0.1.0-20240703.235625-542"
    force "io.mantisrx:mantis-client:0.1.0-20240703.235625-543"
    force "io.mantisrx:mantis-runtime:0.1.0-20240703.235625-543"
    force "io.mantisrx:mantis-runtime-loader:0.1.0-20240703.235625-543"
    force "io.mantisrx:mantis-rxcontrol:0.1.0-20240703.235625-16"
    force "io.mantisrx:mantis-runtime-executor:0.1.0-20240703.235625-78"
    force "io.mantisrx:mantis-connector-job:0.1.0-20240703.235625-543"
    force "io.mantisrx:mantis-connector-iceberg:0.1.0-20240703.235625-541"
    force "io.mantisrx:mantis-testcontainers:0.1.0-20240703.235625-212"
    force "io.mantisrx:mantis-connector-kafka:0.1.0-20240703.235625-543"
    force "io.mantisrx:mantis-control-plane-client:0.1.0-20240703.235625-542"
    force "io.mantisrx:mantis-control-plane-core:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-connector-publish:0.1.0-20240703.235625-542"
    force "io.mantisrx:mantis-shaded:0.1.0-20240703.235625-541"
    force "io.mantisrx:mantis-control-plane-dynamodb:0.1.0-20240703.235625-3"
    force "io.mantisrx:mantis-control-plane-server:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-examples-groupby-sample:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-examples-jobconnector-sample:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-examples-mantis-publish-sample:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-examples-sine-function:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-examples-core:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-examples-twitter-sample:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-examples-wordcount:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-examples-synthetic-sourcejob:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-publish-netty:0.1.0-20240703.235625-535"
    force "io.mantisrx:mantis-publish-core:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-source-job-kafka:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-server-worker-client:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-source-job-publish:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-publish-netty-guice:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-server-agent:0.1.0-20240703.235625-536"
    force "io.mantisrx:mantis-control-plane-store-dynamodb:0.1.0-20240703.235625-30"
}

kmg-stripe commented 5 months ago

@Andyz26 - I pulled @mcowgill-stripe's branch into my forked repo, pushed the branch and created this PR to test building a snapshot. It looks like I need an approval to start the build.

what did you see when you click the details/approval link on the "publish snapshot to ..." task above? I thought the collab/write permission groups give you permission to approve there. BTW I've approved the build.

@Andyz26 - Ah, I do not see anything there right now, so assume I missed it. I'll self-approve next time, assuming I see the approval link.

It looks like the build finished successfully. Based on the comments in #675 , it looks like you wanted to do some testing on the snapshots before we merge it? I also see some comments in #675 . Are you ok with addressing those in follow-on PR(s)?

Andyz26 commented 5 months ago

@Andyz26 - I pulled @mcowgill-stripe's branch into my forked repo, pushed the branch and created this PR to test building a snapshot. It looks like I need an approval to start the build.

what did you see when you click the details/approval link on the "publish snapshot to ..." task above? I thought the collab/write permission groups give you permission to approve there. BTW I've approved the build.

@Andyz26 - Ah, I do not see anything there right now, so assume I missed it. I'll self-approve next time, assuming I see the approval link.

It looks like the build finished successfully. Based on the comments in #675 , it looks like you wanted to do some testing on the snapshots before we merge it? I also see some comments in #675 . Are you ok with addressing those in follow-on PR(s)?

yeah I need to try this build to see if anything is broken. And yeah for major refactor comments/discusion thread they could defer to follow up PRs.

kmg-stripe commented 5 months ago

@Andyz26 - i addressed your comments and fixed a small, unrelated bug in the dynamodb store. lmk if there is anything else. otherwise, i'll merge :)

Netflix / mantis