search

awsdocs / aws-doc-sdk-examples

Welcome to the AWS Code Examples Repository. This repo contains code examples used in the AWS documentation, AWS SDK Developer Guides, and more. For more information, see the Readme.md file below.
Apache License 2.0
9.58k stars 5.63k forks source link

EPIC: MVP for Amazon Keyspaces for Apache Cassandra #3111

Closed brmur closed 10 months ago

brmur commented 2 years ago

Amazon Keyspaces MVP

User guides:

Overview

Amazon Keyspaces is a hosted and managed implementation of Apache Cassandra, which is a noSql database somewhat like DynamoDB. The API focuses on creating keyspaces (which are like clusters in Aurora) and tables. There’s no AWS-provided data client, so connections and queries are handled through 3rd-party libraries. We will use the standard movies.json like we did for DynamoDB to demonstrate table actions.

Reference implementation of scenario and actions in Python [TBD].

Hello Keyspaces

Call ListKeyspaces to display all current Keyspace names and ARNs. By default, an account has a few system keyspaces, so there should be a few to display even if none have yet been created.

Your Hello output should look something like this:

Hello, Amazon Keyspaces! Let's list your keyspaces:

system_schema
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/system_schema/
system_schema_mcs
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/system_schema_mcs/
system
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/system/
system_multiregion_info
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/system_multiregion_info/

Scenario

  1. Create a keyspace (CreateKeyspace)
  2. Check for keyspace existence by looping and calling GetKeyspace until it exists. (It raises ResourceNotFoundException until it’s ready).
  3. List up to 10 keyspaces, optionally using a paginator. (ListKeyspaces)
  4. Create a table with a simple movie data schema, and enable point-in-time recovery. (CreateTable) Python reference code: 
    response = self.keyspaces_client.create_table(
keyspaceName=self.ks_name, tableName=table_name,
schemaDefinition={
    'allColumns': [
        {'name': 'title', 'type': 'text'},
        {'name': 'year', 'type': 'int'},
        {'name': 'release_date', 'type': 'timestamp'},
        {'name': 'plot', 'type': 'text'},
    ],
    'partitionKeys': [{'name': 'year'}, {'name': 'title'}]
},
pointInTimeRecovery={'status': 'ENABLED'})
  5. Check for table readiness by looping and calling GetTable until status == ‘ACTIVE’. You might have to pause for 1s before calling GetTable or it might raise ResourceNotFoundException (or you can handle this in your loop).
  6. Display the table schema that’s returned in the GetTable response.
  7. List all tables in the keyspace. (ListTables)
  8. Languages without a SigV4 Cassandra plugin (everyone other than Go, Java, JavaScript, and Python): print a message to the effect of "Now you can use your favorite Cassandra app to connect to the database and move data in and out of your table." Pause the demo to give the user the opportunity to do this, then continue. If you don't have a SigV4 library, do not write a CRUD example using plain text (username/password) authentication, the Keyspaces team explicitly recommends against this in a production environment.
  9. For Go, Java, JavaScript, and Python ONLY: you have a SigV4 Cassandra plugin available, so use this together with a 3rd party Cassandra driver to CRUD data with the table. This requires a certificate for TLS and the SigV4 plugin for your language to connect with AWS credentials (not username/password). The docs have examples for how to do this here (https://docs.aws.amazon.com/keyspaces/latest/devguide/programmatic.drivers.html).
    1. Download and save the certificate from: https://certs.secureserver.net/repository/sf-class2-root.crt (in code is best, otherwise have the user do it manually).
    2. Connect using your language’s SigV4 library.
    3. I found that you had to specify ConsistencyLevel.LOCAL_QUORUM and a load balancing policy (I picked DCAwareRoundRobinPolicy as a reasonable default) and a protocol_version of 4 (to avoid negotiating down).
    4. Python reference code: 
      ssl_context = SSLContext(PROTOCOL_TLSv1_2)
    ssl_context.load_verify_locations(self.cert_file_path)
    ssl_context.verify_mode = CERT_REQUIRED
    auth_provider = SigV4AuthProvider(self.boto_session)
    contact_point = f"cassandra.{self.boto_session.region_name}.amazonaws.com"
    exec_profile = ExecutionProfile(
        consistency_level=ConsistencyLevel.LOCAL_QUORUM,
        load_balancing_policy=DCAwareRoundRobinPolicy())
    self.cluster = Cluster(
        [contact_point], ssl_context=ssl_context, auth_provider=auth_provider,
        port=9142, execution_profiles={EXEC_PROFILE_DEFAULT: exec_profile},
        protocol_version=4)
    self.cluster.__enter__()
    self.session = self.cluster.connect(self.ks_name)
    return self
    5. Get ~20 movies from resources/sample_files/movies.json and add them to the table. INSERT INTO {table_name} (year, title, release_date, plot) VALUES (?, ?, ?, ?);
    6. Get movies from the table. SELECT title, year from {table_name};
    7. Get a specific movie by title & year (the partition key).
  10. Get a UTC timestamp for the current time.
  11. Update the table schema to add a ‘watched’ boolean column. (UpdateTable) addColumns=[{'name': 'watched', 'type': 'boolean'}]
  12. Update some items as watched. UPDATE {table_name} SET watched=true WHERE title = %s AND year = %s;
  13. Query for items with watched = True. SELECT title, year from {table_name} WHERE watched = %s ALLOW FILTERING;
  14. Restore the table back to the previous state using the timestamp you captured before making the updates. The restore action creates a new table that contains the restored data. Note that the restore operation can take up to 20 minutes, so warn the user (and maybe make this action optional if you do an interactive scenario). (RestoreTable)
  15. Check for completion of the restore action by looping and calling GetTable until status = ‘ACTIVE’.
  16. Delete the table (DeleteTable)
  17. Check for removal of the table by looping and calling GetTable until it doesn’t exist (raises ResourceNotFoundException).
  18. Delete the keyspace (DeleteKeyspace).

NOTE - FOR Java, had to convert the cert to a cassandra_truststore.jks. This was documented here: https://docs.aws.amazon.com/keyspaces/latest/devguide/using_java_driver.html#java_tutorial.SigV4

Actions

CreateKeyspace CreateTable DeleteKeyspace DeleteTable GetKeyspace GetTable ListKeyspaces ListTables RestoreTable UpdateTable

Metadata

In keyspaces_metadata.yaml.

keyspaces_Hello keyspaces_CreateKeyspace keyspaces_CreateTable keyspaces_DeleteKeyspace keyspaces_DeleteTable keyspaces_GetKeyspace keyspaces_GetTable keyspaces_ListKeyspaces keyspaces_ListTables keyspaces_RestoreTable keyspaces_UpdateTable keyspaces_Scenario_GetStartedKeyspaces

Output

Your example should output something like the following:

----------------------------------------------------------------------------------------
Welcome to the Amazon Keyspaces (for Apache Cassandra) demo.
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Let's create a keyspace.
Enter a name for your new keyspace.
The name can contain only letters, numbers and underscores: doc_example_ks
INFO: Keyspace doc_example_ks does not exist.
Created a new keyspace.
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/doc_example_ks/.
The first 10 keyspaces in your account are:

system_schema
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/system_schema/
system_schema_mcs
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/system_schema_mcs/
system
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/system/
system_multiregion_info
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/system_multiregion_info/
doc_example_keyspace_2
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/doc_example_keyspace_2/
doc_example_ks
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/doc_example_ks/
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Let's create a table for movies in your keyspace.
Enter a name for your table: movies
INFO: Table movies does not exist.
Created table movies:
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/doc_example_ks/table/movies
Waiting for your table to be ready...
Your table is ACTIVE. Its schema is:
{'allColumns': [{'name': 'year', 'type': 'int'},
                {'name': 'title', 'type': 'text'},
                {'name': 'plot', 'type': 'text'},
                {'name': 'release_date', 'type': 'timestamp'}],
 'partitionKeys': [{'name': 'year'}, {'name': 'title'}],
 'clusteringKeys': [],
 'staticColumns': []}

The tables in your keyspace are:

movies
        arn:aws:cassandra:us-west-2:123456789012:/keyspace/doc_example_ks/table/movies
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
To connect to your keyspace, you must have a TLS certificate.
Checking for TLS certificate...
Certificate sf-class2-root.crt will be used to secure the connection to your keyspace.
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Added 20 movies to the table:
1. The Great Gatsby
2. Insidious: Chapter 2
3. After Earth
4. Iron Man 3
5. World War Z
6. The Hunger Games: Catching Fire
7. This Is the End
8. Rush
9. We Are What We Are
10. Now You See Me
11. Riddick
12. Gravity
13. Transformers: Age of Extinction
14. Star Trek Into Darkness
15. Thor: The Dark World
16. Divergent
17. Prisoners
19. X-Men: Days of Future Past
20. The Family
21. We're the Millers
Pick one to learn more about it: 15
Thor: The Dark World
        Released: 2013-10-30 00:00:00
        Plot: Faced with an enemy that even Odin and Asgard cannot withstand, Thor must embark on his most perilous and personal journey yet, one that will reunite him with Jane Foster and force him to sacrifice everything to save us all.
----------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------
Let's add a column to record which movies you've watched.
Recorded the current UTC time of 2022-10-25 19:23:13.323520 so we can restore the table later.
Waiting for your table to update...
Column 'watched' added to table.
Let's mark some of the movies as watched. Press Enter when you're ready.

Marked The Great Gatsby as watched.
Marked Insidious: Chapter 2 as watched.
Marked After Earth as watched.
Marked Iron Man 3 as watched.
Marked World War Z as watched.
Marked The Hunger Games: Catching Fire as watched.
Marked This Is the End as watched.
Marked Rush as watched.
Marked We Are What We Are as watched.
Marked Now You See Me as watched.
----------------------------------------------------------------------------------------
The watched movies in our table are:

The Great Gatsby
Insidious: Chapter 2
After Earth
Iron Man 3
World War Z
The Hunger Games: Catching Fire
This Is the End
Rush
We Are What We Are
Now You See Me
----------------------------------------------------------------------------------------
Do you want to restore the table to the way it was before all of these
updates? Keep in mind, this can take up to 20 minutes. (y/n) y
Restored movies_restored to movies_restored at a point in time of 2022-10-25 19:23:13.323520.
Now the movies in our table are:
The Great Gatsby
Insidious: Chapter 2
After Earth
Iron Man 3
World War Z
The Hunger Games: Catching Fire
This Is the End
Rush
We Are What We Are
Now You See Me
Riddick
Gravity
Transformers: Age of Extinction
Star Trek Into Darkness
Thor: The Dark World
Divergent
Prisoners
X-Men: Days of Future Past
The Family
We're the Millers
----------------------------------------------------------------------------------------
Do you want to delete your movies_restored table and doc_example_ks keyspace? (y/n) y
Waiting for the table to be deleted.
INFO: Table movies_restored does not exist.
Table deleted.
Keyspace deleted.

Thanks for watching!
----------------------------------------------------------------------------------------

SDK stories

Exit criteria:

brmur commented 2 years ago

Added to Roadmap.

brmur commented 10 months ago

Reach threshold for minimum SDK languages.