confirm message options for Neptune

[cmharlow]

SQS is confirmed, but what are other options, and is SQS the best of those?

Scope of this is: every completed load (update) on the triplestore should issue an event that causes derivative stores to recreate themselves.

some optimizations worth exploring:

• can the event / message be at the bulk load level
• can there be partial updates to the derivate store(s) (i.e. just add / recreate "records" that changed)
• should there be queue management at the ETL transform code level, or just continuous messages to the service?

[mjgiarlo]

The resource management API within Neptune provides the ability to create, modify, and delete event subscriptions [1,2] which is an integration point with SNS (akin to RDS's more fully documented event subscription integration [3]). An event subscription:

• must have a subscription named
• must have an SNS topic ARN
• may have one or more source identifiers
• may have one source type, chosen from among db-instance | db-cluster | db-parameter-group | db-security-group | db-snapshot | db-cluster-snapshot
• may be enabled or disabled
• may have one or more event categories
• may have one or more tags

1. https://docs.aws.amazon.com/neptune/latest/userguide/API_CreateEventSubscription.html
2. https://docs.aws.amazon.com/neptune/latest/userguide/API_EventSubscription.html
3. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.html

[mjgiarlo]

Another possibility is to monitor a queue of notifications emitted by Neptune's CloudTrail integration -- Amazon suggests consuming these audit events via SNS and SQS together -- though we'd need to confirm:

• Neptune SPARQL Update API calls trigger CloudTrail audit log events
• Parsing (SPARQL Update) requests is "good enough." Ideally the events that populate derivative datastores would be generated by creates and updates, not by requests to do creates and updates. Perhaps a fallback option here is to parse affected URIs out of the SPARQL Update request and then do a SPARQL query for each of these URIs?

[mjgiarlo]

From the Neptune event subscription documentation (HT: @jcoyne):

You can see a list of the categories for a given SourceType by using the describe-event-categories action.

Once we have an instance of Neptune accessible to us, we should be able to query it via the AWS CLI as follows:

aws neptune describe-event-categories

[mjgiarlo]

For the meantime, if we assume that RDS and Neptune support the same event categories, here's what we'll get from the above command: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.html#USER_Events.Messages

This set of event categories does not seem to support events at the level of granularity we need (data inserts/updates/deletes), but we should confirm this rather than just assume it.

[mjgiarlo]

Here is the output of aws neptune describe-event-categories:

{
    "EventCategoriesMapList": [
        {
            "EventCategories": [
                "restoration", 
                "read replica", 
                "creation", 
                "backup", 
                "recovery", 
                "deletion", 
                "backtrack", 
                "low storage", 
                "maintenance", 
                "failover", 
                "failure", 
                "configuration change", 
                "notification", 
                "availability"
            ], 
            "SourceType": "db-instance"
        }, 
        {
            "EventCategories": [
                "failure", 
                "configuration change"
            ], 
            "SourceType": "db-security-group"
        }, 
        {
            "EventCategories": [
                "configuration change"
            ], 
            "SourceType": "db-parameter-group"
        }, 
        {
            "EventCategories": [
                "restoration", 
                "creation", 
                "deletion", 
                "notification"
            ], 
            "SourceType": "db-snapshot"
        }, 
        {
            "EventCategories": [
                "failover", 
                "failure", 
                "notification"
            ], 
            "SourceType": "db-cluster"
        }, 
        {
          "EventCategories": [
                "backup"
            ], 
            "SourceType": "db-cluster-snapshot"
        }
    ]
}

[mjgiarlo]

The above output is identical to that of aws rds describe-event-categories.

[mjgiarlo]

Closing this issue. See above for details about messaging options within Neptune.