Possible update to use Neptune Serverless?

[reidca]

Feature name Possible update to use Neptune Serverless?

Is your feature request related to a problem? Please describe. The costs of this implementation are mostly the Neptune database, now that Neptune serverless is avialble, can this be used instead to reduce or at least "smooth" the costs?

Describe the feature you'd like to see implemented Change the solution to use Neptune serverless

Describe the value this feature will add to AWS Perspective Reduce costs and make the cost more in line with usage

Additional context The link to Neptune serverless announcement is here: https://aws.amazon.com/blogs/aws/introducing-amazon-neptune-serverless-a-fully-managed-graph-database-that-adjusts-capacity-for-your-workloads/?ck_subscriber_id=1447743506

[svozza]

Currently, Neptune Serverless does not support deployment with CloudFormation and is only available in 5 regions so we cannot change our templates yet. Something to note too, is that using the minimum NCU value for serverless Neptune is more expensive than the r5.db.large the solution uses as default.

[reidca]

Thank you for your quick reply.

Understand completely why cloudformation and regional support are needed.

Regarding costs, is it required for the database to always be available? Is there a way in future to scale to zero using the serverless model? For example outside of working hours? Is this possible even on the current version using a schedule?

Whilst we love what the team have done with this solution it feels like this is something that should just be part of the AWS ecosystem rather than something the customer has to pay for therefore we are looking for ways to reduce the costs.

[svozza]

The current minimum value you for NCUs (Neptune Capacity Units) that Neptune Serverless supports is 2.5, as documented here. I'm not sure what the future plans for being able to scale to zero are as I'm not on the Neptune team and don't have access to their backlog.

Something you could do to reduce costs is to have a scheduled lambda to execute the commands detailed in the Cost Optimisation sections of the docs here. It could turn the database and scheduled task off on Friday evening and restart them on Monday morning.

[reidca]

That is probably a good idea. Perhaps one that could be incorporated into the solution?

It might be useful to explain in the linked document what the impacts of having the event rule and database stopped are. If there are no reasons NOT to have it disabled (when you are not using it) then should this be the default setup? Enable when you want to perform a discovery session, turn off afterwards?

I would be interested to understand more of the tradeoffs.

Thanks

[svozza]

The discovery process can be turned off for as long as required, the next time it runs it will calculate the delta from the last time it ran and update the database accordingly.

I'm hesitant to make this the default operation, doing so would require the ability to turn the database off using the UI, which would mean anyone with a log in to the tool could do that. I think we'd need to implement some sort of RBAC system first before we could do this.

[reidca]

OK thank you for the confirmation. This should reduce the costs significantly since the majority of them seem to be from the compute costs of running Neptune. I will try shutting it down when not in use and report back any findings of interest.

[svozza]

We will be adding the option to deploy with Neptune serverless in v2.1.0.

[eliasbedmar]

It looks like this isn't an option in the CF Parameters of v2.1.0

[tmekari]

@eliasbedmar If you scroll up on the NeptuneInstanceClass CF Parameter, you should see "db.serverless" as an option. Please note that you also have to set MinNCUs and MaxNCUs if you'd like to use it.

[eliasbedmar]

Thank you

the db.serverless option wasn't available on previous version but I can now see it on v2.1.0

On Wed, 27 Sept 2023 at 00:13, Tim Mekari @.***> wrote:

@eliasbedmar https://github.com/eliasbedmar If you scroll up on the NeptuneInstanceClass CF Parameter, you should see "db.serverless" as an option. Please note that you also have to set MinNCUs and MaxNCUs if you'd like to use it.

— Reply to this email directly, view it on GitHub https://github.com/aws-solutions/workload-discovery-on-aws/issues/314#issuecomment-1736425210, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALQXTGUZYQNAITUQOGPKC6LX4NOQXANCNFSM6AAAAAARUKD4FM . You are receiving this because you were mentioned.Message ID: @.***>

--

Elias Bedmar @.***