Should the target optionally create an athena catalog ?

[andrewcstewart]

In Athena parlance, catalog is basically analogous to database and database is analogous to schema. We are currently creating the "schema" if it doesn't exist, but should we also potentially create a catalog?

[andrewcstewart]

Some more info: https://docs.aws.amazon.com/cli/latest/reference/athena/create-data-catalog.html

[aaronsteers]

@andrewcstewart - For my part, if it's feasible to do so, I like the idea of "no setup required" and the target can auto-create its own environment.

The reverse argument, I think, is that you might not want to accidentally create a new catalog, for instance if the user accidentally sets the wrong region (true story 😆 ). This could be optionally be mitigated by (and perhaps in a future update) a setting value of allow_create_catalog or similar.

[andrewcstewart]

That seems reasonable. Another thought is that because the default awsdatacatalog catalog already exists for accounts, we could simply use the setting of any other value of a athena_catalog config param as inidiication to create if not exists.

[andrewcstewart]

On second thought, it looks like creating additional catalogs for Athena is not at all trivial.

[aaronsteers]

@andrewcstewart If overly complicated and requiring advanced config, I think it's perfectly reasonable to not have this feature. And especially since the default one is auto-created and we can easily get new/disposable namespaces with the schema/database layer.

[andrewcstewart]

Agreed. Closing this as out of scope.