Engine-wide table name prefix

[tomislacker]

Observation/Situation

When working with a multi-tenant AWS account, it's a reasonable practice to prefix table names with some sort of token to indicate what tenant/application a table belongs to. This same design pattern is also relevant for having tables for different environments, such as dev/qa/prod.

Hacky Failed Attempt

I attempted to handle this with a sort of configuration parameter for a table prefix, which I could them update the class models' __metadata__ attributes with, but it did not work:

APP_NAME = 'abc'
APP_ENV = 'dev'
TABLE_NAME_PREFIX = '-'.join([
    APP_NAME,
    APP_ENV,
])

# Register our models with the engine
for this_model in [
    OAuthUser,
    User,
]:
    # Insert the table name prefix
    if '_name' in this_model.__metadata__:
        print("Updating {}.__metadata__".format(this_model.__name__))
        print("  (old)_name='{}'".format(this_model.__metadata__['_name']))

        this_model.__metadata__.update({
            '_name': TABLE_NAME_PREFIX + this_model.__metadata__['_name'],
        })

        print("  (new)_name='{}'".format(this_model.__metadata__['_name']))

    engine.register(this_model)

engine.create_schema()

>>> Updating OAuthUser.__metadata__
>>>   (old)_name='oauth_user'
>>>   (new)_name='abc-dev_oauth_user'
>>> Updating User.__metadata__
>>>   (old)_name='user'
>>>   (new)_name='abc-dev_user'

This resulted in the creation of tables that were still only named oauth_user and user.

Potential Resolutions

• Add ability to the engine to set an engine-wide table prefix string

• Add ability to the engine to set an engine-wide table name function

  def table_namer(original_name):
    return "my-prefix_" + original_name
  
  engine.set_name_function(table_namer)

  This implementation may also resolve @stevearc's enhancement request in

  4 as the table_namer function could independently handle the naming of

  the table with a timestamp:

  import datetime
  
  def table_namer(original_name):
    if original_name == 'timeseries_table':
        now = datetime.datetime.now()
        return ''.join([
            original_name,
            str(now.year),
            str(now.month),
        ])
    else:
        return original_name

[stevearc]

I think this is one of those rare cases where I just have a solution lying around. Try passing in a namespace to the Engine constructor. I think that does what you want. See: http://flywheel.readthedocs.io/en/latest/ref/flywheel.engine.html#flywheel.engine.Engine

[tomislacker]

@stevearc that's perfect, I'm sorry I didn't see that earlier. Any thoughts on the idea of being able to pass a callable to it?

[stevearc]

Might be possible depending on what it's resolving to and when you need it to resolve to a different prefix. What's the specific use case?

[tomislacker]

One such use-case would be like what I believe you were describing in #4 where you'd want a table to be named something like "timeseries_data20160513" ("timeseries_dataYYYYMMDD")

[stevearc]

I think that's very tricky, and that putting it in via a callback at the engine-level probably wouldn't work very well. You'd want some controls for which table to use for any given query (and whether to bridge across tables). I'd want to do a lot of thinking about how to redesign the API for something as complex as time-sharded tables.

If that's an immediate need for you, you could try directly setting MyModel.meta_.name = 'timeseries_data_20160513'. The changes will be local to the model (instead of engine-wide) and don't require any new features.

[tomislacker]

@stevearc If you don't foresee the wildcarded sort of naming structure as being a feature that may creep in, I believe this issue can be closed.