[FEATURE] Make settings.py more extensible

[sdc50]

Currently the settings.py file is abstracted away from the Tethys user and all user manipulated settings are accessed through the portal_config.yml file.

This has the advantage of being simpler and easier to access for users, but takes away some of the capability that the settings.py file provides, e.g. loading settings from environment variables and using Python logic to specify settings.

I propose that we enhance the settings in some/all of the following ways:

• Add an ADDITIONAL_SETTINGS_FILES setting that takes a list of dot-path, importable modules to load into the settings.py
• Add a pre-processor to the portal_config.yml file to expand environment variables

[sdc50]

Thoughts on ADDITIONAL_SETTINGS_FILES:

Example usage in portal_config.yml:

settings:
  ADDITIONAL_SETTINGS_FILES:
  - tethysext.my_extension.settings.py
  - my_package.tethys_addons.settings.py
  - tethys_portal.additional_settings.optional_feature_settings.py

Possible implementation in settings.py:

def get__all__(mod):
    try:
        return mod.__all__
    except AttributeError:
        return [a for a in dir(mod) if not a.startswith('__')]

ADDITIONAL_SETTINGS_FILES = portal_config_settings.pop("ADDITIONAL_SETTINGS_FILES", [])
for settings_module in ADDITIONAL_SETTINGS_FILES:
    mod = import_module(settings_module)
    all_settings = get__all__(mod)
    for setting in all_settings:
        value = getattr(setting, mod)
        setattr(this_module, setting, value)

[sdc50]

@swainn The other thing that I had been thinking about is if we could add more settings into the database rather than having them in the settings.py file.

I think any settings that can be loaded dynamically should be "site-settings" rather than "settings". I'm specifically thinking of BYPASS_TETHYS_HOME_PAGE, but I'm sure there are others that are similar.

If BYPASS_TETHYS_HOME_PAGE were in the database then it's value could be changed through the admin pages and it would be effective immediately without needing a restart.

[swainn]

• Add an ADDITIONAL_SETTINGS_FILES setting that takes a list of dot-path, importable modules to load into the settings.py

Is there a reason you wouldn't want to support absolute file system paths as well? Here's my thought process:

• settings.py files often contain values that you wouldn't want to "package" into a module, which would be the easy way to install into the import path (site-packages).
• The other way to get it into the import path would be to add it to site-packages manually or add it to the PYTHONPATH.
• The former may be challenging if you are working in an environment where you don't know where the site-packages path is (e.g. extending the Dockerfile).

[sdc50]

It seems like absolute system file paths could work as well as long as there's a reliable way to determine that it is a system path rather than a dot-path.

This would be a simple way:

from importlib.machinery import SourceFileLoader

def is_file_path(module):
    return '/' in module

ADDITIONAL_SETTINGS_FILES = portal_config_settings.pop("ADDITIONAL_SETTINGS_FILES", [])
for settings_module in ADDITIONAL_SETTINGS_FILES:
    mod = SourceFileLoader("mod", settings_module).load_module() if is_file_path(settings_module) else import_module(settings_module)
    all_settings = get__all__(mod)
    for setting in all_settings:
        value = getattr(setting, mod)
        setattr(this_module, setting, value)

Maybe a more reliable way would be this:

def is_file_path(module):
    return Path(module).is_file()

[sdc50]

added in #1024