Event-Driven-Architecture starter with Cement, Dramatiq, Grpc, NiceGUI

[TomFreudenberg]

Cement Issue Reporting and Feature Requests

Thanks for your work!

I have created a starter repository to use dramatiq and cement. Grpc is coming next.

Maybe this is interesting for you to see: https://github.com/tokeo/tokeo

[TomFreudenberg]

@derks could you please guide me a moment.

I want to implement a diskcache based app handler in the Tokeo cement app.

Would you decide to implement as Extension or as Plugin?

I am struggling when and for what to use plugin or extension.

Thanks in advance Tom

[derks]

Quick reply (on mobile)... think of extensions as application agnostic, where plugins are application specific. Extensions extend the framework, plugins extend YOUR application.

[TomFreudenberg]

Thx! Ok, so implementing diskcache is extension like your's memcached and rediscache.

I checkout and send an PR when ready - maybe it is an extension in general

[TomFreudenberg]

Dear @derks another short question I didn't find yet.

Is there a way already defined by Cement (you) to access the initialized app object global from outside controller, handler etc.

E.g. I wrote some actors for dramatiq inside my app and there I need to access app.

I won't push the app always forward thru the different instances (too lazy). Currently I created a global_module to export app.

To export app in the main.py is not possible because of circular module errors afterwards.

Feels a bit quirky so I wanted to ask if you have something in mind?

[derks]

Apologies, on mobile. There is not officially. I would have to workout some ideas to give a good suggestion. If you had an MVCE it would help... I will try to work out an example idea though.

[TomFreudenberg]

Hi @derks thanks for quick feedback again.

So in case that you don't have something prepared, I will create an extension like 'share_app' and share the app object in that extension and register by hook.

[TomFreudenberg]

Hi @derks

it works with this very small extension, when enabled as extension.

I call it appshare :-)

In main.py just added to the extensions list:

    class Meta:

        extensions = [
            ...
            'tokeo.ext.appshare',
            ...
        ]

The file: tokeo/ext/appshare.py

"""

This module shares a globally access to the running app object

"""

class App():

    def __init__(self):
        self._app = None

    def __getattr__(self, key):
        # test _app object
        if self._app is None:
            raise AttributeError(f'\'App\' object has no attribute \'{key}\'')
        # return attribute
        return getattr(self._app, key)

app = App()

def load(app_to_share):
    app._app = app_to_share

and then it's accessible somewhere, e.g. actors by:

from tokeo.ext.appshare import app

def some_method():
    app.log.info('Here we are!')

[TomFreudenberg]

Hi @derks

after reading the foundation.py I realized to directly use the load() hook for extensions to directly activate the extensions. So now it's easy to setup just via extension.

Pretty fine

[TomFreudenberg]

Dear @derks

I switched over all my developments to this open source project:

https://github.com/tokeo/tokeo

I would be very happy if you could have a look on that especially about the extensions:

https://github.com/tokeo/tokeo/tree/master/tokeo/ext

Maybe you like also some of them to include in Cement core as well but not sure about this:

appshare.py | allow access to the main app object diskcache.py | use diskcache.py as key/value cache and lock dramatiq.py | include dramatiq grpc.py | run a grpc server from proto files pocketbase.py | use pocketbase as db scheduler.py | integrate Apscheduler and have an interactive cron style scheduler smtp.py | changes as you already know

---

I am very pleased, when you check the source codes and give me some feedback if I used the overall Cement framework conceptions in the right manner.

Thanks in advance for your feedback Tom

[TomFreudenberg]

Hi @derks just a question if you have decided already how to handle different prod(uction) test(ing) or dev(elopment) configs? Currently I have not found some kind of this management. Only placing the config in different folder /etc /project etc. Thanks for a short note if yes or no. Cheers, Tom

[TomFreudenberg]

@derks Sorry for coming up again, but could you please give me your guidiance about config:

Hi @derks just a question if you have decided already how to handle different prod(uction) test(ing) or dev(elopment) configs? Currently I have not found some kind of this management. Only placing the config in different folder /etc /project etc. Thanks for a short note if yes or no. Cheers, Tom

Does ist make sense to add python-dotenv in addition or is there something similar already in place?

[derks]

Hi @derks just a question if you have decided already how to handle different prod(uction) test(ing) or dev(elopment) configs? Currently I have not found some kind of this management. Only placing the config in different folder /etc /project etc. Thanks for a short note if yes or no. Cheers, Tom

I have had this use-case in other projects, but have not implemented anything officially. I'll have to try out your extension... I think one approach I would be interested in is something like profiles. Where you can have a set default_profile, but then be able to override:

# dynamically override profile
myapp -p <other_profile> ...

# set active profile
myapp profile set <other_profile>

# get active profile (default command)
myapp profile

default
profile-1
profile-2 **
other-profile

Something like that.... but I'll need to spend some time on it to consider a variety of use-cases.

[derks]

Dear @derks

I switched over all my developments to this open source project:

https://github.com/tokeo/tokeo

I would be very happy if you could have a look on that especially about the extensions:

https://github.com/tokeo/tokeo/tree/master/tokeo/ext

Maybe you like also some of them to include in Cement core as well but not sure about this:

appshare.py | allow access to the main app object diskcache.py | use diskcache.py as key/value cache and lock dramatiq.py | include dramatiq grpc.py | run a grpc server from proto files pocketbase.py | use pocketbase as db scheduler.py | integrate Apscheduler and have an interactive cron style scheduler smtp.py | changes as you already know

I am very pleased, when you check the source codes and give me some feedback if I used the overall Cement framework conceptions in the right manner.

Thanks in advance for your feedback Tom

Really impressed with the work you've done, and am happy you've chosen Cement for it. I definitely want to set aside time to review how you've used the framework, as well as the extensions you've built. I will do my best to provide feedback (now that 3.0.10 is done).

[TomFreudenberg]

Thanks @derks for your feedback.

I have created an appenv extension now for the config part and be very happy with that.

You may have a look here:

https://github.com/tokeo/tokeo/blob/master/tokeo/ext/appenv.py

That allows to have config like:

config/app.yaml
config/app.development.yaml
config/app.development.local.yaml
config/app.production.yaml
config/app.production.local.yaml

So you can put secrets as well into ENV_VARS like you defined by cement or put them into the .local yaml config.

You can start your app by:

MYAPP_ENV=dev myapp --help

or

MYAPP_ENV=production myapp --help

Settings get merged by cement config handler (last setting wins)

---

I really appriciate the work with cement - thanks for that piece of codes

[TomFreudenberg]

Dear @derks

to "navigate" in project folder, I need the location directory of the main.py from my cement app.

For that I have added an element to Meta but wonder if you already had something prepared I missed currently. That is the solution currently:

main.py

class Tokeo(App):
    """The Tokeo primary application."""

    class Meta:
        # this app name
        label = 'tokeo'

        # this app main path
        main_dir = os.path.dirname(fs.abspath(__file__))

        # configuration defaults
        config_defaults = dict(
            debug=False,
        )

It's just the line about main_dir of interest

In the appenv extension I can know prepare some APPDIR vars relatively based on app._meta.main_dir

Do you have another suggestion or is this a proper way?

[TomFreudenberg]

Hi @derks

today I added a YamlConfigParser which will deep merge nested yaml configurations.

Maybe this would be an interesting feature to put directly into the framework at ConfigParser level?

Checkout: https://github.com/tokeo/tokeo/blob/master/tokeo/ext/yaml.py