Python Common utilities?

[bschroeter]

As per slack chat 21/09/23, there is an argument for a set of common Python utilities across the NRI. While standard libraries are fine on their own, there is often a lot of repeated boilerplate code in order to use them.

Some examples:

• Creating a logger with a standard output format that points to stdout,stderr,files etc.
• Executing shell commands safely and efficiently with appropriate command escaping.
• Various decorators (singletons, observers etc.)
• File system operations.
• PBS interaction (experience suggests that there is a bit of fiddling required for Gadi when using standard tools - personally this is probably a package on its own!)

I'd like to propose that we put together a Python package (accessnri-common?) that would simplify boilerplate code, reduce repetition and encourage a "standard way" of doing things across the NRI.

For instance:

import accessnri_common as ac
logger = ac.get_logger() # Arguments TBA

Instead of...

import logging

logger = logging.getLogger(__name__)

# Create handlers
c_handler = logging.StreamHandler()
f_handler = logging.FileHandler('file.log')
c_handler.setLevel(logging.WARNING)
f_handler.setLevel(logging.ERROR)

# Create formatters and add it to handlers
c_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
f_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
c_handler.setFormatter(c_format)
f_handler.setFormatter(f_format)

# Add handlers to the logger
logger.addHandler(c_handler)
logger.addHandler(f_handler)

Obviously there would be some discussion around standard configuration etc. but I really like the idea of calling 2 lines of code each time I need a logger in a script, as opposed to building one from scratch.

I've personally written a lot of this stuff many times, so the time/effort required is somewhat shortened to bring snippets from various projects.

Now, the issue I see here is the concern about "another thing to support". I would see this as more internal than external-facing, helping to speed up ACCESS-NRI development rather than providing a supported package for the community to use (of course they would be welcome to).

Discuss :)

[headmetal]

@aidanheerdegen @rbeucher @dougiesquire @atteggiani

[dougiesquire]

I'll preface my comment by noting that my work typically doesn't involve a lot of the functionality you're describing, or where it does, I've found the using standard libraries directly to be not too burdensome. So my opinion may be ill-informed.

I'm not a huge fan of "kitchen sink" packages like this. IMO, packages should have a clear scope and function.

I would see this as more internal than external-facing, helping to speed up ACCESS-NRI development rather than providing a supported package for the community to use

As in, we wouldn't include this as a dependency of any of our supported tool? If so, sure, though it would be good to know how many ACCESS-NRI developers would use this. If not, then it's still "another thing to support" isn't it?

[rbeucher]

I second @dougiesquire. I am generally against kitchen sink packages. We tend to rely on them too much and get lost when they are not available. There is also the burden of maintaining them.

[harshula]

Please update https://github.com/ACCESS-NRI/dev-docs/wiki with best practice coding patterns as you discover or utilise them.