Generate dynamic module for custom UDF

[ckunki]

Factory on the fly

See

• https://joeletts.com/2020/creating-dynamic-modules-python-3/
• https://www.b-list.org/weblog/2007/dec/03/making-magic/

Tasks

• [x] Investigate creating dynamic modules
• [x] Use dynamic modules for adhoc implementation of an example Query Handler
• [x] Integration test for example based on dynamic modules

[ckunki]

Currently when using

--/
CREATE OR REPLACE PYTHON3_AAF SET SCRIPT "MY_SCHEMA"."MY_QUERY_HANDLER_UDF"(...)
EMITS (outputs VARCHAR(2000000)) AS
xyz = create_module("xyz", [ExampleQueryHandler, ExampleQueryHandlerFactory])
/ 

and

EXECUTE SCRIPT MY_SCHEMA.AAF_RUN_QUERY_HANDLER('{
    "query_handler": {
        "factory_class": {
            "module": "xyz",
            "name": "ExampleQueryHandlerFactory"
        },

I get the error message

udf_framework/query_handler_runner_udf.py", line 215, in _create_state
    module = importlib.import_module(self.parameter.python_class_module)
    ModuleNotFoundError: No module named 'xyz'

Maybe we need to call create_module() in query_handler_runner_udf.py?

[ckunki]

Now, when using

new_module = DynamicModule("xyz").add(ExampleQueryHandler)
import xyz

class ExampleQueryHandlerFactory:
    def create(self, parameter: str, query_handler_context: QueryHandlerContext):
        return xyz.ExampleQueryHandler(parameter, query_handler_context)

new_module.add(ExampleQueryHandlerFactory)

... I get the error message

_pickle.PicklingError: Can't pickle <class 'ExampleQueryHandler'>: 
it's not found as builtins.ExampleQueryHandler
'" caught in script "MY_SCHEMA"."AAF_RUN_QUERY_HANDLER"
 at line 628 

[ckunki]

I only was successful when using the module builtins:

new_module = DynamicModule("builtins").add(ExampleQueryHandler)
import builtins

class ExampleQueryHandlerFactory:
    def create(self, parameter: str, query_handler_context: QueryHandlerContext):
        return builtins.ExampleQueryHandler(parameter, query_handler_context)

new_module.add(ExampleQueryHandlerFactory)

[ckunki]

Obsolete tasks from initial plan

My initial plan included the following tasks

• [x] Split example by language: SQL, python, json
• [x] Nox task for updating the example in the user guide
• [x] Developer Guide describes how to update the user guide
• [x] GitHub workflow verifies if user guide is up-to-date

These tasks have now been replaced

New plan

Remove sentence

For this the proxy maintains a reference counter.

As this is an irrelevant implementation detail and also not 100% correct in this place

• Move content to separate explicit files 1) create script (sql incl. python) 2) execute script (json)
• Change user guide to just point to these two files
• Change integration test
  • use the identical DB schemas as defined by the example (MY_SCHEMA and TEMP_SCHEMA)
  • create and clean up the temporary schemas
  • replace the bfsconnection name string in (2)