Separating arguments processing out of recipe `run` function

[TrangNguyenBC]

This PR solves a part of issue #40 Example of "config_durations" - need review

• Split recipe args into QUERY_ARGUMENTS and PP_ARGUMENTS. Common arguments (in ARGUMENT_GROUPS) are passed by strings while specific arguments are listed fully (key, type, dest, default, help)
• Remove the query context out of run function:
• In run function, each specific recipe will call execute_query(query_name, new_context=None). The recipe only has to know the name of query it calls. In case the previous processing change context, they only have to pass the "changed context", not the whole context.
• The recipe class will do the argument parsing and has the execute_query method which really call run_query(query_name, G_CONFIG, **new_context).

[TrangNguyenBC]

Hi @ahal. Following our discussion, I submitted the new PR which implements another structure of queries and recipes: Writing query involves of query context definition and query template, writing recipe involves of query calls (by only query names) and post-processing arguments.

• Consider a query is the smallest unit instead of recipe, which has: 1) definition of query context, 2) query template. We build a common definition of context for query (the same with ARGUMENT_GROUPS now). Almost query files use the similar common definition in ARGUMENT_GROUPS so almost no need to modify query files, also this basically eliminate FAKE_CONTEXT by using default values of common context definitions.
• In recipe, we only need to define post-processing parameters. List of queries inside a recipe can be retrieved by inspecting run() function. We can extract query list using inspect and ast libraries and got the list of query names as the first string parameter of run_query function call. From the list of query names, the recipe.py can get the list of query definitions and expose these to argument parsers in CLI of HTML web form. With this structure, the web tool can have build query function, which allows the user to write raw query (the same with how Active Data Query Tool works now), then generate a query by using {$eval: }, then users can submit it to query pool and wait for administration approval. This code still need to be updated (only implementing config_durations recipe as an example).

[TrangNguyenBC]

Hi @ahal I refactor the code using some terms to make code clearer.

1. Terms and meanings:
   • A context includes: name, definition, value. For example with this context definition below, the context name is sort_key (‘dest’) and the context value can be any int number: [['--sort-key'], {'type': int, 'dest': 'sort_key', 'default': 4, 'help': "Key to sort on (int, 0-based index)", }]
   • The context definition of query/run/recipe is an array of corresponding context definition. In detailed, query context definition (query_context_def) is an array of context definitions of all contexts which will be used in this query; Run context definition (run_context_def) is an array of context definitions of all contexts which will be used for post-processing in run function of this recipe; Recipe context definitions (recipe_context_def) is an array of context definitions of all contexts for this recipe (a combination of query_context_def and run_context_def).

• When writing the query or recipe, no need to give the context definitions of contexts in COMMON_CONTEXTS, so we will have kind of “shortcut” to query/run context definition, which can be call query_context or run_context. It is a mix array of context definition and string. For example, if “limit” is in COMMON_CONTEXTS while “sort-key” isn’t, a run_context can be like this: ["limit", [['--sort-key'], {'type': int, 'dest': 'sort_key', 'default': 4, 'help': "Key to sort on (int, 0-based index)", }]]

In this case, “sort-key” is a specific run context of this recipe. A query also can have it own specific query context.

• Arguments (args) are the set of context names and context values given by the users through command lines or webpage.

2. Implementation:
   • The query file (*.queries - YAML file) has context names of common contexts and/or context definitions of specific query contexts. For example, in config_durations.queries we have in: {repo.branch.name: {$eval: branch}} with “branch” is a context name.

• The query.py file: the run_query function reads the query file, extract the query itself and execute it.
• The recipe file (*.py - * is recipe name) has 1 required function (run) and 2 optional functions (get_queries and get_run_context). The run function call execute_query function supplied by recipe.py with 2 parameters: name of query (required), and override context (optional). The override context is used in case the result of previous query affects context values of subsequent query.
• The recipe.py file: the run_recipe function does: First, extract the list of queries called in the recipe and the recipe context definition automatically from the run function of the recipe file. If there are get_queries and/or get_run_contexts functions, use these functions; Then, extract the the query context definition automatically from the query file by 3 steps: uses $eval as navigator to extract the context name; extracts the specific context definition if there are; makes the query_context_def by combination of the definitions of contexts in COMMON_CONTEXTS and other specific context definitions.

[ahal]

Hi @ahal I refactor the code using some terms to make code clearer.

1. Terms and meanings:

Thanks for the explanation, this is a lot clearer!