VP code summary- How does code generator work?

[xp1632]

Now I'll do a final summary of generator_related_code in the VP project:

---

visual_program:

• we define a class that is composed of Nodes and Edges.
• and we have related utility functions for Nodes and Edges
  • isConnected, getIncomingNodes(), getNodeById(), etc.

• This is a quick recap of the structure and its terms:

generation result

• This is a class we defined to store the generation result of code from the visual program

• we have GenResult --Extends-> FunctiongenRes-->NodeGenRes --> InputGenRes

• Basic GenResult includes message and code

• FuncGenResult includes one more imports

  • and toGenResult function that add imports back to code

• InputGenRes includes one more prerequisiteCode that stores the required code that should be executed before current node

---

code_generator and python generator

• code_generator ---extends--> python generator

• In code_generator, we define the method we need to generate codes from VisualProgram

• programToCode() which do:

  • checkEntryPoint : check the main start of a program
  • topoSortFuncsInProgram: sort the order of nodes for correct dependency of each other
  • functionsToCode: that check if the program is not recursive and call functionToCode for every Node or function

---

• there are also abstract method we defined here that should be implemented in python_generator or other languages' generator

• For example, we implement the required methods of code_generator in python_generator

• nodeToCode: this is the main function that do the generation from VP node to Python Code

  • for every Input of current Node
  • we take widgetValue if this Node is a valueNode by quote()
  • we takes care of ImageConversion if input is Image
  • we add the code previous Node needs to current Node's prerequisiteCode
  • for every Output,
  • we also check the beWatched flag
  • and achieve the goal of showing intermediate result by captureImageCode()

[xp1632]

generators/visual_program.ts

• We start with understanding visual_program in VP

  • because it's used in code_generator.ts file

    This is the visual summary of class and method in visual_program and index.ts:

[xp1632]

• FunctionCallsFunc

• We search for FunctionCalls Functions from startNode by configType of function call

[xp1632]

generators/generation_result.ts

• These are the classes and its grandclasses we used in generation result.
• Imports, we used Set to store imports to avoid duliplication

[xp1632]

code_generator.ts

• code_generator is the basis of the following python_generator
• here we
  • create a new instance to hold code result
  • check main node of flow
  • topo sort the node list based on the dependency of nodes function calls and definition pair
  • call functionToNode method we defined in python_generator for every node in sortedNodeList

[xp1632]

python_generator.ts:

• This file is more complex than I thought so I'll first write down the basic understanding to help sort it out

• NodeToCode():

  • This function processes the input and output of a node to python code we need
  • we create result with NodeGenRes type to store generation result

  • we create prerequisiteCodeOfInputs to store the code should be executed before our current code

• For every Input, we have getInputValueOfNode() method to

  • • For Node that input connects with no other handles
    • indicates this node itself is a valueNode
    • we get its value and convert it to python format
    • details are in #57
    • For ImageInput
    • set converter between imageInput and imageOutput
    • add imports that convertFunction needs
    • For ValueInput
    • We calls nodeToCode() for its incomingNode
    • And put the code we get to currentNode's prerequisiteCode
    • add imports for incomingNode's code's import

  • Then we check the prerequisiteCode of this input

    • To avoid duplication, we only add the prerequisiteCode of current input when it's not already in the main prerequisiteCodeOfInputs

[xp1632]

• Then we do a similar process for node's output:

• ---

• For every output of node:
• we call getOutputValueOfNode() and add the next exec output node's code via this.nodeToCode(outgoingExecNode, program);

[xp1632]

• After going through the inputs and outputs of current node

  

• we add current externalImports to result.imports

• we add prerequisiteCode of All Inputs into result.code

[xp1632]

CaptureImageCode function for intermediate result:

• This function is also a bit confusing for me, so I'll just write down my current understanding:

• After the main code part of node is generated and before we return the final generation result
• We also have a function to generate the code that should capture Intermediate image of a node's output:

• which's the flag of an output's beWatched

  captureImageCode():

• if the captureImageFunction of this PythonGenerator class is not defined.
• we define one here, which captures the image and encodes the image into base64

---

• If this.commInJupyterLab is not defined, it defines a new Jupyter comm for capturing images and stores it in this.commInJupyterLab. The comm's name is also based on the current timestamp.

-Then it gets a conversion from the current data type of the image to a numpy array using the getConversion method of this.imageTypeConvert.

---

• At last, it generates Python code that converts the image to a numpy array, converts the numpy array to a PIL image, and calls the capture image function with the PIL image and the given DOM element ID. The code is returned as a FunctionGenRes object, along with any necessary import statements and the definitions of the capture image function and the Jupyter comm.

---

In summary, the captureImageCode function generates Python code that captures an image from a given variable, converts it to a base64 string, and sends it to a specific DOM element in a Jupyter notebook.

[xp1632]

More information about getInputValueOfNode and getOutputValueOfNode

• I checked the NodeTypeSpecification.md again in https://github.com/c3di/Visual-Programming-Language/blob/main/doc/NodeTypeSpecification.md

• And found something that I overlooked: In here: https://github.com/c3di/Visual-Programming-Language/blob/main/doc/NodeTypeSpecification.md#generating-a-codegenerator-for-python

---

/**
 * Generates the Python code for an exec node given its inputs and outputs.
 * The function can be adapted to handle more complex scenarios and various metadata types.
 *
 * @param inputs An array of strings representing literals or variable names e.g., '1', '"str"'. 
 * @param outputs An array of strings for variable names or subsequent Python code.
 * @param node The node for which code is being generated.
 * @param generator An object providing context and utility functions for code generation.
 *
 * @returns  A string containing the generated Python code snippet.
 */
function code(inputs, outputs, node, generator) {  
 // 1. Read an image and assign it to an output variable.
 // 2. Create a metadata dictionary and attach when dealing with image data.
 // 3. Prepare the execution sequence for the subsequent code.
  const code = `${outputs[1]} = io.read_image(${inputs[1]}, ${inputs[2]})
${outputs[1]} = {
  'value': ${outputs[1]},
  'dataType': 'torch.tensor',
  'metadata': {
    'colorChannel': 'rgb',
    'channelOrder': 'channelFirst',
    'isMiniBatched': false,
    'intensityRange': '0-255',
    'device': 'cpu'
  }
}
${outputs[0]}`;
  return code;
}

// Convert the code generation function to a string format for JSON configuration.
const codeGeneratorFunction = code; // Assign the function for conversion
const jsonFormattedString = JSON.stringify(codeGeneratorFunction.toString());
console.log(jsonFormattedString);

• it says:

• @param inputs An array of strings representing literals or variable names e.g., '1', '"str"'.
• @param outputs An array of strings for variable names or subsequent Python code.

• so in getInputValueOfNode()

• result.code = this._getUniqueNameOfHandle(incomingNode, outputHandle!);

• is actually adding literal of input value such as '1' (when the value we get from the outputHandle from previous node is 1) to InputGenRes.code

• And this InputGenRes.code for this input is added to inputs[] in method nodeToCode() when iterating the inputs of a node

• Then inputs[] is passed to nodeSourceGeneration()

nodeSourceGeneration(
    node: Node,
    inputs: string[],
    outputs: string[]
  ): string {
    const nodeGenerator: string = node.data.codeGenerator;
    if (!nodeGenerator) {
      console.warn(
        `Node type:${node.data.configType as string} has no source code`
      );
      return '';
    }
    // eslint-disable-next-line no-eval
    const func = eval(`(${nodeGenerator})`);
    // remove the trailing new line
    return func(inputs, outputs, node, this).replace(/\n+$/, '');
  }

• and we use eval function to make the nodeGenerator into JS functioning code

---

• Supplementary information about eval
  • simple example:

    let code = "console.log('Hello, World!')";
    eval(code);  // This will print "Hello, World!" to the console

  • our case of nodeGenerator
  • nodeGenerator is we get from a Node's JSON file's nodeGenerator item
  • like let nodeGenerator = "(inputs, outputs, node, context) => { /* function body here */ }";
  • And we interprete the string as a function definition by let func = eval((${nodeGenerator}));

---

• create a new function func,
• then we call func with parameter inputs[], outputs[] we get from NodeToCode()'s getInputValueOfNode and getInputValueOfNode in return
• Just as we quoted before, inputs are known values' literal, output are not known yet so it's variable names, this variable name is the this._getUniqueNameOfHandle(node, outputId) , which is n_${node.id}_${handleId}:

  • @param inputs An array of strings representing literals or variable names e.g., '1', '"str"'.
  • @param outputs An array of strings for variable names or subsequent Python code.

---

python_generator.ts:

• This file is more complex than I thought so I'll first write down the basic understanding to help sort it out

• NodeToCode():

  • This function processes the input and output of a node to python code we need
  • we create result with NodeGenRes type to store generation result
  • we create prerequisiteCodeOfInputs to store the code should be executed before our current code

• For every Input, we have getInputValueOfNode() method to

  - **For Node that input connects with no other handles**
  - indicates this node itself is a `valueNode`
  - we get its value and convert it to python format
  - details are in #57 
  - 
  - **For ImageInput**
  - set `converter` between `imageInput` and `imageOutput`
  - add imports that `convertFunction` needs
  - 
  - **For ValueInput**
  - We calls `nodeToCode()` for its `incomingNode`
  - And put the code we get to `currentNode`'s `prerequisiteCode`
  - add imports for `incomingNode`'s code's import 

• Then we check the prerequisiteCode of this input

  • To avoid duplication, we only add the prerequisiteCode of current input when it's not already in the main prerequisiteCodeOfInputs