[Lambda Function] Building a Knowledge Graph from a Go Codebase Using ASTs

[tomsmith8]

Task Overview

Codebase: https://github.com/stakwork/sphinx-tribes Language: go

Develop a tool or script that can be packaged up as an AWS Lambda function that analyzes an entire Go (go) codebase to extract and represent its structural elements into json.

The tool should parse the code using Abstract Syntax Trees (ASTs) and generate nodes and edges in a predefined JSON format. The goal is to accurately extract various components of the codebase, including:

• Files
• Functions
• Methods
• Arguments
• Variables
• Types (Structs, Interfaces)
• Packages
• Constants

(Include any additional elements that ASTs can accurately extract and are valuable for understanding the codebase)

Strategy

1. Setup and Tooling

Utilize Go's standard library packages (go/parser, go/ast, go/token) for parsing the codebase. OR use tree-sitter

2. Parsing the Codebase

Iterate over all relevant .go files in the codebase. Parse each file into an AST for detailed analysis.

3. Extracting Code Elements

Traverse the ASTs to identify and extract the required node types. Capture details such as names, parameters, return types, variable declarations, and type definitions.

4. Building Nodes and Edges

Define nodes for each extracted element. Establish edges representing relationships between nodes (e.g., function calls, method receivers, type embeddings).

5. Use a predefined JSON structure for consistency.

6. Generating the Knowledge Graph

Compile the nodes and edges into a coherent knowledge graph. Output the graph in the specified JSON format.

7. Validation and Testing

Verify the accuracy of the extracted data. Ensure that the knowledge graph correctly represents the codebase structure.

Nodes and Edges Structure (JSON)

Node Structure

Each node will have the following structure:

{
  "node_type": "<NodeType>",
  "node_data": {
    "name": "<Name>"
  }
}

• node_type: The type of the node (e.g., "Function", "Method", "Type", "Variable", "Package").
• node_data: Contains detailed information about the node.
• name: The name of the element.

Edge Structure

Each edge will have the following structure:

{
  "edge": {
    "edge_type": "<EdgeType>"
  },
  "source": {
    "node_type": "<SourceNodeType>",
    "node_data": {
      "name": "<SourceName>"
    }
  },
  "target": {
    "node_type": "<TargetNodeType>",
    "node_data": {
      "name": "<TargetName>"
    }
  }
}

• edge.edge_type: The type of relationship (e.g., "CALLS", "IMPLEMENTS", "IMPORTS", "EMBEDS").
• source: The source node in the relationship.
• target: The target node in the relationship.

Example Nodes

Node

{
  "node_type": "Function",
  "node_data": {
    "name": "CalculateTotal",
    "package": "github.com/example/project/helpers",
    "file": "helpers/order_helpers.go",
    "line_number": 42
  }
}

Example Edges

Function- > Calls -> Function

{
  "edge": {
    "edge_type": "CALLS"
  },
  "source": {
    "node_type": "Function",
    "node_data": {
      "name": "CalculateTotal"
    }
  },
  "target": {
    "node_type": "Function",
    "node_data": {
      "name": "ApplyDiscount"
    }
  }
}

Required Output

• JSON: A single JSON output containing a nodes and edges array, structured as per the defined schemas.
• Knowledge Graph: The JSON file represents the knowledge graph of the codebase, capturing all specified elements and their relationships.
• Documentation: A README file explaining how to use the tool, the structure of the output, and any dependencies or prerequisites.

Deliverables

• [ ]Functional AWS Lambda package (with dependencies included if any)
• [ ] Can be done in python, node, ruby or a dockerised version of them.
• [ ] Knowledge Graph JSON: The output file containing the nodes and edges extracted from the codebase.
• [ ] Usage Instructions: Documentation on how to run the tool, including setup and execution step

Acceptance Criteria

• Accuracy: The tool must accurately extract and represent the specified elements and their relationships.
• Completeness: All relevant parts of the codebase are analyzed, and the knowledge graph is comprehensive.
• Performance: The tool should efficiently handle large codebases without significant performance degradation.
• Usability: The tool should be easy to set up and run, with clear instructions provided.
• Code Quality: The code should be well-organized, commented, and adhere to Go best practices

[tomsmith8]

Bounty posted: https://community.sphinx.chat/bounty/2442

[tomsmith8]

Since the codebase is in Go and requires parsing Go code, it's most efficient to use Go for the parsing logic. However, since AWS Lambda supports custom runtimes and Docker images, we can package the Go application in a Docker container for Lambda deployment.

If anyone has experience with this, let me know