feat: Add terminal Experience to parse codebase and generate semi structured metadata

[JayGhiya]

• To capture all possible details from chapi data struct in markdown with different headers under class (right now only class code is used)
• Only focus on Java Codebases
• Convert To pydantic pojo representation to get summary working.
• To work on pydantic pojos to land on Mermaid flow diagram of classes through tree summary to land on final spec of codebase. (add codebase overview based on tree summary right now) - md

Ouput:

• Md covering most important aspects per codebase from chapi json and our custom summary attributes

[JayGhiya]

Launch for users post this using instructions on how to use

[JayGhiya]

Add output summary spec based out of chapi struct and then change the summary generation code accordingly.

[JayGhiya]

cc: @vipinshreyaskumar

[JayGhiya]

At codebase level we would want to convert to a c4 component diagram. Trying this first @vipinshreyaskumar

[JayGhiya]

pydantic changes are done but there is representation recursion exception while loading into models. looks like i need to skip the list base model and iteratively load so validation happens @vipinshreyaskumar

[JayGhiya]

iteratively validating also did not help @vipinshreyaskumar need help lets have a look tomr

[JayGhiya]

Fixed - https://github.com/pydantic/pydantic/issues/2279

[JayGhiya]

Basic TUI experience added through pytermgui. Changelog:

• loguru for logging
• Improved Exception Handling
• Ability to show progress based on number of files scanned out of total files
• Successful conversion to Pydantic

Current TODO for next milestone:

• [ ] Loading the semi structured metadata into unoplat archguard modified pydantic schema.
• [ ] Summarising per node (class/interface) and adding it to summary attribute of pydantic schema

cc: @vipinshreyaskumar

[JayGhiya]

21st May Tasks:

• [x] Rename chapi json based on input codebase name.
• [x] Load the json in pydantic schema.
• [x] Integrate Litellm for per class summary and in future release allow user to choose between different providers of open source.
• [x] Need Open source 70b model for accurate metadata generation. DeepSeekv2/TogetherAi will be used to do the experiment. The provider needs to be externalised through litellm.
  • [ ] Evaluate between deepseek v2 vs llama 70b instruct
• [x] Take a open source microservice project with mvc layer to do the poc in minimum budget of deepseek v2. Spring reference project(rest + nosql)
• [x] Land on markdown format per class which accurately helps in next goal of figuring out flow diagram of codebase.

[JayGhiya]

Here is the standard for metadata per class:

Class Summary: [ClassName]

• Package: [PackageName]
• File Path: [FilePath]
• Responsibility: [Brief description of what the class does]

Fields

• [fieldName]: [FieldType]
  • Type: [Description of the field, including any dependency injections if applicable]

Methods

• methodName: [ReturnType]
  • Summary: [Brief description of what the method does]
  • Internal Calls:
  • [OtherMethodName]() to [description of what the call is for and if the call is within the same class]
  • External Calls:
  • [ExternalClassName].[ExternalMethodName]() to [description of what the call is for and if the call is to some other interface or class]

[JayGhiya]

Update for today:

1. Integrated phi3: 14b 4k instruct model for summarising better metadata based on markdown specification mentioned above. (phi3 follows instructions better than llama3 8b which is sota as of now for majority of all the tasks)
2. Designed and implemented one of the best possible prompt which should work well with most of small models <33b.
3. Started Integrating tests through pytest.