Agent Tool: Diagnosis Reporting; Prompt Generator | src/agent/tools/

[taylor-ennen]

What does this do?

This tool will take in any given text "problem". This could be code output, logs, errors. With the input text a few steps are taken to generate a unique report, as well as a small JSON artifact with the proper Key for forming relationships to the report in the conversation. The output report is small and can be stored before passed to the chat completion endpoint. The desired result of the endpoint is a single diagnostic report that is useful yet concise.

Ideal use case is to feed these reports to a given Actionable Model, such as what has been demonstrated in the Autonomous Agents circles in the past few years.

This prompt features a >99% return rate for ONLY the Diagnostic Report as the output from most models with better than average reasoning think at least ChatGPT 3.5 reasoning, however results from smaller fine-tuned models such as PHI-2 are also promising. Diagnostic quality goes up with bigger and more recent models, as would be expected, however that does not mean the smaller models do not produce something useful for say everyday tasks or education in the k-12 range of problems.

Along with the JSON artifact created, use of that artifact's Metadata can be used to track usages for monitoring resource spikes from the relevant calls and the times they were called/created.

Why do we need this?

This tool fits in with the current plans to add python functionality and allows those nodes with python or bash>python3 functionality to use this tool immediately for stable Diagnostic Reports on any error thrown. This may turn into an essential pillar for the frameworks that have been discussed by other developers on the project as well in terms of automation and maintenance of the project.

[taylor-ennen]

Usage: Feed the output code from this script DIRECTLY to the LLM.

ChatGPT 3.5: https://chat.openai.com/share/3a85f6a6-cf28-4526-89de-db1b184ea17e

## Diagnostic Issue Invoice
**Issue ID**: ISS-72af9822  
**Description**: I have a schema for an API that is not working as expected. Our sliding window is not working correctly.  
**Severity**: High  
**Date Reported**: 2024-03-15 15:32:40 EDT  

## Requirements for Diagnosis
- **Observation**: Carefully examine the issue description and any related patterns or anomalies.
- **Analysis**: Identify potential causes based on the symptoms and severity of the issue.
- **Hypothesis Formation**: Formulate possible explanations for the observed behavior.

## Expected Diagnostic Actions
- **Data Gathering**: Collect relevant data points and logs that may shed light on the issue.
- **Problem Isolation**: Narrow down the possible causes to isolate the root cause.
- **Solution Exploration**: Explore potential solutions or workarounds that can mitigate the issue.

## Desired Outcome
- **Instruction Set for Resolution**: Provide a clear and actionable set of steps that can resolve the identified issue.
- **Preventive Measures**: Suggest measures that can prevent the recurrence of this issue in the future.

## Additional Notes
- Please ensure that the diagnosis and proposed solutions are documented thoroughly for future reference.
- Consider the impact of proposed solutions on the system's overall stability and performance.
- Only respond with the diagnostic report form.
- Use keywords in the report to help other chatbots derive the actions needed to be taken.
- The report received will be read directly by a chatbot and reviewed by human observers.

---

ChatGPT 4.0: https://chat.openai.com/share/65f51cb1-fc05-4662-843a-de3a1567035f

## Diagnostic Report Form

### Issue Summary
- **Issue ID**: ISS-72af9822
- **Description**: The API schema's sliding window mechanism is not functioning as intended.
- **Severity**: High
- **Date Reported**: 2024-03-15 15:32:40 EDT

### Diagnosis
- **Observation**: The sliding window, a critical component for managing data flow or access within specified limits over time, is not operating correctly.
- **Analysis**: Possible causes include incorrect window size, improper window sliding logic (e.g., not advancing based on time or events), or data processing errors within the window.
- **Hypothesis Formation**: 
  1. Configuration issue: The window size or time frame may be incorrectly set.
  2. Logic error: The mechanism for moving the window forward could be flawed.
  3. Data handling issue: Errors in processing data within the window (e.g., incorrect aggregation, filtering, or data type mismatches).

### Expected Diagnostic Actions
- **Data Gathering**: 
  - Collect logs related to the sliding window operations.
  - Review configuration settings for the sliding window to verify correctness.
- **Problem Isolation**:
  - Test the sliding window with known data and time intervals to observe behavior.
  - Isolate changes to the API schema that might have affected the sliding window functionality.
- **Solution Exploration**:
  - Adjust configuration settings if found incorrect.
  - Debug and correct the sliding window logic.
  - Validate data processing within the window for correctness and efficiency.

### Desired Outcome
- **Instruction Set for Resolution**:
  1. Verify and correct sliding window configuration settings.
  2. Debug and fix any identified logic errors in the sliding mechanism.
  3. Ensure accurate and efficient data processing within the window.
  4. Test the corrected implementation thoroughly to confirm issue resolution.
- **Preventive Measures**:
  - Implement rigorous configuration and code review processes for changes affecting critical components like the sliding window.
  - Enhance monitoring and alerting based on sliding window performance metrics.
  - Regularly test system components to identify and rectify potential issues before they affect operations.

### Additional Notes
- Documentation of the diagnostic process and resolution steps is crucial for future reference and understanding.
- Solutions should be evaluated for their impact on system stability and performance.
- This report employs keywords such as "configuration issue," "logic error," "data handling issue," "problem isolation," and "solution exploration" to facilitate action by automated systems and review by humans.

[taylor-ennen]

@betterbrand

[DavidAJohnston]

@betterbrand did you have a chance to review this?

[betterbrand]

I did. The JSON artifact may be a good tool for agent confidence control. Still unclear as to the other features. It may need to be scaled back and merged into lite client option set.

[DavidAJohnston]

@taylor-ennen Thanks for the contribution. As @betterbrand mentioned. This can be contributed over in the lite client repo. I'm going to close the PR in this repo.