Prompts

[oneaiguru]

Understanding Your Task and Requirements

To ensure I fully grasp your task and address all your points, I'll summarize your requirements and key details from your messages:

1. Objective: Develop a tool that automates updating your codebase with code generated by LLMs (e.g., Claude.ai, ChatGPT with Code Interpreter), with minimal human intervention.

2. Current Challenges:

   • LLM outputs vary in format:
     • Some outputs are complete files.
     • Others are incomplete, with placeholders like "rest of methods are not changed."
     • Multiple files might be included in one code block, separated by comments with filenames.
   • Filenames or file paths are usually included in comment lines but may not always be precise.
   • Inconsistent formatting makes automation challenging.
   • You sometimes get code that isn't directly usable and needs cleaning or manual review.

3. Constraints and Considerations:

   • The tool should focus on updating files with minimal manual effort.
   • Complex features like change approvals or rollbacks are not priorities for the first version.
   • You're willing to adjust your process to simplify the tool's development (e.g., providing only code blocks without full messages).
   • Tools like ChatGPT's Code Interpreter and Claude 3.5 have been effective in generating code and understanding tasks.
   • Modularizing tasks is important due to prompt size limitations with some models.
   • You have previous experience using custom GPT instructions to automate complex tasks, including handling zip files, using SQLite databases to track tasks, and performing file operations.

4. Desired Approach:

   • Break down the task into manageable pieces.
   • In each step, know how to instruct ChatGPT and get separate working modules step by step.
   • Develop an outline of a development plan that includes prompts and code snippets to use with ChatGPT, ensuring you get working code.
   • Understand how to convert the code when it passes all tests to run it locally.
   • Use previous custom GPT instructions as inspiration for structuring the task.

5. Resources Provided:

   • Sample outputs from LLMs (numbered .py files) that include code updates.
   • Your project's code structure for context.
   • Examples of custom GPT instructions you've used in the past for complex tasks.

---

Implementation Plan

To achieve your objective with minimal human intervention, we'll develop a step-by-step plan that leverages ChatGPT with Code Interpreter and possibly Claude 3.5. The plan will focus on:

• Breaking down the task into manageable modules.
• Creating clear prompts and code snippets to guide ChatGPT in generating the required code.
• Using Test-Driven Development (TDD) to ensure reliability.
• Building a pipeline that can be adapted to run locally after validation.

---

Step 1: Define the Workflow and Requirements

Goal: Clearly outline each component of the tool and the overall workflow.

Actions:

• Input: LLM-generated code updates (code blocks), possibly in simplified format (only code blocks without extra text).
• Processing:
  • Parse the LLM outputs.
  • Extract code blocks and associate them with the correct filenames.
  • Handle incomplete code or placeholders.
  • Update existing codebase files with the new code.
• Output: Updated codebase with changes applied.

Deliverables:

• A detailed description of each module/function the tool needs.
• Define data structures (e.g., dictionaries, databases) to keep track of tasks and file mappings.

---

Step 2: Break Down the Task into Modules

Goal: Identify and define the individual modules/functions needed.

Modules:

1. File Input Module:

   • Purpose: Read LLM outputs from files (e.g., numbered .py files).
   • Functionality:
     • Read code blocks from files.
     • Handle different formats (single file, multiple files separated by comments).
     • Ignore non-code content if necessary.

2. Parsing Module:

   • Purpose: Parse the code blocks to extract filenames and corresponding code.
   • Functionality:
     • Identify file separators (e.g., comments with filenames).
     • Handle cases where filenames are incomplete or missing.
     • Manage incomplete code blocks (detect placeholders like "rest of methods are not changed").

3. Mapping Module:

   • Purpose: Map extracted code blocks to the correct files in your codebase.
   • Functionality:
     • Resolve filenames to actual file paths in your project directory.
     • Handle potential ambiguities or discrepancies in filenames.

4. Backup Module:

   • Purpose: Create backups of original files before updating.
   • Functionality:
     • Copy original files to a backup directory.
     • Ensure backups are properly versioned or timestamped.

5. Update Module:

   • Purpose: Update the codebase with the new code.
   • Functionality:
     • Overwrite existing files with new code blocks.
     • Handle partial updates intelligently (e.g., only replace certain sections).
     • Ensure code syntax is valid after updates.

6. Validation Module:

   • Purpose: Validate that the updates have been applied correctly.
   • Functionality:
     • Compare updated files with original versions.
     • Run automated tests (if available).
     • Report any issues or discrepancies.

7. Task Tracking Module:

   • Purpose: Keep track of the update tasks and their statuses.
   • Functionality:
     • Use a simple data structure (e.g., a list or a SQLite database) to record tasks.
     • Update statuses as tasks are completed.

---

Step 3: Craft Prompts for ChatGPT

Goal: Develop clear and specific prompts to guide ChatGPT in generating code for each module.

Actions:

• General Tips for Prompts:
  • Be explicit about the input and expected output.
  • Provide examples of the LLM outputs the code needs to handle.
  • Specify any constraints or edge cases.
  • Ask for code that includes comments explaining the logic.
  • Request unit tests to validate functionality (TDD approach).

Sample Prompts:

1. File Input Module Prompt:

   I need a Python function that reads code blocks from multiple `.py` files. Each file contains code generated by an LLM and may include one or more code blocks. Code blocks may be separated by comments that contain filenames or file paths. The function should:
   
   - Read all `.py` files from a specified directory.
   - Extract code blocks, along with any preceding comments that indicate the filename.
   - Return a list of tuples containing the filename and the corresponding code block.
   
   Please include error handling for cases where the filename is missing or the code block is incomplete. Also, write unit tests to verify the function works as expected with various input formats.

2. Parsing Module Prompt:

   Building upon the previous function that reads code blocks, I need to parse the extracted data to handle different cases:
   
   - If a code block includes a placeholder like "rest of methods are not changed," detect that the code is incomplete and mark it for manual review.
   - Normalize filenames from the comments to match actual file paths in the codebase.
   - Handle cases where multiple files are included in one code block.
   
   Please update the function accordingly and provide unit tests for these scenarios.

3. Backup Module Prompt:

   I need a Python function that creates backups of files before they are updated. The function should:
   
   - Accept a list of file paths to back up.
   - Copy each file to a backup directory, preserving the directory structure.
   - Include a timestamp or version number in the backup filenames to prevent overwriting previous backups.
   
   Please ensure the function handles errors gracefully, such as when a file does not exist.

4. Update Module Prompt:

   I need a Python function that updates files in the codebase with new code. The function should:
   
   - Accept a list of tuples containing file paths and new code blocks.
   - Overwrite the existing files with the new code.
   - If the code block is incomplete or marked for manual review, skip the update and log a warning.
   
   Include unit tests to verify that the function correctly updates files and handles skipped updates.

5. Validation Module Prompt:

   I need a Python function that validates the updated files. The function should:
   
   - Compare each updated file with its original backup to ensure changes were applied.
   - Optionally run any existing unit tests in the codebase to ensure no functionality is broken.
   - Generate a report summarizing the results.
   
   Please include code that runs a simple test suite and outputs the results.

6. Task Tracking Module Prompt:

   I need a simple task tracking system to monitor the status of file updates. The system should:
   
   - Use a SQLite database to record tasks, with fields for file name, status, and any error messages.
   - Update the task status as each file is processed (e.g., 'pending', 'updated', 'skipped', 'error').
   - Provide functions to add tasks, update statuses, and retrieve task summaries.
   
   Include code to initialize the database and unit tests for the task tracking functions.

---

Step 4: Use ChatGPT with Code Interpreter to Generate Code

Goal: Use the crafted prompts to have ChatGPT generate the code for each module.

Actions:

• Input: Provide the prompts and any necessary sample data to ChatGPT with Code Interpreter.
• Iteration:
  • Review the generated code for each module.
  • Test the code within the Code Interpreter environment using sample inputs.
  • Provide feedback or request adjustments if the code doesn't meet the requirements.
• Validation: Ensure that each module works correctly before moving to the next.

---

Step 5: Assemble and Integrate Modules

Goal: Combine the individual modules into a cohesive tool.

Actions:

• Integration:
  • Import the modules into a main script.
  • Define the overall workflow that orchestrates the modules (e.g., reading files, parsing, backing up, updating, validating).
• Testing:
  • Run end-to-end tests using sample LLM outputs.
  • Verify that the tool handles various scenarios, including edge cases.

---

Step 6: Adapt the Tool to Run Locally

Goal: Transfer the tool from the Code Interpreter environment to your local machine.

Actions:

• Export Code:
  • Download the generated scripts from the Code Interpreter session.
• Set Up Local Environment:
  • Ensure all necessary dependencies are installed (e.g., Python version, required libraries).
  • Adjust file paths and configurations as needed.
• Testing:
  • Run the tool on a copy of your codebase.
  • Verify that it functions correctly outside the Code Interpreter environment.

---

Step 7: Implement Error Handling and Logging

Goal: Enhance the tool's robustness by adding error handling and logging.

Actions:

• Error Handling:
  • Ensure all modules handle exceptions gracefully.
  • Provide meaningful error messages and log them appropriately.
• Logging:
  • Implement logging to track the tool's operations and any issues encountered.
  • Use Python's logging module to output logs to a file or console.

---

Step 8: Final Validation and Documentation

Goal: Ensure the tool meets all requirements and is properly documented.

Actions:

• Validation:
  • Run the tool on actual LLM outputs.
  • Confirm that it updates the codebase correctly and handles all expected cases.
• Documentation:
  • Write documentation explaining how to use the tool.
  • Include information on how to configure, run, and troubleshoot the tool.
• Code Comments:
  • Ensure all code is well-commented to facilitate future maintenance.

---

Step 9: Plan for Future Enhancements

Goal: Identify areas for improvement and plan for additional features.

Actions:

• Identify Limitations:
  • Note any limitations in the current version (e.g., handling of incomplete code blocks).
• Feature List:
  • List features to implement in future versions, such as:
  • Approval of changes before applying them.
  • Rollback mechanisms.
  • Enhanced parsing for more complex LLM outputs.
• Prioritize Features:
  • Determine which enhancements provide the most value and plan their implementation accordingly.

---

Sample Code and Instructions

Below are examples inspired by your previous custom GPT instructions, adapted to fit the current task.

1. File Input and Parsing Module:

   import os
   import re
   
   def read_and_parse_llm_outputs(directory):
      file_data = []
      for filename in os.listdir(directory):
          if filename.endswith('.py'):
              filepath = os.path.join(directory, filename)
              with open(filepath, 'r', encoding='utf-8') as file:
                  content = file.read()
   
                  # Split content into code blocks based on filename comments
                  blocks = re.split(r'#{10,}\s*(.*?)\s*#{10,}', content)
                  for i in range(1, len(blocks), 2):
                      file_comment = blocks[i].strip()
                      code_block = blocks[i + 1].strip()
                      if code_block:
                          file_data.append((file_comment, code_block))
      return file_data

2. Backup Module:

   import shutil
   import datetime
   
   def backup_files(file_paths, backup_root):
      timestamp = datetime.datetime.now().strftime('%Y%m%d%H%M%S')
      for file_path in file_paths:
          backup_path = os.path.join(backup_root, timestamp, file_path)
          os.makedirs(os.path.dirname(backup_path), exist_ok=True)
          shutil.copy2(file_path, backup_path)

3. Update Module:

   def update_files(file_updates):
      for file_path, new_code in file_updates:
          with open(file_path, 'w', encoding='utf-8') as file:
              file.write(new_code)

4. Validation Module:

   import filecmp
   
   def validate_updates(original_files, updated_files):
      mismatches = []
      for orig, updated in zip(original_files, updated_files):
          if not filecmp.cmp(orig, updated, shallow=False):
              mismatches.append(orig)
      return mismatches

---

Guidelines for Instructing ChatGPT

• Be Specific: Clearly define the input and expected output for each function.
• Provide Examples: Include snippets of the LLM outputs to illustrate the formats the code needs to handle.
• Ask for Tests: Request unit tests to validate the functionality.
• Iterate: If the generated code isn't perfect, refine your prompt or ask for adjustments.
• Use Simple Language: Avoid overly complex sentences to ensure the model understands your request.

---

Converting Code to Run Locally

• Dependencies: List all external libraries used (e.g., re, shutil) and ensure they are available in your local Python environment.
• File Paths: Adjust any hardcoded paths to relative paths or configurable settings.
• Testing: Create a test suite to verify the tool's functionality on your local machine.

---

Conclusion

By following this plan, you can systematically develop a tool that automates updating your codebase with LLM-generated code, minimizing manual intervention. Breaking the task into manageable modules allows you to leverage ChatGPT's strengths effectively, ensuring each component works correctly before integration.

Next Steps

• Start with Step 1: Clearly define your workflow and requirements.
• Proceed to Step 2: Break down the task and craft your prompts.
• Use ChatGPT: Begin generating code for each module, testing, and refining as you go.
• Reach Out for Assistance: If you need help refining prompts or troubleshooting code, feel free to ask.

Additional Considerations

• Feedback Loop: Continuously test the tool with real LLM outputs and adjust as needed.
• Community Resources: Consider sharing parts of your tool with the developer community for feedback or enhancements.
• Documentation: Keep detailed documentation throughout the process to facilitate future development and onboarding of collaborators.

I hope this detailed plan aligns with your needs and provides a clear path forward. Let me know if you'd like further assistance with any of the steps or if you have additional questions.