Explore Highlighting Technology Based on User Feedback

[delattre1]

Options for Implementing Highlight Detection in Terminals: Pros and Cons

1. Embedding OpenCV with Swift

• Pros:
  • High performance: OpenCV is well-optimized for image processing tasks.
  • Direct control over image processing algorithms.
• Cons:
  • Complexity: Requires bridging C++ code with Swift, which can be challenging and time-consuming.
  • Maintenance: More complex integration might lead to harder maintenance and debugging.
  • https://medium.com/@hdpoorna/integrating-opencv-to-your-swift-ios-project-in-xcode-and-working-with-uiimages-4c614e62ac88

2. Embedding Python Interpreter with OpenCV

• Pros:
  • Flexibility: Python's vast ecosystem of libraries, including OpenCV, can be leveraged directly.
  • Easier to prototype and iterate due to Python's simpler syntax and dynamic nature.
• Cons:
  • Performance overhead: Running a Python interpreter embedded in another application might introduce performance bottlenecks.
  • Integration complexity: Embedding Python in another runtime environment requires careful handling of the execution environment and dependencies.
  • https://medium.com/swift2go/embedding-python-interpreter-inside-a-macos-app-and-publish-to-app-store-successfully-309be9fb96a5

3. Using Vision Framework from Apple

• Pros:
  • Native integration: Built specifically for macOS and Swift, offering seamless integration and support.
  • Performance: Optimized for Apple hardware, potentially offering better performance and power efficiency.
• Cons:
  • Limited control: While highly optimized, the Vision framework might not offer as much flexibility or control over specific image processing tasks as OpenCV.
  • Dependency on Apple's update cycle and framework capabilities.
  • https://developer.apple.com/documentation/vision/

4. FastAPI with OpenCV

• Pros:
  • Separation of concerns: By offloading the image processing to a separate API, the main application can remain cleaner and simpler.
  • Scalability: The API can be scaled independently from the main application if needed.
• Cons:
  • Network overhead: Requires network calls to an external service, which could introduce latency.
  • Deployment complexity: Requires maintaining and deploying an additional service, which could increase operational complexity.

[delattre1]

What I've Already Tried:

• Embedding C++ with OpenCV: The setup was complex, and I discontinued the effort halfway due to the challenges.
• Using Apple's Vision Framework: I attempted to identify and extract the contour of the largest area on the screen to create a mask for targeted OCR. However, I faced difficulties as the framework's documentation on these specific computer vision tasks is not very straightforward. Utilizing ChatGPT for guidance was also less effective, indicating a need for thorough documentation review and hands-on experimentation, which could entail a steep learning curve over some days.

Current Working Solution:

• Using OpenAI Vision API for Screenshot Analysis: The data extraction is functioning effectively but is slow. This approach could potentially be viable in production if the update flow is optimized. For instance, initial results could be based on simpler, locally performed OCR. Subsequent, more refined suggestions from the GPT Vision API could then be integrated as they become available, thus minimizing perceived delays for the user.

[srosro]

Prioritize exploring simultaneous queries (with OpenAI vision).

[delattre1]

Tasks:

• Review user feedback from the demo regarding highlighting technology.
• Explore and evaluate potential technologies or methods to improve highlighting.
• Implement the chosen solution and ensure it meets user expectations.

[delattre1]

It looks like the highlight region segmentation is working. Now we need to add text extraction for the regions of interest, and then integrate the code into an API

---