Add graphing capabilities to AIAssistant

vladtar commented 2 months ago

This is achieved via a new tool called GraphingTool that gets python code from ChatGPT and executes it in a lambda sandbox.

graph1 graph2

Security

REMINDER: All file contents are public.

[x] I have ensured no secure credentials or sensitive information remain in code, metadata, comments, etc.
- [x] Please verify that you double checked that .storybook/preview.js does not contain your participant access key details.
- [x] There are no temporary testing changes committed such as API base URLs, access tokens, print/log statements, etc.
[x] These changes do not introduce any security risks, or any such risks have been properly mitigated.

Describe briefly what security risks you considered, why they don't apply, or how they've been mitigated.

Checklist

Testing

[ ] MyDataHelps iOS app
[ ] MyDataHelps Android app
[x] MyDataHelps website

Documentation

[ ] I have added relevant Storybook updates to this PR as well.
[ ] If this feature requires a developer doc update, tag @CareEvolution/api-docs.

Consider "Squash and merge" as needed to keep the commit history reasonable on main.

Reviewers

Assign to the appropriate reviewer(s). Minimally, a second set of eyes is needed ensure no non-public information is published. Consider also including:

Subject-matter experts
Style/editing reviewers
Others requested by the content owner

Summary by CodeRabbit

New Features
- Introduced support for image messages in the chat, allowing users to send and receive images.
- Added a new GraphingTool to generate graphs from Python code and return them as images.
- Added new tools for querying uploaded files and retrieving presigned URLs for downloads.
- Enhanced the AIAssistant component with a new option to save graph images.
- Updated story configuration for the AIAssistant component to include new controls for image handling.
Bug Fixes
- Improved message handling logic to ensure proper rendering of different message types, including images.
Documentation
- Updated type definitions and interfaces to reflect new message types and tools.

coderabbitai[bot] commented 2 months ago

[!NOTE]

Reviews paused

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.

@coderabbitai review to trigger a single review.

Walkthrough

The changes introduce several enhancements to the AIAssistant component and related files. A new CSS rule is added for image handling, while the AIAssistant component is updated to support a new message type for images. The addToolMessage function is replaced with a more flexible addMessage function. Additionally, new tools are introduced to handle graph generation and file querying. The Chat component is also updated to handle the new "received-image" message type, allowing for image rendering within chat messages.

Changes

File	Change Summary
`src/components/container/AIAssistant/AIAssistant.css`	Added CSS rule `.mdhui-chat .image` with `height: 350px`.
`src/components/container/AIAssistant/AIAssistant.tsx`	Expanded `AIAssistantMessageType` to include `"image"`, replaced `addToolMessage` with `addMessage`, updated `ask` method for async handling, modified logic for tool messages.
`src/components/presentational/Chat/Chat.tsx`	Updated `ChatMessageType` to include `"received-image"`, modified rendering logic to handle image messages.
`src/components/container/AIAssistant/AIAssistant.stories.tsx`	Added control property `saveGraphImages` and new story `SaveGraphImages`.
`src/helpers/AIAssistant/AIAssistant.ts`	Added `GraphingTool`, `UploadedFileQueryTool`, and `GetUploadedFileTool` to `MyDataHelpsAIAssistant` class.
`src/helpers/AIAssistant/Tools.ts`	Introduced `GraphingTool`, `UploadedFileQueryTool`, and `GetUploadedFileTool` with respective functionalities.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant AIAssistant
    participant Chat
    participant GraphingTool

    User->>AIAssistant: Send message
    AIAssistant->>AIAssistant: Process message
    AIAssistant->>GraphingTool: Generate image from Python code
    GraphingTool->>AIAssistant: Return base64 image
    AIAssistant->>Chat: Add message with type "received-image"
    Chat->>User: Display image message

🪧 Tips

### Chat There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai): - Review comments: Directly reply to a review comment made by CodeRabbit. Example: - `I pushed a fix in commit , please review it.` - `Generate unit testing code for this file.` - `Open a follow-up GitHub issue for this discussion.` - Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples: - `@coderabbitai generate unit testing code for this file.` - `@coderabbitai modularize this function.` - PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples: - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.` - `@coderabbitai read src/utils.ts and generate unit testing code.` - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.` - `@coderabbitai help me debug CodeRabbit configuration file.` Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. ### CodeRabbit Commands (Invoked using PR comments) - `@coderabbitai pause` to pause the reviews on a PR. - `@coderabbitai resume` to resume the paused reviews. - `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository. - `@coderabbitai full review` to do a full review from scratch and review all the files again. - `@coderabbitai summary` to regenerate the summary of the PR. - `@coderabbitai resolve` resolve all the CodeRabbit review comments. - `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository. - `@coderabbitai help` to get help. ### Other keywords and placeholders - Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed. - Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description. - Add `@coderabbitai` or `@coderabbitai title` anywhere in the PR title to generate the title automatically. ### CodeRabbit Configuration File (`.coderabbit.yaml`) - You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository. - Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information. - If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json` ### Documentation and Community - Visit our [Documentation](https://coderabbit.ai/docs) for detailed information on how to use CodeRabbit. - Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback. - Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

vladtar commented 1 month ago

Hooked this up so that you can pass in an option to the AIAssistant to save the generated graphs to your files. Still needs a lot of testing but putting it out there for feedback.

reneefromhold commented 1 month ago

@vladtar Can I see a demo of this?

ClaysGit commented 1 month ago

RE: Garmin, we only give options in the Type field for v2 queries (Fitbit and Apple only currently). So the LLM just makes up a non-existent Type some portion of the time. Furthermore, the shape of data is not always clear. Even if you tell the LLM that Garmin has a "Daily" Type, it still doesn't know that to get step data it can query for "Daily" ddps and look at result[n].properties["steps"] for the step data.

(this is also an issue for the web view step copilot when making survey steps that load device data)

vladtar commented 1 month ago

@coderabbitai pause

vladtar commented 1 month ago

This is ready for a final look.

Refactored the event tracking into a custom callback handler
To enable saving the last graph I ended up using a langgraph in memory store to save the last graph bits

Saving a graph to your files:

save to files

Retrieving it later:

show the file

vladtar commented 1 month ago

Yes, the "Save the graph" capability is a good suggestion I think I'm going to try to tackle that next. It will be interesting when to show the suggestions, perhaps some the first time the user starts interacting, then perhaps some different ones again after certain actions such as drawing a graph.

CareEvolution / MyDataHelpsUI