search

PromptMixerDev / prompt-mixer-app-ce

A desktop application for comparing outputs from different Large Language Models (LLMs).
https://www.promptmixer.dev
MIT License
27 stars 3 forks source link

Fixing a Failed End-to-End (e2e) Test #27

Open tomatyss opened 1 month ago

tomatyss commented 1 month ago

Feature: Run Model with Prompt to Get Output

Scenario: User runs a model with a prompt and expects the run button to be disabled during processing Given I have opened the Prompt Mixer application And I have entered "Hello world" in the input field And I have selected the "claude-2.0" model And I have set up the Anthropic AI Connector with a valid API key

When I click on the "Run" button

Then I expect the "Run" button to be disabled And I expect an output wrapper to appear And I expect to see model information for "Anthropic AI Connector, claude-2.0" And I expect to see a loading indicator And I expect the output content to be initially hidden And I expect the loading indicator to disappear after a short delay And I expect to see step buttons And I expect to see the output content

Notes:

  1. The main issue appears to be that the "Run" button remains enabled when it should be disabled during processing.
  2. The test fails at the step checking if the "Run" button is disabled, causing subsequent steps to be skipped.
  3. The task should focus on investigating why the "Run" button is not being disabled as expected when clicked.
  4. Additional checks for the appearance of output elements and loading indicators should be verified once the primary issue is resolved.

Logs:

[chrome 126.0.6478.185 mac #0-8] Running: chrome (v126.0.6478.185) on mac
[chrome 126.0.6478.185 mac #0-8] Session ID: 903a82d35439491e95c923c59406be1f
[chrome 126.0.6478.185 mac #0-8]
[chrome 126.0.6478.185 mac #0-8] » /tests/src/features/runModelWithPromptToGetOutput.feature
[chrome 126.0.6478.185 mac #0-8] Run model with prompt to get output
[chrome 126.0.6478.185 mac #0-8] Name
[chrome 126.0.6478.185 mac #0-8]    ✓ Given I have opened the application
[chrome 126.0.6478.185 mac #0-8]    ✓ And I pause for 1000ms
[chrome 126.0.6478.185 mac #0-8]    ✓ When I add "Hello world" to the inputfield "[class*='public-DraftEditor-content']"
[chrome 126.0.6478.185 mac #0-8]    ✓ And I pause for 250ms
[chrome 126.0.6478.185 mac #0-8]    ✓ Then I expect that element "[class*='public-DraftEditor-content']" contains the text "Hello world"
[chrome 126.0.6478.185 mac #0-8]    ✓ Then I expect that "1" child elements matching "[class*='PromptItemInfo_number']" exist inside container element "[class*='EditorArea_wrapper']"
[chrome 126.0.6478.185 mac #0-8]    ✓ When I click on the element "//div[text()='Choose model']"
[chrome 126.0.6478.185 mac #0-8]    ✓ Then I expect that "2" child elements matching "[class*='SelectProperty_modelGroup']" exist inside container element "[class*='SelectProperty_modelOptions']"
[chrome 126.0.6478.185 mac #0-8]    ✓ When I click on the element "//div[text()='claude-2.0']"
[chrome 126.0.6478.185 mac #0-8]    ✓ Then I expect that element "[class*='SelectProperty_selectedModel']" contains the text "claude-2.0"
[chrome 126.0.6478.185 mac #0-8]    ✓ When I click on the element "#connectors"
[chrome 126.0.6478.185 mac #0-8]    ✓ Then I expect "1" tabs named "Connectors" to be open
[chrome 126.0.6478.185 mac #0-8]    ✓ When I click on the element "#installed-tab"
[chrome 126.0.6478.185 mac #0-8]    ✓ And I click on the element "//div[text()='Anthropic AI Connector']"
[chrome 126.0.6478.185 mac #0-8]    ✓ Then I expect that element "[class*='ConnectorPage_title']" contains the text "Anthropic AI Connector"
[chrome 126.0.6478.185 mac #0-8]    ✓ When I set "ANTHROPIC_AI_API_KEY" environment variable to the inputfield "#API_KEY"
[chrome 126.0.6478.185 mac #0-8]    ✓ And I close the "Connectors" tab
[chrome 126.0.6478.185 mac #0-8]    ✓ And I click on the element "#run-button"
[chrome 126.0.6478.185 mac #0-8]    ✖ Then I expect that element "#run-button" is not enabled
[chrome 126.0.6478.185 mac #0-8]    - Then I expect that "1" child elements matching "[class*='Output_wrapper']" exist inside container element "[class*='Layout_content']"
[chrome 126.0.6478.185 mac #0-8]    - Then I expect that element "[class*='Output_modelInfo']" contains the text "Anthropic AI Connector, claude-2.0"
[chrome 126.0.6478.185 mac #0-8]    - Then I expect that element "[class*='Output_outputLoading']" does exist
[chrome 126.0.6478.185 mac #0-8]    - Then I expect that element "[class*='Output_content']" does not exist
[chrome 126.0.6478.185 mac #0-8]    - And I pause for 2000ms
[chrome 126.0.6478.185 mac #0-8]    - Then I expect that element "[class*='Output_outputLoading']" does not exist
[chrome 126.0.6478.185 mac #0-8]    - Then I expect that element "[class*='Output_stepButtons']" does exist
[chrome 126.0.6478.185 mac #0-8]    - Then I expect that element "[class*='Output_content']" does exist
[chrome 126.0.6478.185 mac #0-8]
[chrome 126.0.6478.185 mac #0-8] 18 passing (7.1s)
[chrome 126.0.6478.185 mac #0-8] 1 failing
[chrome 126.0.6478.185 mac #0-8] 8 skipped
[chrome 126.0.6478.185 mac #0-8]
[chrome 126.0.6478.185 mac #0-8] 1) Name Then I expect that element "#run-button" is not enabled
[chrome 126.0.6478.185 mac #0-8] Error: expect(received).not.toEqual(expected) // deep equality
[chrome 126.0.6478.185 mac #0-8] Error: expect(received).not.toEqual(expected) // deep equality
[chrome 126.0.6478.185 mac #0-8]
[chrome 126.0.6478.185 mac #0-8] Expected: not true
[chrome 126.0.6478.185 mac #0-8]
[chrome 126.0.6478.185 mac #0-8]     at World.default (file:///Users/ivan/projects/prompt-mixer-app-ce/tests/src/support/check/isEnabled.ts:14:37)
[chrome 126.0.6478.185 mac #0-8]     at processTicksAndRejections (node:internal/process/task_queues:105:5)
prompt-mixer commented 2 weeks ago

@tanyak1601 FYI - Still reproducible on macOS and Linux.