search

Shopify / shopify-cli

Shopify CLI helps you build against the Shopify platform faster.
https://shopify.dev/tools/cli
Other
1.01k stars 200 forks source link

Flaky tests for Node::Commands::Create and Rails::Commands::Create #1200

Closed cecyc closed 3 years ago

cecyc commented 3 years ago

Issue summary

3 tests have been identified as being flaky tests (they fail only sometimes).

Expected behavior

Test suite should be consistent, tests should pass

Actual behavior

The following tests fail sporadically:

Failure:
Rails::Commands::CreateTest#test_can_create_new_app [***/shopify-app-cli/test/project_types/rails/commands/create_test.rb:86]
Minitest::Assertion: --- expected
+++ actual
@@ -2,4 +2,5 @@
 SHOPIFY_API_SECRET=secret
 SHOP=testshop.myshopify.com
 SCOPES=write_products,write_customers,write_draft_orders
+UUID=uuid
 "

Failure:
ShopifyCli::Resources::EnvFileTest#test_update_writes_new_value_to_file [***/shopify-app-cli/lib/shopify-cli/resources/env_file.rb:84]
Minitest::Assertion: unexpected invocation: #<TestHelpers::FakeContext:0x7fc678416fb8>.write(".env", "SHOPIFY_API_KEY=foo\nSHOPIFY_API_SECRET=bar\nHOST=boo\nUUID=uuid\n")
unsatisfied expectations:
- expected exactly once, invoked never: #<TestHelpers::FakeContext:0x7fc678416fb8>.write(".env", "SHOPIFY_API_KEY=foo\nSHOPIFY_API_SECRET=bar\nHOST=boo\n")
satisfied expectations:
- allowed any number of times, invoked never: CLI::UI::Frame.open(any_parameters)
- allowed any number of times, invoked once: #<AnyInstance:CLI::UI::Spinner::SpinGroup>.add(any_parameters)
- allowed any number of times, invoked never: CLI::UI::Progress.progress(any_parameters)
- expected exactly once, invoked once: #<TestHelpers::FakeContext:0x7fc678416fb8>.print_task("writing .env file")

Failure:
Node::Commands::CreateTest#test_can_create_new_app [***/shopify-app-cli/test/project_types/node/commands/create_test.rb:152]
Minitest::Assertion: --- expected
+++ actual
@@ -2,4 +2,5 @@
 SHOPIFY_API_SECRET=secret
 SHOP=testshop.myshopify.com
 SCOPES=write_products,write_customers,write_draft_orders
+UUID=uuid
 "

Failure:
Node::Commands::CreateTest#test_can_create_new_app_registry_not_found [***/shopify-app-cli/test/project_types/node/commands/create_test.rb:207]
Minitest::Assertion: --- expected
+++ actual
@@ -2,4 +2,5 @@
 SHOPIFY_API_SECRET=secret
 SHOP=testshop.myshopify.com
 SCOPES=write_products,write_customers,write_draft_orders
+UUID=uuid
 "
904 tests, 2337 assertions, 4 failures, 0 errors, 0 skips

Steps to reproduce the problem

Run the tests suite

Reduced test case

Specifications

jacobsteves commented 3 years ago

👋 I just merged https://github.com/Shopify/shopify-app-cli/pull/1195 which I found to be the culprit, so this should be fixed now. Let me know if its not.