search

smithmicro / jmeter-ecs

JMeter Docker Image for Distributed Testing on EC2 Container Service (ECS)
Apache License 2.0
44 stars 34 forks source link

Script cant handle more than 100 instances #25

Closed psywolf85 closed 5 years ago

psywolf85 commented 6 years ago

Hi,

First, thanks for this very cool and easy to setup lovely "script"

I planned a huge load tests ;) The script has problems with more than 99 Instances.

Error: when calling the DescribeContainerInstances operation: instanceIds can have at most 100 items.

Also there are a lot of "nonenonenonenonenonenonenonenonenonenone" in the output

Instance count is 100
Instance count is 101
time="2018-08-15T16:55:36Z" level=warning msg="Environment variable is unresolved. Setting it to a blank value..." key name=CUSTOM_PLUGIN_URL
time="2018-08-15T16:55:36Z" level=info msg="Using ECS task definition" TaskDefinition="jmeter:51"
time="2018-08-15T16:55:37Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:37Z" level=warning msg="Environment variable is unresolved. Setting it to a blank value..." key name=CUSTOM_PLUGIN_URL
time="2018-08-15T16:55:37Z" level=info msg="Using ECS task definition" TaskDefinition="jmeter:51"
time="2018-08-15T16:55:37Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:37Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:38Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:38Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:38Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:38Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:38Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:38Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:38Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
time="2018-08-15T16:55:39Z" level=info msg="Couldn't run containers" reason="RESOURCE:MEMORY"
Container instances IDs: arn:aws:ecs:eu-central-1:530324959084:container-instance/00431776-7cbb-4fb5-ba76-0161d18c0c47 arn:aws:ecs:eu-central-1:530324959084:container-instance/0240eb1b-684b-41c4-9584-48ddd66ca3ba arn:aws:ecs:eu-central-1:530324959084:container-instance/0a93f309-ffa9-4d7d-8d75-b285c623b023 arn:aws:ecs:eu-central-1:530324959084:container-instance/0aa603ae-ddf5-4459-b8d6-60a44fa54965 arn:aws:ecs:eu-central-1:530324959084:container-instance/0b070e27-557b-4c24-9377-e178d7f75be2 arn:aws:ecs:eu-central-1:530324959084:container-instance/0bc56275-867f-432c-a747-71187b13c592 arn:aws:ecs:eu-central-1:530324959084:container-instance/0f293991-a548-4832-88a1-2bcd78fbdb66 arn:aws:ecs:eu-central-1:530324959084:container-instance/13a88b31-269c-44e6-91a7-d683d830e214 arn:aws:ecs:eu-central-1:530324959084:container-instance/16aafbe6-ba03-4b96-9ed4-e647036eb405 arn:aws:ecs:eu-central-1:530324959084:container-instance/16e20247-bc51-4b4c-bb4e-86bb4a95cb7d arn:aws:ecs:eu-central-1:530324959084:container-instance/177ee7f1-6f3e-4c9a-9b16-0aefc942fc7b arn:aws:ecs:eu-central-1:530324959084:container-instance/18e590a0-eefa-4b29-ab73-4c9a6ab67b46 arn:aws:ecs:eu-central-1:530324959084:container-instance/1a89d5d0-8023-43e1-ac81-c3505c41ff2d arn:aws:ecs:eu-central-1:530324959084:container-instance/1f8dfef5-e263-41fe-8552-225722228676 arn:aws:ecs:eu-central-1:530324959084:container-instance/21bad63f-e2d9-4b99-961b-e13f35f57b76 arn:aws:ecs:eu-central-1:530324959084:container-instance/23a709b2-b896-426d-a2c4-00cbb0cf2f5f arn:aws:ecs:eu-central-1:530324959084:container-instance/25ed026e-ae2e-4a66-9bea-bf8b4e979b46 arn:aws:ecs:eu-central-1:530324959084:container-instance/299b9bd0-c5c2-4821-a047-c29cf52e1314 arn:aws:ecs:eu-central-1:530324959084:container-instance/2c4e17a3-104c-4b16-9909-68131950010c arn:aws:ecs:eu-central-1:530324959084:container-instance/2cebaaf3-0334-4f87-bb58-61778800a1a4 arn:aws:ecs:eu-central-1:530324959084:container-instance/305354a7-c378-440c-9d9e-9cbcd5c71089 arn:aws:ecs:eu-central-1:530324959084:container-instance/3625bd5a-1d11-4211-8ffd-a0e3ae1c03f5 arn:aws:ecs:eu-central-1:530324959084:container-instance/3a97b1f1-9d0b-4e53-827a-97809d753838 arn:aws:ecs:eu-central-1:530324959084:container-instance/3ebd4896-83ab-4672-b141-7f7dd925f655 arn:aws:ecs:eu-central-1:530324959084:container-instance/3f4b0539-32f6-49d4-aa82-c6ad38e47be4 arn:aws:ecs:eu-central-1:530324959084:container-instance/47c88693-19ba-43e1-8ac9-c2f8ba32cd15 arn:aws:ecs:eu-central-1:530324959084:container-instance/484090fe-ce66-4087-82b4-694b5df5c9ca arn:aws:ecs:eu-central-1:530324959084:container-instance/494c1220-0fe7-46e4-896f-41732140c866 arn:aws:ecs:eu-central-1:530324959084:container-instance/4b9a341b-0188-4b21-abec-67f69fa3650f arn:aws:ecs:eu-central-1:530324959084:container-instance/4faeee65-8a5d-43e5-936a-8dfcf1dc47e9 arn:aws:ecs:eu-central-1:530324959084:container-instance/51fbfb63-23fc-48a0-951d-023511f38322 arn:aws:ecs:eu-central-1:530324959084:container-instance/55a00536-0639-4647-a556-2d65444bbee2 arn:aws:ecs:eu-central-1:530324959084:container-instance/55cf9541-a317-4a1c-beb1-d656fe40a644 arn:aws:ecs:eu-central-1:530324959084:container-instance/59961526-097f-467e-bd98-5ac8a145b7dc arn:aws:ecs:eu-central-1:530324959084:container-instance/5cbd6a2e-9ae9-4ab1-969c-7d7f65019eb0 arn:aws:ecs:eu-central-1:530324959084:container-instance/5dbe1e46-a20c-4872-9d2f-ca4e3764cdea arn:aws:ecs:eu-central-1:530324959084:container-instance/60c31908-eeec-4be9-8dd1-979ec84b269e arn:aws:ecs:eu-central-1:530324959084:container-instance/616c6336-d1d2-4938-aa34-30d9372b37dd arn:aws:ecs:eu-central-1:530324959084:container-instance/6202af95-0a9f-48ce-a6da-822ab1b9db2e arn:aws:ecs:eu-central-1:530324959084:container-instance/670a96e1-3038-4ae1-b5f8-895328e8535e arn:aws:ecs:eu-central-1:530324959084:container-instance/672cf831-2533-47f7-894a-5cff336de2ea arn:aws:ecs:eu-central-1:530324959084:container-instance/6a7cd8a9-43fa-417d-ac7b-3a6f548ae02d arn:aws:ecs:eu-central-1:530324959084:container-instance/72e0c16f-0825-4741-bd5a-941d6a13f101 arn:aws:ecs:eu-central-1:530324959084:container-instance/737a5bf9-75b4-44c6-b57c-761b2555dafa arn:aws:ecs:eu-central-1:530324959084:container-instance/73f5d35f-2eaa-4ab0-9949-053d309a4548 arn:aws:ecs:eu-central-1:530324959084:container-instance/741bcb49-27a3-4293-823f-20a2e709227e arn:aws:ecs:eu-central-1:530324959084:container-instance/7468f119-e33e-47b0-bdc2-ec5f2194b204 arn:aws:ecs:eu-central-1:530324959084:container-instance/7526a671-1de7-4d5e-a386-6fbc3c6d4212 arn:aws:ecs:eu-central-1:530324959084:container-instance/77ff09fd-3bb6-4b88-8284-df50c17805b0 arn:aws:ecs:eu-central-1:530324959084:container-instance/78f460c1-0f71-4cdd-8b1f-612e131f530e arn:aws:ecs:eu-central-1:530324959084:container-instance/793f9b35-cb1c-49e3-ad57-ac104c291c9b arn:aws:ecs:eu-central-1:530324959084:container-instance/803262b7-a2fd-416e-b362-24e0c3000800 arn:aws:ecs:eu-central-1:530324959084:container-instance/816e7eb7-08e4-4aec-bc75-4372d04968cb arn:aws:ecs:eu-central-1:530324959084:container-instance/81ffa859-9f23-45da-a2e4-5f621189313f arn:aws:ecs:eu-central-1:530324959084:container-instance/83f64305-1c62-4756-9307-8054f1916140 arn:aws:ecs:eu-central-1:530324959084:container-instance/856124fc-4f6f-4a53-a15e-caee0d459d74 arn:aws:ecs:eu-central-1:530324959084:container-instance/8829d4c7-bcda-4c6d-9cc7-75f0219f0b7a arn:aws:ecs:eu-central-1:530324959084:container-instance/89b91e05-3e76-4d97-825c-2fb34858116e arn:aws:ecs:eu-central-1:530324959084:container-instance/8a29f26f-8d04-4ff4-b5cc-c85023dee586 arn:aws:ecs:eu-central-1:530324959084:container-instance/8e758800-01f4-4fb4-a960-33b7d9d60c73 arn:aws:ecs:eu-central-1:530324959084:container-instance/91a4f691-bca8-47d7-b536-61c08ceb6935 arn:aws:ecs:eu-central-1:530324959084:container-instance/92f753ab-115c-403e-8bbc-71c66ce0a317 arn:aws:ecs:eu-central-1:530324959084:container-instance/9449429d-0b47-4a8f-b669-b39e4acb059d arn:aws:ecs:eu-central-1:530324959084:container-instance/954e305c-ddc0-4c29-9d76-d0144f92ec5d arn:aws:ecs:eu-central-1:530324959084:container-instance/9c90f545-5157-43c9-aac1-d048afa6059d arn:aws:ecs:eu-central-1:530324959084:container-instance/9d45932b-6f8a-4eb6-bd36-2efc54562bfd arn:aws:ecs:eu-central-1:530324959084:container-instance/a171e178-2332-49ba-aa45-d191b1369e2d arn:aws:ecs:eu-central-1:530324959084:container-instance/a24ebe59-488f-4341-86ef-6edc4099e6f9 arn:aws:ecs:eu-central-1:530324959084:container-instance/a2549a60-6a02-40d5-83ba-87c15f976c88 arn:aws:ecs:eu-central-1:530324959084:container-instance/a31e4bdd-c300-4789-90d5-d5afcf1eca4f arn:aws:ecs:eu-central-1:530324959084:container-instance/a33a3b8f-f75c-4bac-9356-69c7c5b0942e arn:aws:ecs:eu-central-1:530324959084:container-instance/a3f99022-9863-4fac-97bc-aea4005007de arn:aws:ecs:eu-central-1:530324959084:container-instance/a4b1eb80-0d08-4ea4-a27a-225fbc613588 arn:aws:ecs:eu-central-1:530324959084:container-instance/a5173b2a-aff5-4ed7-8192-8ae32e92cb0d arn:aws:ecs:eu-central-1:530324959084:container-instance/a544d67f-3d9d-4a51-a881-3ffa662430db arn:aws:ecs:eu-central-1:530324959084:container-instance/ad129594-6f92-4575-8c37-88ee291b5843 arn:aws:ecs:eu-central-1:530324959084:container-instance/adf8c45f-30f7-4502-9d06-77bca222b691 arn:aws:ecs:eu-central-1:530324959084:container-instance/af1e0eb0-77f4-4d7b-8e9a-950bc766c7aa arn:aws:ecs:eu-central-1:530324959084:container-instance/b0a1a999-c17a-4282-8681-f1c57422f6f7 arn:aws:ecs:eu-central-1:530324959084:container-instance/b0be02f8-f361-46f2-8238-b7c4ddf30da6 arn:aws:ecs:eu-central-1:530324959084:container-instance/b30c1641-6702-4efa-b1fa-714041dab21f arn:aws:ecs:eu-central-1:530324959084:container-instance/b3d29de6-3f92-4d79-afac-fd84ab50a0c7 arn:aws:ecs:eu-central-1:530324959084:container-instance/bb4b4d00-1de7-4f75-9246-300d06180b05 arn:aws:ecs:eu-central-1:530324959084:container-instance/bb68d9df-7ece-4153-b68f-b1a079451b12 arn:aws:ecs:eu-central-1:530324959084:container-instance/c12137c7-380a-447a-8ff8-85a9887ab1e5 arn:aws:ecs:eu-central-1:530324959084:container-instance/c1ee77d3-3869-41d8-86f6-23a73506d1c8 arn:aws:ecs:eu-central-1:530324959084:container-instance/c23061b0-5f88-4e98-a3f7-4455d3cfb227 arn:aws:ecs:eu-central-1:530324959084:container-instance/c2efb177-ce70-407f-81ad-3617031a02fa arn:aws:ecs:eu-central-1:530324959084:container-instance/c6815d23-756f-4d59-8c14-abc9cd3ce75c arn:aws:ecs:eu-central-1:530324959084:container-instance/caaf37ab-14a4-4c29-8d09-225622be4726 arn:aws:ecs:eu-central-1:530324959084:container-instance/db22345a-72ea-4bc1-aea6-dd4d03aaaf9a arn:aws:ecs:eu-central-1:530324959084:container-instance/dc81bd74-2178-4cee-bd66-3417acb05608 arn:aws:ecs:eu-central-1:530324959084:container-instance/e06f51a2-6a0b-434e-b527-7555b08d7151 arn:aws:ecs:eu-central-1:530324959084:container-instance/eddce58a-4b37-4edb-adb9-05af682b5494 arn:aws:ecs:eu-central-1:530324959084:container-instance/f237fa8f-f71b-4f01-b545-e8a03833b506 arn:aws:ecs:eu-central-1:530324959084:container-instance/f4212a71-b17e-4399-86a2-694ea62a36cf arn:aws:ecs:eu-central-1:530324959084:container-instance/f68cc5c0-0ce6-44ca-b9f2-3bbc0ef24c15 arn:aws:ecs:eu-central-1:530324959084:container-instance/f898ae4e-a85d-487a-8825-06dd2d47f4ed arn:aws:ecs:eu-central-1:530324959084:container-instance/fc62bffe-6f6f-4efd-b2d6-cfc07fbba2e7 arn:aws:ecs:eu-central-1:530324959084:container-instance/fd433ed7-29d9-49e4-a22f-a96fe489be75 arn:aws:ecs:eu-central-1:530324959084:container-instance/fdac2474-b374-4e66-9e98-3038f38868ad 

An error occurred (InvalidParameterException) when calling the DescribeContainerInstances operation: instanceIds can have at most 100 items.
Gru instance ID: 

An error occurred (InvalidParameterException) when calling the DescribeContainerInstances operation: instanceIds can have at most 100 items.
Minion instances IDs: 
Gru at 18.184.165.179NoneNoneNoneNoneNoneNoneNoneNoneNoneNone18.184.77.12054.93.62.18518.185.124.1118.196.88.18618.197.123.4118.184.3.22454.93.67.24218.185.48.5818.184.204.17654.93.231.3918.195.35.12752.59.203.118.195.61.4554.93.253.1683.120.32.13454.93.234.14752.59.193.3818.196.249.2954.93.67.8252.57.237.20852.59.113.6552.57.133.24518.195.241.12035.156.245.21018.197.143.1393.120.116.14318.184.117.20054.93.51.018.184.154.5018.184.6.9018.184.222.19518.194.68.952.58.107.19535.159.53.22718.185.54.10818.197.199.2173.120.32.1133.120.37.2918.184.14.12152.59.194.216NoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNoneNone18.184.217.17818.185.138.11218.196.150.17518.185.83.9218.185.123.16935.159.51.22518.194.45.3052.28.201.13554.93.251.7218.185.120.2543.120.38.4418.184.22.10418.197.104.18852.58.123.713.120.116.22254.93.249.13835.158.95.6852.58.219.15552.58.85.2418.185.18.8918.185.8.7035.158.118.17018.196.176.11118.185.21.24852.59.190.14752.59.162.1903.120.116.21235.158.119.11018.184.77.13635.158.222.10252.59.231.8518.185.35.23254.93.191.13518.194.219.6054.93.214.2754.93.251.6735.158.239.4618.194.252.718.194.83.3054.93.115.18618.184.222.352.28.36.10254.93.212.19718.194.223.4418.185.102.8518.185.120.7418.197.57.4454.93.195.9435.159.18.18452.59.208.1818.185.125.19618.196.205.9918.185.125.21918.184.13.18318.195.174.254.93.173.21754.93.114.7418.194.42.13818.184.112.23535.158.118.143NoneNoneNoneNoneNoneNoneNoneNoneNoneNone
Minions at 10.74.2.228,None,None,None,None,None,None,None,None,None,None,10.74.2.90,10.74.2.58,10.74.2.122,10.74.2.155,10.74.2.57,10.74.2.111,10.74.2.47,10.74.2.124,10.74.2.198,10.74.2.180,10.74.2.26,10.74.2.235,10.74.2.175,10.74.2.12,10.74.2.61,10.74.2.141,10.74.2.98,10.74.2.211,10.74.2.177,10.74.2.161,10.74.1.106,10.74.1.57,10.74.1.28,10.74.1.93,10.74.1.114,10.74.1.209,10.74.1.116,10.74.1.197,10.74.1.53,10.74.1.149,10.74.2.136,10.74.2.184,10.74.2.217,10.74.2.63,10.74.2.79,10.74.2.147,10.74.2.81,10.74.2.166,10.74.2.36,10.74.2.181,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,None,10.74.1.158,10.74.1.131,10.74.1.128,10.74.1.224,10.74.1.134,10.74.1.87,10.74.1.85,10.74.1.181,10.74.1.72,10.74.1.216,10.74.1.154,10.74.1.56,10.74.1.200,10.74.1.185,10.74.1.79,10.74.1.76,10.74.1.109,10.74.1.179,10.74.1.215,10.74.1.55,10.74.1.26,10.74.1.90,10.74.1.235,10.74.1.41,10.74.1.254,10.74.1.77,10.74.1.162,10.74.1.230,10.74.1.135,10.74.1.36,10.74.2.95,10.74.2.156,10.74.2.240,10.74.2.70,10.74.2.101,10.74.2.27,10.74.2.43,10.74.2.238,10.74.2.142,10.74.2.46,10.74.1.169,10.74.1.31,10.74.1.143,10.74.1.45,10.74.1.237,10.74.1.178,10.74.1.99,10.74.1.151,10.74.1.119,10.74.1.213,10.74.1.201,10.74.1.140,10.74.1.98,10.74.1.211,10.74.1.19,10.74.1.144,10.74.1.65,10.74.1.38,10.74.1.246,10.74.1.150,None,None,None,None,None,None,None,None,None,None,
Copying /plans/example.jmx to Gru
ssh: Could not resolve hostname 18.184.165.179nonenonenonenonenonenonenonenonenonenone18.184.77.12054.93.62.18518.185.124.1118.196.8: Name does not resolve
dsperling commented 6 years ago

DescribeContainerInstances indeed has a limit of 100. Thanks for finding this. https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_DescribeContainerInstances.html

A short term fix is to error earlier if MINION_COUNT > 99. I am not sure what a long term fix could be.

psywolf85 commented 6 years ago

This fixed it for me: https://github.com/smithmicro/jmeter-ecs/pull/26

With the new image, a newer cli version is used where the bug is fixed.