Open fractaloop opened 11 years ago
Can you give a dump of that error with -VV
?
DEBUG: Using configuration from /Users/fractaloop/infochimps/infochimps-homebase/.chef/knife.rb
Inventorying servers in schloss cluster, all facets, all servers
INFO: Loading cluster /Users/fractaloop/infochimps/infochimps-homebase/clusters/schloss.rb
schloss: Loading chef
schloss: Loading ec2
schloss: - loading clients
DEBUG: Signing the request as fractaloop
DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/client
schloss: - loading nodes schloss: - loading machines
DEBUG: Signing the request as fractaloop
DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/node
schloss: - loading ebs_volumes schloss: - loading roles
DEBUG: Signing the request as fractaloop
DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/role
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://api.opscode.com/organizations/infochimps_v2/search/client?q=name:schloss-*%20OR%20clientname:schloss-*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000, retrying 1/5 in 4s
schloss: - loading keypairs
schloss: - loading security_groups
<SNIPPED OUT DEBUG KEYPAIRS>
schloss: - loaded keypairs
<SNIPPED OUT DEBUG EBS VOLUMES>
schloss: - loaded ebs_volumes
<SNIPPED OUT DEBUG SECURITY GROUPS>
schloss: - loaded security_groups
DEBUG: Loaded <Node schloss-store-0 role[nfs_client], role[systemwide], role[chef_client], role[ssh], role[set_hostname], role[volumes], role[org_base], role[org_users], role[zookeeper_client], role[kafka], role[package_set], role[minidash], role[schloss_cluster], role[schloss_store]>
DEBUG: Loaded <Node schloss-master-0 role[nfs_client], role[systemwide], role[chef_client], role[ssh], role[set_hostname], role[volumes], role[org_base], role[org_users], role[zookeeper_server], role[zookeeper_client], role[storm_master], role[storm_ui], role[package_set], role[minidash], role[schloss_cluster], role[schloss_master]>
DEBUG: Signing the request as fractaloop
DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/node
schloss: - loaded roles
schloss: - loaded nodes
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://api.opscode.com/organizations/infochimps_v2/search/client?q=name:schloss-*%20OR%20clientname:schloss-*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000, retrying 2/5 in 5s
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://api.opscode.com/organizations/infochimps_v2/search/client?q=name:schloss-*%20OR%20clientname:schloss-*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000, retrying 3/5 in 14s
INFO: HTTP Request Returned 500 Internal Server Error: internal service error
ERROR: Server returned error for https://api.opscode.com/organizations/infochimps_v2/search/client?q=name:schloss-*%20OR%20clientname:schloss-*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000, retrying 4/5 in 27s
DEBUG: Loaded <Client schloss-store-0 /Users/fractaloop/infochimps/infochimps-homebase/knife/credentials/client_keys/client-schloss-store-0.pem>
DEBUG: Loaded <Client schloss-config-0 /Users/fractaloop/infochimps/infochimps-homebase/knife/credentials/client_keys/client-schloss-config-0.pem>
DEBUG: Loaded <Client schloss-master-0 /Users/fractaloop/infochimps/infochimps-homebase/knife/credentials/client_keys/client-schloss-master-0.pem>
schloss: - loaded clients
schloss: Reconciling DSL and provider information
<SNIPPED MEGAHASH>
+------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+
| Name | Chef? | State | Flavor | AZ | Env | MachineID | Public IP | Private IP | Created On | Image | Volumes | SSH Key | Startable | Launchable |
+------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+
| schloss-master-0 | yes | running | m1.large | us-east-1d | dev | i-XXXXXXXX | XX.XX.XX.XX | XX.XX.XX.XX | 2012-10-03 | ami-4d18d624 | vol-XXXXXXXX, vol-XXXXXXXX, vol-XXXXXXXX | schloss | no | no |
| schloss-store-0 | yes | running | m1.large | us-east-1d | dev | i-XXXXXXXX | XX.XX.XX.XX | XX.XX.XX.XX | 2012-10-03 | ami-4d18d624 | vol-XXXXXXXX | schloss | no | no |
| schloss-worker-0 | no | not running | m1.large | us-east-1d | dev | | | | | | | | no | yes |
+------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+
Those are errors I usually see when the chef server is slow; in the dump you showed, chef recovered after the progressive backoff. Are you sure this is actually correlated with the shutdown?
flip
On Wed, Oct 3, 2012 at 11:51 AM, Logan Lowell notifications@github.comwrote:
DEBUG: Using configuration from /Users/fractaloop/infochimps/infochimps-homebase/.chef/knife.rb Inventorying servers in schloss cluster, all facets, all servers INFO: Loading cluster /Users/fractaloop/infochimps/infochimps-homebase/clusters/schloss.rb schloss: Loading chef schloss: Loading ec2 schloss: - loading clients DEBUG: Signing the request as fractaloop DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/client schloss: - loading nodes schloss: - loading machines
DEBUG: Signing the request as fractaloop DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/node schloss: - loading ebs_volumes schloss: - loading roles
DEBUG: Signing the request as fractaloop DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/role INFO: HTTP Request Returned 500 Internal Server Error: internal service error ERROR: Server returned error for https://api.opscode.com/organizations/infochimps_v2/search/client?q=name:schloss-*%20OR%20clientname:schloss-*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000, retrying 1/5 in 4s schloss: - loading keypairs schloss: - loading security_groups
schloss: - loaded keypairs
schloss: - loaded ebs_volumes
schloss: - loaded security_groups DEBUG: Loaded DEBUG: Loaded DEBUG: Signing the request as fractaloop DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/node schloss: - loaded roles schloss: - loaded nodes INFO: HTTP Request Returned 500 Internal Server Error: internal service error ERROR: Server returned error for https://api.opscode.com/organizations/infochimps_v2/search/client?q=name:schloss-*%20OR%20clientname:schloss-*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000, retrying 2/5 in 5s INFO: HTTP Request Returned 500 Internal Server Error: internal service error ERROR: Server returned error for https://api.opscode.com/organizations/infochimps_v2/search/client?q=name:schloss-*%20OR%20clientname:schloss-*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000, retrying 3/5 in 14s INFO: HTTP Request Returned 500 Internal Server Error: internal service error ERROR: Server returned error for https://api.opscode.com/organizations/infochimps_v2/search/client?q=name:schloss-*%20OR%20clientname:schloss-*&sort=X_CHEF_id_CHEF_X%20asc&start=0&rows=1000, retrying 4/5 in 27s DEBUG: Loaded DEBUG: Loaded DEBUG: Loaded schloss: - loaded clients schloss: Reconciling DSL and provider information
+------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+ | Name | Chef? | State | Flavor | AZ | Env | MachineID | Public IP | Private IP | Created On | Image | Volumes | SSH Key | Startable | Launchable |
+------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+ | schloss-master-0 | yes | running | m1.large | us-east-1d | dev | i-XXXXXXXX | XX.XX.XX.XX | XX.XX.XX.XX | 2012-10-03 | ami-4d18d624 | vol-XXXXXXXX, vol-XXXXXXXX, vol-XXXXXXXX | schloss | no | no | | schloss-store-0 | yes | running | m1.large | us-east-1d | dev | i-XXXXXXXX | XX.XX.XX.XX | XX.XX.XX.XX | 2012-10-03 | ami-4d18d624 | vol-XXXXXXXX | schloss | no | no | | schloss-worker-0 | no | not running | m1.large | us-east-1d | dev | | | | | | | | no | yes |
+------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+
— Reply to this email directly or view it on GitHubhttps://github.com/infochimps-labs/ironfan/issues/186#issuecomment-9117658.
infochimps.com - discover data
I can repro it with every shutdown I've tried. It always resolves when the server is no longer shutting-down. It never occurs during any other knife command.
It will NOT happen if I terminate the instance via the AWS console and it's
"shutting-down", only from a knife cluster kill
It WILL happen if I have manually terminate the instance via the AWS
console, wait for it to finish shutting down, and then knife cluster kill
the server that has a Chef node and client.
Basically, it will always happen after knife deletes the node/client from Chef.
-Logan
On Thu, Oct 4, 2012 at 1:51 PM, Philip (flip) Kromer < notifications@github.com> wrote:
Those are errors I usually see when the chef server is slow; in the dump you showed, chef recovered after the progressive backoff. Are you sure this is actually correlated with the shutdown?
flip
On Wed, Oct 3, 2012 at 11:51 AM, Logan Lowell notifications@github.comwrote:
DEBUG: Using configuration from /Users/fractaloop/infochimps/infochimps-homebase/.chef/knife.rb Inventorying servers in schloss cluster, all facets, all servers INFO: Loading cluster /Users/fractaloop/infochimps/infochimps-homebase/clusters/schloss.rb schloss: Loading chef schloss: Loading ec2 schloss: - loading clients DEBUG: Signing the request as fractaloop DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/client schloss: - loading nodes schloss: - loading machines
DEBUG: Signing the request as fractaloop DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/node schloss: - loading ebs_volumes schloss: - loading roles
DEBUG: Signing the request as fractaloop DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/role INFO: HTTP Request Returned 500 Internal Server Error: internal service error ERROR: Server returned error for
retrying 1/5 in 4s schloss: - loading keypairs schloss: - loading security_groups
schloss: - loaded keypairs
schloss: - loaded ebs_volumes
schloss: - loaded security_groups DEBUG: Loaded DEBUG: Loaded DEBUG: Signing the request as fractaloop DEBUG: Sending HTTP Request via GET to api.opscode.com:443/organizations/infochimps_v2/search/node schloss: - loaded roles schloss: - loaded nodes INFO: HTTP Request Returned 500 Internal Server Error: internal service error ERROR: Server returned error for
retrying 2/5 in 5s INFO: HTTP Request Returned 500 Internal Server Error: internal service error ERROR: Server returned error for
retrying 3/5 in 14s INFO: HTTP Request Returned 500 Internal Server Error: internal service error ERROR: Server returned error for
retrying 4/5 in 27s DEBUG: Loaded DEBUG: Loaded DEBUG: Loaded schloss: - loaded clients schloss: Reconciling DSL and provider information
+------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+
| Name | Chef? | State | Flavor | AZ | Env | MachineID | Public IP | Private IP | Created On | Image | Volumes | SSH Key | Startable | Launchable |
+------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+
schloss-master-0 yes running m1.large us-east-1d dev i-XXXXXXXX XX.XX.XX.XX XX.XX.XX.XX 2012-10-03 ami-4d18d624 vol-XXXXXXXX, vol-XXXXXXXX, vol-XXXXXXXX schloss no no schloss-store-0 yes running m1.large us-east-1d dev i-XXXXXXXX XX.XX.XX.XX XX.XX.XX.XX 2012-10-03 ami-4d18d624 vol-XXXXXXXX schloss no no schloss-worker-0 no not running m1.large us-east-1d dev no yes +------------------+-------+-------------+----------+------------+-----+------------+---------------+--------------+------------+--------------+------------------------------------------+---------+-----------+------------+
— Reply to this email directly or view it on GitHub< https://github.com/infochimps-labs/ironfan/issues/186#issuecomment-9117658>.
infochimps.com - discover data
— Reply to this email directly or view it on GitHubhttps://github.com/infochimps-labs/ironfan/issues/186#issuecomment-9152611.
If an EC2 instance is in the process of shutting down when running
knife cluster show
then the Opscode server will return an error for the given query. It resolves as soon as the server transitions fromshutting-down
toterminated