search

HariSekhon / DevOps-Python-tools

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.
https://www.linkedin.com/in/HariSekhon
MIT License
772 stars 340 forks source link

ambari_trigger_service_checks.py does not wait when used on all services #8

Closed maciex closed 4 years ago

maciex commented 5 years ago

Command

$ docker run harisekhon/pytools ambari_trigger_service_checks.py -H bf-hdp-1.mydomain.com -u admin -p admin --all --wait --debug

Expected result

I expected that service check will be triggered on all services and the command will wait till all finished.

Actual result

Command finished after the first service check was finished

Debug log

$ docker run harisekhon/pytools ambari_trigger_service_checks.py -H bf-hdp-1.mydomain.com -u admin -p admin --all --wait --debug
2019-10-18 12:06:25,103 - cli.py[__parse_verbose__:372](1) - DEBUG - enabling debug logging
2019-10-18 12:06:25,104 - cli.py[verbose:297](1) - DEBUG - setting verbose to 3
2019-10-18 12:06:25,105 - cli.py[__parse_timeout__:387](1) - DEBUG - getting $TIMEOUT value None
2019-10-18 12:06:25,105 - cli.py[__parse_timeout__:397](1) - DEBUG - timeout not set, using default timeout 7200
2019-10-18 12:06:25,105 - utils.py[log_option:2233](1) - INFO  - timeout:  7200
2019-10-18 12:06:25,105 - cli.py[timeout:254](1) - DEBUG - setting timeout to 7200 secs
2019-10-18 12:06:25,106 - cli.py[main:159](1) - INFO  - Hari Sekhon ambari_trigger_service_checks.py version 0.2.1 =>  CLI version 0.9.3 =>  Utils version 0.12.4
2019-10-18 12:06:25,106 - cli.py[main:160](1) - INFO  - https://github.com/harisekhon/devops-python-tools
2019-10-18 12:06:25,106 - cli.py[main:161](1) - INFO  - verbose level: 3 (DEBUG)
2019-10-18 12:06:25,106 - utils.py[log_option:2233](1) - INFO  - timeout:  7200
2019-10-18 12:06:25,106 - cli.py[main:164](1) - DEBUG - setting timeout alarm (7200)
2019-10-18 12:06:25,106 - utils.py[log_option:2233](1) - INFO  - host:  bf-hdp-1.mydomain.com
2019-10-18 12:06:25,106 - utils.py[log_option:2233](1) - INFO  - port:  8080
2019-10-18 12:06:25,107 - utils.py[log_option:2233](1) - INFO  - user:  admin
2019-10-18 12:06:25,108 - utils.py[log_option:2233](1) - INFO  - password:  <omitted>
2019-10-18 12:06:25,108 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:25,108 - ambari_trigger_service_checks.py[req:184](1) - DEBUG - GET http://bf-hdp-1.mydomain.com:8080/api/v1/clusters
2019-10-18 12:06:25,267 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Content-Encoding:gzip
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node0161orqznhpk47gfjyfwc8eo6b236.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Vary:Accept-Encoding, User-Agent
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:25 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:25,267 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 200
2019-10-18 12:06:25,267 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters",
  "items" : [
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster",
      "Clusters" : {
        "cluster_name" : "cluster"
      }
    }
  ]
}
2019-10-18 12:06:25,268 - ambari_trigger_service_checks.py[run:142](1) - INFO  - no --cluster specified, but only one cluster managed by Ambari, inferring --cluster='cluster'
2019-10-18 12:06:25,268 - utils.py[log_option:2233](1) - INFO  - cluster:  cluster
2019-10-18 12:06:25,268 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:25,268 - ambari_trigger_service_checks.py[req:184](1) - DEBUG - GET http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services
2019-10-18 12:06:25,456 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Content-Encoding:gzip
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node01q6l4v5zzi0cjdjsiepci6ebs237.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Vary:Accept-Encoding, User-Agent
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:25 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:25,456 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 200
2019-10-18 12:06:25,456 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services",
  "items" : [
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/AMBARI_METRICS",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "AMBARI_METRICS"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/HBASE",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "HBASE"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/HDFS",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "HDFS"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/HIVE",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "HIVE"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/KAFKA",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "KAFKA"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/KNOX",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "KNOX"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/MAPREDUCE2",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "MAPREDUCE2"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/SPARK2",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "SPARK2"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/SQOOP",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "SQOOP"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/SUPERSET",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "SUPERSET"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/TEZ",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "TEZ"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/YARN",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "YARN"
      }
    },
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/services/ZOOKEEPER",
      "ServiceInfo" : {
        "cluster_name" : "cluster",
        "service_name" : "ZOOKEEPER"
      }
    }
  ]
}
2019-10-18 12:06:25,457 - ambari_trigger_service_checks.py[request_service_checks:278](1) - DEBUG - requesting service checks for services: [u'AMBARI_METRICS', u'HBASE', u'HDFS', u'HIVE', u'KAFKA', u'KNOX', u'MAPREDUCE2', u'SPARK2', u'SQOOP', u'SUPERSET', u'TEZ', u'YARN', u'ZOOKEEPER']
2019-10-18 12:06:25,457 - ambari_trigger_service_checks.py[gen_payload:332](1) - DEBUG - generating payload for services: [u'AMBARI_METRICS', u'HBASE', u'HDFS', u'HIVE', u'KAFKA', u'KNOX', u'MAPREDUCE2', u'SPARK2', u'SQOOP', u'SUPERSET', u'TEZ', u'YARN', u'ZOOKEEPER']
2019-10-18 12:06:25,459 - ambari_trigger_service_checks.py[gen_payload:389](1) - DEBUG - generated payload:
[
    {
        "RequestSchedule": {
            "batch": [
                {
                    "requests": [
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "AMBARI_METRICS_SERVICE_CHECK",
                                    "context": "AMBARI_METRICS Service Check (batch 1 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "AMBARI_METRICS"
                                    }
                                ]
                            },
                            "order_id": 1,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "HBASE_SERVICE_CHECK",
                                    "context": "HBASE Service Check (batch 2 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "HBASE"
                                    }
                                ]
                            },
                            "order_id": 2,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "HDFS_SERVICE_CHECK",
                                    "context": "HDFS Service Check (batch 3 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "HDFS"
                                    }
                                ]
                            },
                            "order_id": 3,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "HIVE_SERVICE_CHECK",
                                    "context": "HIVE Service Check (batch 4 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "HIVE"
                                    }
                                ]
                            },
                            "order_id": 4,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "KAFKA_SERVICE_CHECK",
                                    "context": "KAFKA Service Check (batch 5 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "KAFKA"
                                    }
                                ]
                            },
                            "order_id": 5,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "KNOX_SERVICE_CHECK",
                                    "context": "KNOX Service Check (batch 6 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "KNOX"
                                    }
                                ]
                            },
                            "order_id": 6,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "MAPREDUCE2_SERVICE_CHECK",
                                    "context": "MAPREDUCE2 Service Check (batch 7 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "MAPREDUCE2"
                                    }
                                ]
                            },
                            "order_id": 7,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "SPARK2_SERVICE_CHECK",
                                    "context": "SPARK2 Service Check (batch 8 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "SPARK2"
                                    }
                                ]
                            },
                            "order_id": 8,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "SQOOP_SERVICE_CHECK",
                                    "context": "SQOOP Service Check (batch 9 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "SQOOP"
                                    }
                                ]
                            },
                            "order_id": 9,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "SUPERSET_SERVICE_CHECK",
                                    "context": "SUPERSET Service Check (batch 10 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "SUPERSET"
                                    }
                                ]
                            },
                            "order_id": 10,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "TEZ_SERVICE_CHECK",
                                    "context": "TEZ Service Check (batch 11 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "TEZ"
                                    }
                                ]
                            },
                            "order_id": 11,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "YARN_SERVICE_CHECK",
                                    "context": "YARN Service Check (batch 12 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "YARN"
                                    }
                                ]
                            },
                            "order_id": 12,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        },
                        {
                            "RequestBodyInfo": {
                                "RequestInfo": {
                                    "command": "ZOOKEEPER_QUORUM_SERVICE_CHECK",
                                    "context": "ZOOKEEPER Service Check (batch 13 of 13)"
                                },
                                "Requests/resource_filters": [
                                    {
                                        "service_name": "ZOOKEEPER"
                                    }
                                ]
                            },
                            "order_id": 13,
                            "type": "POST",
                            "uri": "/api/v1/clusters/cluster/requests"
                        }
                    ]
                },
                {
                    "batch_settings": {
                        "batch_separation_in_seconds": 1,
                        "task_failure_tolerance": 1
                    }
                }
            ]
        }
    }
]
2019-10-18 12:06:25,459 - ambari_trigger_service_checks.py[request_service_checks:283](1) - INFO  - sending batch schedule check request for services: AMBARI_METRICS, HBASE, HDFS, HIVE, KAFKA, KNOX, MAPREDUCE2, SPARK2, SQOOP, SUPERSET, TEZ, YARN, ZOOKEEPER
2019-10-18 12:06:25,459 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:25,459 - ambari_trigger_service_checks.py[req:180](1) - DEBUG - POST http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules
2019-10-18 12:06:25,459 - ambari_trigger_service_checks.py[req:181](1) - DEBUG - POSTing data:

[{"RequestSchedule": {"batch": [{"requests": [{"order_id": 1, "RequestBodyInfo": {"RequestInfo": {"command": "AMBARI_METRICS_SERVICE_CHECK", "context": "AMBARI_METRICS Service Check (batch 1 of 13)"}, "Requests/resource_filters": [{"service_name": "AMBARI_METRICS"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 2, "RequestBodyInfo": {"RequestInfo": {"command": "HBASE_SERVICE_CHECK", "context": "HBASE Service Check (batch 2 of 13)"}, "Requests/resource_filters": [{"service_name": "HBASE"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 3, "RequestBodyInfo": {"RequestInfo": {"command": "HDFS_SERVICE_CHECK", "context": "HDFS Service Check (batch 3 of 13)"}, "Requests/resource_filters": [{"service_name": "HDFS"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 4, "RequestBodyInfo": {"RequestInfo": {"command": "HIVE_SERVICE_CHECK", "context": "HIVE Service Check (batch 4 of 13)"}, "Requests/resource_filters": [{"service_name": "HIVE"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 5, "RequestBodyInfo": {"RequestInfo": {"command": "KAFKA_SERVICE_CHECK", "context": "KAFKA Service Check (batch 5 of 13)"}, "Requests/resource_filters": [{"service_name": "KAFKA"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 6, "RequestBodyInfo": {"RequestInfo": {"command": "KNOX_SERVICE_CHECK", "context": "KNOX Service Check (batch 6 of 13)"}, "Requests/resource_filters": [{"service_name": "KNOX"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 7, "RequestBodyInfo": {"RequestInfo": {"command": "MAPREDUCE2_SERVICE_CHECK", "context": "MAPREDUCE2 Service Check (batch 7 of 13)"}, "Requests/resource_filters": [{"service_name": "MAPREDUCE2"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 8, "RequestBodyInfo": {"RequestInfo": {"command": "SPARK2_SERVICE_CHECK", "context": "SPARK2 Service Check (batch 8 of 13)"}, "Requests/resource_filters": [{"service_name": "SPARK2"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 9, "RequestBodyInfo": {"RequestInfo": {"command": "SQOOP_SERVICE_CHECK", "context": "SQOOP Service Check (batch 9 of 13)"}, "Requests/resource_filters": [{"service_name": "SQOOP"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 10, "RequestBodyInfo": {"RequestInfo": {"command": "SUPERSET_SERVICE_CHECK", "context": "SUPERSET Service Check (batch 10 of 13)"}, "Requests/resource_filters": [{"service_name": "SUPERSET"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 11, "RequestBodyInfo": {"RequestInfo": {"command": "TEZ_SERVICE_CHECK", "context": "TEZ Service Check (batch 11 of 13)"}, "Requests/resource_filters": [{"service_name": "TEZ"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 12, "RequestBodyInfo": {"RequestInfo": {"command": "YARN_SERVICE_CHECK", "context": "YARN Service Check (batch 12 of 13)"}, "Requests/resource_filters": [{"service_name": "YARN"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}, {"order_id": 13, "RequestBodyInfo": {"RequestInfo": {"command": "ZOOKEEPER_QUORUM_SERVICE_CHECK", "context": "ZOOKEEPER Service Check (batch 13 of 13)"}, "Requests/resource_filters": [{"service_name": "ZOOKEEPER"}]}, "type": "POST", "uri": "/api/v1/clusters/cluster/requests"}]}, {"batch_settings": {"task_failure_tolerance": 1, "batch_separation_in_seconds": 1}}]}}]
2019-10-18 12:06:26,178 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node01r5hk4u9fvaypyrd530kl3h37238.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:25 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:26,178 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 201
2019-10-18 12:06:26,179 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "resources" : [
    {
      "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7",
      "RequestSchedule" : {
        "id" : 7
      }
    }
  ]
}
2019-10-18 12:06:26,179 - ambari_trigger_service_checks.py[request_service_checks:288](1) - INFO  - RequestSchedule 7 submitted
2019-10-18 12:06:26,179 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:26,179 - ambari_trigger_service_checks.py[req:184](1) - DEBUG - GET http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7
2019-10-18 12:06:26,346 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Content-Encoding:gzip
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node0omygpf8xseemqnjujpxx366j240.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Vary:Accept-Encoding, User-Agent
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:26 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:26,346 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 200
2019-10-18 12:06:26,346 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7",
  "RequestSchedule" : {
    "authenticated_user" : 1,
    "batch" : {
      "batch_requests" : [
        {
          "order_id" : 1,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"AMBARI_METRICS_SERVICE_CHECK\",\"context\":\"AMBARI_METRICS Service Check (batch 1 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"AMBARI_METRICS\"}]}",
          "request_status" : "Accepted",
          "return_code" : 202
        },
        {
          "order_id" : 2,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HBASE_SERVICE_CHECK\",\"context\":\"HBASE Service Check (batch 2 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HBASE\"}]}"
        },
        {
          "order_id" : 3,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HDFS_SERVICE_CHECK\",\"context\":\"HDFS Service Check (batch 3 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HDFS\"}]}"
        },
        {
          "order_id" : 4,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HIVE_SERVICE_CHECK\",\"context\":\"HIVE Service Check (batch 4 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HIVE\"}]}"
        },
        {
          "order_id" : 5,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KAFKA_SERVICE_CHECK\",\"context\":\"KAFKA Service Check (batch 5 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KAFKA\"}]}"
        },
        {
          "order_id" : 6,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KNOX_SERVICE_CHECK\",\"context\":\"KNOX Service Check (batch 6 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KNOX\"}]}"
        },
        {
          "order_id" : 7,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"MAPREDUCE2_SERVICE_CHECK\",\"context\":\"MAPREDUCE2 Service Check (batch 7 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"MAPREDUCE2\"}]}"
        },
        {
          "order_id" : 8,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SPARK2_SERVICE_CHECK\",\"context\":\"SPARK2 Service Check (batch 8 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SPARK2\"}]}"
        },
        {
          "order_id" : 9,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SQOOP_SERVICE_CHECK\",\"context\":\"SQOOP Service Check (batch 9 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SQOOP\"}]}"
        },
        {
          "order_id" : 10,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SUPERSET_SERVICE_CHECK\",\"context\":\"SUPERSET Service Check (batch 10 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SUPERSET\"}]}"
        },
        {
          "order_id" : 11,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"TEZ_SERVICE_CHECK\",\"context\":\"TEZ Service Check (batch 11 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"TEZ\"}]}"
        },
        {
          "order_id" : 12,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"YARN_SERVICE_CHECK\",\"context\":\"YARN Service Check (batch 12 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"YARN\"}]}"
        },
        {
          "order_id" : 13,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"ZOOKEEPER_QUORUM_SERVICE_CHECK\",\"context\":\"ZOOKEEPER Service Check (batch 13 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"ZOOKEEPER\"}]}"
        }
      ],
      "batch_settings" : {
        "batch_separation_in_seconds" : 1,
        "task_failure_tolerance_limit" : 1
      }
    },
    "cluster_name" : "cluster",
    "create_time" : "2019-10-18 12:06:25",
    "create_user" : "admin",
    "description" : null,
    "id" : 7,
    "last_execution_status" : "Accepted",
    "schedule" : null,
    "status" : "SCHEDULED",
    "update_time" : "2019-10-18 12:06:25",
    "update_user" : "admin"
  }
}
2019-10-18 12:06:26,347 - ambari_trigger_service_checks.py[parse_scheduled_request:325](1) - INFO  - Accepted: AMBARI_METRICS_SERVICE_CHECK: AMBARI_METRICS Service Check (batch 1 of 13)
2019-10-18 12:06:27,347 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:27,347 - ambari_trigger_service_checks.py[req:184](1) - DEBUG - GET http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7
2019-10-18 12:06:27,510 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Content-Encoding:gzip
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node01tot21v7tibfw16491j20poejl242.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Vary:Accept-Encoding, User-Agent
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:27 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:27,510 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 200
2019-10-18 12:06:27,510 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7",
  "RequestSchedule" : {
    "authenticated_user" : 1,
    "batch" : {
      "batch_requests" : [
        {
          "order_id" : 1,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"AMBARI_METRICS_SERVICE_CHECK\",\"context\":\"AMBARI_METRICS Service Check (batch 1 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"AMBARI_METRICS\"}]}",
          "request_status" : "IN_PROGRESS",
          "return_code" : 202
        },
        {
          "order_id" : 2,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HBASE_SERVICE_CHECK\",\"context\":\"HBASE Service Check (batch 2 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HBASE\"}]}"
        },
        {
          "order_id" : 3,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HDFS_SERVICE_CHECK\",\"context\":\"HDFS Service Check (batch 3 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HDFS\"}]}"
        },
        {
          "order_id" : 4,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HIVE_SERVICE_CHECK\",\"context\":\"HIVE Service Check (batch 4 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HIVE\"}]}"
        },
        {
          "order_id" : 5,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KAFKA_SERVICE_CHECK\",\"context\":\"KAFKA Service Check (batch 5 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KAFKA\"}]}"
        },
        {
          "order_id" : 6,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KNOX_SERVICE_CHECK\",\"context\":\"KNOX Service Check (batch 6 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KNOX\"}]}"
        },
        {
          "order_id" : 7,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"MAPREDUCE2_SERVICE_CHECK\",\"context\":\"MAPREDUCE2 Service Check (batch 7 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"MAPREDUCE2\"}]}"
        },
        {
          "order_id" : 8,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SPARK2_SERVICE_CHECK\",\"context\":\"SPARK2 Service Check (batch 8 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SPARK2\"}]}"
        },
        {
          "order_id" : 9,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SQOOP_SERVICE_CHECK\",\"context\":\"SQOOP Service Check (batch 9 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SQOOP\"}]}"
        },
        {
          "order_id" : 10,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SUPERSET_SERVICE_CHECK\",\"context\":\"SUPERSET Service Check (batch 10 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SUPERSET\"}]}"
        },
        {
          "order_id" : 11,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"TEZ_SERVICE_CHECK\",\"context\":\"TEZ Service Check (batch 11 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"TEZ\"}]}"
        },
        {
          "order_id" : 12,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"YARN_SERVICE_CHECK\",\"context\":\"YARN Service Check (batch 12 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"YARN\"}]}"
        },
        {
          "order_id" : 13,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"ZOOKEEPER_QUORUM_SERVICE_CHECK\",\"context\":\"ZOOKEEPER Service Check (batch 13 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"ZOOKEEPER\"}]}"
        }
      ],
      "batch_settings" : {
        "batch_separation_in_seconds" : 1,
        "task_failure_tolerance_limit" : 1
      }
    },
    "cluster_name" : "cluster",
    "create_time" : "2019-10-18 12:06:25",
    "create_user" : "admin",
    "description" : null,
    "id" : 7,
    "last_execution_status" : "IN_PROGRESS",
    "schedule" : null,
    "status" : "SCHEDULED",
    "update_time" : "2019-10-18 12:06:25",
    "update_user" : "admin"
  }
}
2019-10-18 12:06:27,511 - ambari_trigger_service_checks.py[parse_scheduled_request:325](1) - INFO  - IN_PROGRESS: AMBARI_METRICS_SERVICE_CHECK: AMBARI_METRICS Service Check (batch 1 of 13)
2019-10-18 12:06:28,511 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:28,512 - ambari_trigger_service_checks.py[req:184](1) - DEBUG - GET http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7
2019-10-18 12:06:28,688 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Content-Encoding:gzip
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node01qkj6ve6xuzn7zfewbkiaxvcb243.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Vary:Accept-Encoding, User-Agent
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:28 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:28,688 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 200
2019-10-18 12:06:28,688 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7",
  "RequestSchedule" : {
    "authenticated_user" : 1,
    "batch" : {
      "batch_requests" : [
        {
          "order_id" : 1,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"AMBARI_METRICS_SERVICE_CHECK\",\"context\":\"AMBARI_METRICS Service Check (batch 1 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"AMBARI_METRICS\"}]}",
          "request_status" : "IN_PROGRESS",
          "return_code" : 202
        },
        {
          "order_id" : 2,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HBASE_SERVICE_CHECK\",\"context\":\"HBASE Service Check (batch 2 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HBASE\"}]}"
        },
        {
          "order_id" : 3,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HDFS_SERVICE_CHECK\",\"context\":\"HDFS Service Check (batch 3 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HDFS\"}]}"
        },
        {
          "order_id" : 4,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HIVE_SERVICE_CHECK\",\"context\":\"HIVE Service Check (batch 4 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HIVE\"}]}"
        },
        {
          "order_id" : 5,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KAFKA_SERVICE_CHECK\",\"context\":\"KAFKA Service Check (batch 5 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KAFKA\"}]}"
        },
        {
          "order_id" : 6,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KNOX_SERVICE_CHECK\",\"context\":\"KNOX Service Check (batch 6 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KNOX\"}]}"
        },
        {
          "order_id" : 7,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"MAPREDUCE2_SERVICE_CHECK\",\"context\":\"MAPREDUCE2 Service Check (batch 7 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"MAPREDUCE2\"}]}"
        },
        {
          "order_id" : 8,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SPARK2_SERVICE_CHECK\",\"context\":\"SPARK2 Service Check (batch 8 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SPARK2\"}]}"
        },
        {
          "order_id" : 9,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SQOOP_SERVICE_CHECK\",\"context\":\"SQOOP Service Check (batch 9 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SQOOP\"}]}"
        },
        {
          "order_id" : 10,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SUPERSET_SERVICE_CHECK\",\"context\":\"SUPERSET Service Check (batch 10 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SUPERSET\"}]}"
        },
        {
          "order_id" : 11,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"TEZ_SERVICE_CHECK\",\"context\":\"TEZ Service Check (batch 11 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"TEZ\"}]}"
        },
        {
          "order_id" : 12,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"YARN_SERVICE_CHECK\",\"context\":\"YARN Service Check (batch 12 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"YARN\"}]}"
        },
        {
          "order_id" : 13,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"ZOOKEEPER_QUORUM_SERVICE_CHECK\",\"context\":\"ZOOKEEPER Service Check (batch 13 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"ZOOKEEPER\"}]}"
        }
      ],
      "batch_settings" : {
        "batch_separation_in_seconds" : 1,
        "task_failure_tolerance_limit" : 1
      }
    },
    "cluster_name" : "cluster",
    "create_time" : "2019-10-18 12:06:25",
    "create_user" : "admin",
    "description" : null,
    "id" : 7,
    "last_execution_status" : "IN_PROGRESS",
    "schedule" : null,
    "status" : "SCHEDULED",
    "update_time" : "2019-10-18 12:06:25",
    "update_user" : "admin"
  }
}
2019-10-18 12:06:28,689 - ambari_trigger_service_checks.py[parse_scheduled_request:325](1) - INFO  - IN_PROGRESS: AMBARI_METRICS_SERVICE_CHECK: AMBARI_METRICS Service Check (batch 1 of 13)
2019-10-18 12:06:29,689 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:29,689 - ambari_trigger_service_checks.py[req:184](1) - DEBUG - GET http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7
2019-10-18 12:06:29,876 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Content-Encoding:gzip
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node01rls9xgy5t4cv7p5vhbgaqkmq244.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Vary:Accept-Encoding, User-Agent
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:29 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:29,876 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 200
2019-10-18 12:06:29,876 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7",
  "RequestSchedule" : {
    "authenticated_user" : 1,
    "batch" : {
      "batch_requests" : [
        {
          "order_id" : 1,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"AMBARI_METRICS_SERVICE_CHECK\",\"context\":\"AMBARI_METRICS Service Check (batch 1 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"AMBARI_METRICS\"}]}",
          "request_status" : "IN_PROGRESS",
          "return_code" : 202
        },
        {
          "order_id" : 2,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HBASE_SERVICE_CHECK\",\"context\":\"HBASE Service Check (batch 2 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HBASE\"}]}"
        },
        {
          "order_id" : 3,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HDFS_SERVICE_CHECK\",\"context\":\"HDFS Service Check (batch 3 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HDFS\"}]}"
        },
        {
          "order_id" : 4,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HIVE_SERVICE_CHECK\",\"context\":\"HIVE Service Check (batch 4 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HIVE\"}]}"
        },
        {
          "order_id" : 5,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KAFKA_SERVICE_CHECK\",\"context\":\"KAFKA Service Check (batch 5 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KAFKA\"}]}"
        },
        {
          "order_id" : 6,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KNOX_SERVICE_CHECK\",\"context\":\"KNOX Service Check (batch 6 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KNOX\"}]}"
        },
        {
          "order_id" : 7,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"MAPREDUCE2_SERVICE_CHECK\",\"context\":\"MAPREDUCE2 Service Check (batch 7 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"MAPREDUCE2\"}]}"
        },
        {
          "order_id" : 8,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SPARK2_SERVICE_CHECK\",\"context\":\"SPARK2 Service Check (batch 8 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SPARK2\"}]}"
        },
        {
          "order_id" : 9,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SQOOP_SERVICE_CHECK\",\"context\":\"SQOOP Service Check (batch 9 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SQOOP\"}]}"
        },
        {
          "order_id" : 10,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SUPERSET_SERVICE_CHECK\",\"context\":\"SUPERSET Service Check (batch 10 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SUPERSET\"}]}"
        },
        {
          "order_id" : 11,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"TEZ_SERVICE_CHECK\",\"context\":\"TEZ Service Check (batch 11 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"TEZ\"}]}"
        },
        {
          "order_id" : 12,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"YARN_SERVICE_CHECK\",\"context\":\"YARN Service Check (batch 12 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"YARN\"}]}"
        },
        {
          "order_id" : 13,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"ZOOKEEPER_QUORUM_SERVICE_CHECK\",\"context\":\"ZOOKEEPER Service Check (batch 13 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"ZOOKEEPER\"}]}"
        }
      ],
      "batch_settings" : {
        "batch_separation_in_seconds" : 1,
        "task_failure_tolerance_limit" : 1
      }
    },
    "cluster_name" : "cluster",
    "create_time" : "2019-10-18 12:06:25",
    "create_user" : "admin",
    "description" : null,
    "id" : 7,
    "last_execution_status" : "IN_PROGRESS",
    "schedule" : null,
    "status" : "SCHEDULED",
    "update_time" : "2019-10-18 12:06:25",
    "update_user" : "admin"
  }
}
2019-10-18 12:06:29,876 - ambari_trigger_service_checks.py[parse_scheduled_request:325](1) - INFO  - IN_PROGRESS: AMBARI_METRICS_SERVICE_CHECK: AMBARI_METRICS Service Check (batch 1 of 13)
2019-10-18 12:06:30,877 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:30,877 - ambari_trigger_service_checks.py[req:184](1) - DEBUG - GET http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7
2019-10-18 12:06:31,035 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Content-Encoding:gzip
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node01qxjd9dedkecfkf2fpfjmgev6245.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Vary:Accept-Encoding, User-Agent
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:31 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:31,035 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 200
2019-10-18 12:06:31,035 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7",
  "RequestSchedule" : {
    "authenticated_user" : 1,
    "batch" : {
      "batch_requests" : [
        {
          "order_id" : 1,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"AMBARI_METRICS_SERVICE_CHECK\",\"context\":\"AMBARI_METRICS Service Check (batch 1 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"AMBARI_METRICS\"}]}",
          "request_status" : "IN_PROGRESS",
          "return_code" : 202
        },
        {
          "order_id" : 2,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HBASE_SERVICE_CHECK\",\"context\":\"HBASE Service Check (batch 2 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HBASE\"}]}"
        },
        {
          "order_id" : 3,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HDFS_SERVICE_CHECK\",\"context\":\"HDFS Service Check (batch 3 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HDFS\"}]}"
        },
        {
          "order_id" : 4,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HIVE_SERVICE_CHECK\",\"context\":\"HIVE Service Check (batch 4 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HIVE\"}]}"
        },
        {
          "order_id" : 5,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KAFKA_SERVICE_CHECK\",\"context\":\"KAFKA Service Check (batch 5 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KAFKA\"}]}"
        },
        {
          "order_id" : 6,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KNOX_SERVICE_CHECK\",\"context\":\"KNOX Service Check (batch 6 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KNOX\"}]}"
        },
        {
          "order_id" : 7,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"MAPREDUCE2_SERVICE_CHECK\",\"context\":\"MAPREDUCE2 Service Check (batch 7 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"MAPREDUCE2\"}]}"
        },
        {
          "order_id" : 8,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SPARK2_SERVICE_CHECK\",\"context\":\"SPARK2 Service Check (batch 8 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SPARK2\"}]}"
        },
        {
          "order_id" : 9,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SQOOP_SERVICE_CHECK\",\"context\":\"SQOOP Service Check (batch 9 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SQOOP\"}]}"
        },
        {
          "order_id" : 10,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SUPERSET_SERVICE_CHECK\",\"context\":\"SUPERSET Service Check (batch 10 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SUPERSET\"}]}"
        },
        {
          "order_id" : 11,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"TEZ_SERVICE_CHECK\",\"context\":\"TEZ Service Check (batch 11 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"TEZ\"}]}"
        },
        {
          "order_id" : 12,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"YARN_SERVICE_CHECK\",\"context\":\"YARN Service Check (batch 12 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"YARN\"}]}"
        },
        {
          "order_id" : 13,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"ZOOKEEPER_QUORUM_SERVICE_CHECK\",\"context\":\"ZOOKEEPER Service Check (batch 13 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"ZOOKEEPER\"}]}"
        }
      ],
      "batch_settings" : {
        "batch_separation_in_seconds" : 1,
        "task_failure_tolerance_limit" : 1
      }
    },
    "cluster_name" : "cluster",
    "create_time" : "2019-10-18 12:06:25",
    "create_user" : "admin",
    "description" : null,
    "id" : 7,
    "last_execution_status" : "IN_PROGRESS",
    "schedule" : null,
    "status" : "SCHEDULED",
    "update_time" : "2019-10-18 12:06:25",
    "update_user" : "admin"
  }
}
2019-10-18 12:06:31,035 - ambari_trigger_service_checks.py[parse_scheduled_request:325](1) - INFO  - IN_PROGRESS: AMBARI_METRICS_SERVICE_CHECK: AMBARI_METRICS Service Check (batch 1 of 13)
2019-10-18 12:06:32,036 - ambari_trigger_service_checks.py[req:173](1) - DEBUG - X-Requested-By: admin
2019-10-18 12:06:32,036 - ambari_trigger_service_checks.py[req:184](1) - DEBUG - GET http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7
2019-10-18 12:06:32,190 - ambari_trigger_service_checks.py[req:190](1) - DEBUG - headers:
X-XSS-Protection:1; mode=block
X-Content-Type-Options:nosniff, nosniff
Content-Encoding:gzip
Transfer-Encoding:chunked
Set-Cookie:AMBARISESSIONID=node0ga3wdc5gp57p293upof7q2fz247.node0;Path=/;HttpOnly
Expires:Thu, 01 Jan 1970 00:00:00 GMT
Vary:Accept-Encoding, User-Agent
User:admin
Pragma:no-cache
Cache-Control:no-store
Date:Fri, 18 Oct 2019 12:06:32 GMT
X-Frame-Options:DENY
Content-Type:text/plain;charset=utf-8
2019-10-18 12:06:32,190 - ambari_trigger_service_checks.py[req:191](1) - DEBUG - status code: 200
2019-10-18 12:06:32,191 - ambari_trigger_service_checks.py[req:192](1) - DEBUG - body:
{
  "href" : "http://bf-hdp-1.mydomain.com:8080/api/v1/clusters/cluster/request_schedules/7",
  "RequestSchedule" : {
    "authenticated_user" : 1,
    "batch" : {
      "batch_requests" : [
        {
          "order_id" : 1,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"AMBARI_METRICS_SERVICE_CHECK\",\"context\":\"AMBARI_METRICS Service Check (batch 1 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"AMBARI_METRICS\"}]}",
          "request_status" : "COMPLETED",
          "return_code" : 202
        },
        {
          "order_id" : 2,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HBASE_SERVICE_CHECK\",\"context\":\"HBASE Service Check (batch 2 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HBASE\"}]}"
        },
        {
          "order_id" : 3,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HDFS_SERVICE_CHECK\",\"context\":\"HDFS Service Check (batch 3 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HDFS\"}]}"
        },
        {
          "order_id" : 4,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"HIVE_SERVICE_CHECK\",\"context\":\"HIVE Service Check (batch 4 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"HIVE\"}]}"
        },
        {
          "order_id" : 5,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KAFKA_SERVICE_CHECK\",\"context\":\"KAFKA Service Check (batch 5 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KAFKA\"}]}"
        },
        {
          "order_id" : 6,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"KNOX_SERVICE_CHECK\",\"context\":\"KNOX Service Check (batch 6 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"KNOX\"}]}"
        },
        {
          "order_id" : 7,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"MAPREDUCE2_SERVICE_CHECK\",\"context\":\"MAPREDUCE2 Service Check (batch 7 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"MAPREDUCE2\"}]}"
        },
        {
          "order_id" : 8,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SPARK2_SERVICE_CHECK\",\"context\":\"SPARK2 Service Check (batch 8 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SPARK2\"}]}"
        },
        {
          "order_id" : 9,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SQOOP_SERVICE_CHECK\",\"context\":\"SQOOP Service Check (batch 9 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SQOOP\"}]}"
        },
        {
          "order_id" : 10,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"SUPERSET_SERVICE_CHECK\",\"context\":\"SUPERSET Service Check (batch 10 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"SUPERSET\"}]}"
        },
        {
          "order_id" : 11,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"TEZ_SERVICE_CHECK\",\"context\":\"TEZ Service Check (batch 11 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"TEZ\"}]}"
        },
        {
          "order_id" : 12,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"YARN_SERVICE_CHECK\",\"context\":\"YARN Service Check (batch 12 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"YARN\"}]}"
        },
        {
          "order_id" : 13,
          "request_type" : "POST",
          "request_uri" : "/api/v1/clusters/cluster/requests",
          "request_body" : "{\"RequestInfo\":{\"command\":\"ZOOKEEPER_QUORUM_SERVICE_CHECK\",\"context\":\"ZOOKEEPER Service Check (batch 13 of 13)\"},\"Requests/resource_filters\":[{\"service_name\":\"ZOOKEEPER\"}]}"
        }
      ],
      "batch_settings" : {
        "batch_separation_in_seconds" : 1,
        "task_failure_tolerance_limit" : 1
      }
    },
    "cluster_name" : "cluster",
    "create_time" : "2019-10-18 12:06:25",
    "create_user" : "admin",
    "description" : null,
    "id" : 7,
    "last_execution_status" : "COMPLETED",
    "schedule" : null,
    "status" : "SCHEDULED",
    "update_time" : "2019-10-18 12:06:25",
    "update_user" : "admin"
  }
}
2019-10-18 12:06:32,191 - ambari_trigger_service_checks.py[parse_scheduled_request:311](1) - INFO  - COMPLETED
HariSekhon commented 5 years ago

I've just pushed a potential fix, can you please do:

docker run -ti harisekhon/pytools bash
git pull
git checkout ambari_trigger_service_checks_wait_fix

and then run the ./ambari_trigger_service_checks.py ... command again to test the version I've just pushed in that branch.

HariSekhon commented 4 years ago

@maciex did you get a change to try this update?

maciex commented 4 years ago

@HariSekhon I'm not using the tools on a daily basis.I just used it once to upgrade the dev cluster, which I don't have already. Sorry I can't retest the fix.

HariSekhon commented 4 years ago

Merged and closed.