Create Smoke test(s) for 2.0

[gshif]

There is no E2E test(s) for 2.0. Currently, only integration tests for gateway and etcd components ( to create/edit/delete/show users/organizations/buckets) exist. Need to create E2E test(s) that touches all of the components (or almost all the components with the exception of the ones that are not yet working - such as tasks)

[gshif]

litmus_run_master script has been already checking the status of the following services by calling http://<gateway>:9999/healthz rest api:

• gateway
• kafka
• etcd

1. Need to add check for the rest of the components to the litmus_run_master script, i.e.:

   • storage (status checked by queryd)
   • etcd-tasks
   • tasks
   • transpilerde

       curl -GET http://localhost:8098/healthz
   
      {
         "status": "healthy",
         "checks": [
              {
                     "name": "internal",
                     "status": "healthy"
               }
                         ]
       }

• queryd (also checks status of storage)

     curl -GET http://localhost:8093/healthz

    {
         "status": "healthy",
         "checks": [
                {
                    "name": "internal",
                    "status": "healthy"
                },
                {
                    "name": "storage",
                     "status": "healthy",
                     "checks": [
                               {
                                   "name": "partition 0",
                                  "status": "healthy",
                                  "checks": [
                                         {
                                              "status": "healthy",
                                              "message": "connection state to storage-0.storage:8082 is READY"
                                         }
                                                   ]
                                 }
                                        ]
              }
                               ]
     }

[gshif]

tasks and etcd-task services health check is not implemented yet. Need to ask the tasks team what is the correct url.

[gshif]

• Added an option to the pytest to be able to pass a transpilerde url in order to get a health of the transpiler component.
• Added health check for queryd, transpilerde and storage services in addition to already existing check for gateway, etcd and kafka services. This is a high level check, e.g. health check of queryd service also checks for health of storage service and the health of each partition, so current check gets oversll health of the storage and queryd services, but not of each service in particular. https://github.com/influxdata/Litmus/pull/89

[gshif]

2. Smoke E2E test:
   • Create an Organization. Check organization has been successfully created.
   • Create a bucket for the created organization. Check bucket has been successfully created.
   • Create a user. Check user was successfully created.
   • Give user permissions to write to a bucket and read from the same bucket.
   • Write data to a bucket. (Would be nice to check kafka got the data). Check data was written to one of the storage partitions.
   • Query written data using queryd.
   • Query written data using transpilerde.
   • Query written data using gateway. In the future querying data with three different services will be divided into three different test cases.

[gshif]

3. Smoke test for tasks service. Has two parts:
   • Health Check of the task and task-etcd services. It is not implemented to be used remotely. Have created an issue for task team to expose the service to the outside world as it is done for other services. The workaround is to get to the telegraf container in the tasks pod and run a health check locally, i.e.: curl -vv -GET http://localhost:8276/health. The example output (when task service is not healthy):

   • Trying 127.0.0.1...
   • TCP_NODELAY set
   • Connected to localhost (127.0.0.1) port 8276 (#0)

     GET /health HTTP/1.1 Host: localhost:8276 User-Agent: curl/7.52.1 Accept: /

     < HTTP/1.1 503 Service Unavailable < Date: Fri, 09 Nov 2018 21:36:36 GMT < Content-Length: 225 < Content-Type: text/plain; charset=utf-8 < { "status": "unhealthy", "checks": [ { "name": "internal", "status": "healthy" }, { "name": "etcdStore", "status": "unhealthy", "message": "underlying session closed" } ] }

   • Curl_http_done: called premature == 0
   • Connection #0 to host localhost left intact

     
     or

{
  "status": "healthy",
  "checks": [
    {
      "name": "internal",
      "status": "healthy"
    },
    {
      "name": "etcdStore",
      "status": "healthy"
    }
  ]
}

[gshif]

• Tasks Smoke Test steps:
  1. Create organization and verify it was created successfully.
  2. Create bucket to read data from and verify it was created successfully.
  3. Create bucket to write data to and verify it was created successfully.
  4. Create user.
  5. Create an authorization token.
  6. Create task.
  7. Verify created task.
  8. Write a point to a bucket the data is read from.
  9. Verify data was written to kafka.
  10. Verify data was written to storage.
  11. Query written data using gateway endpoint.
  12. Wait for task to be executed.
  13. Verify data was written to kafka.
  14. Verify data was written to a bucket we wrote a data to.
  15. Query the data from a bucket the data was written to.

[gshif]

• Tasks service is using two different endpoints to check for readiness and health:

  • http://:8276/health

    {
       "status": "healthy",
       "checks": [
           {
               "name": "internal",
               "status": "healthy"
            },
            {
               "name": "etcdStore",
               "status": "healthy"
             } 
        ]
    }

  • http://:8276/ready

    {
       "status": "healthy",
       "checks": [
              {
                "name": "internal",
                "status": "healthy"
              },
              {
                 "name": "Kafka producer",
                 "status": "healthy"
              },
              {
                 "name": "Query Service",
                "status": "healthy"
               }
        ]
    }

• Readiness: Tasks service is not ready to accept requests if ready endpoint returns unhealthy status.

• Liveness: Tasks pod(s) will be restarted if health endpoint returns unhealthy status.

I am using health endpoint to check the health of the tasks, i.e. if tasks is not connected to etcd-tasks, then task cannot really be created, but, to the best of my knowledge, if queryd or kafka is unhealthy, tasks can still queue the task(s) to be executed.

[gshif]

• Tasks health has been merged into master of idpe : https://github.com/influxdata/idpe/pull/2142

[gshif]

Once the platform(influxdb) code is merged into idpe, the smoke tests would have to be revised due to changes in rest api. Once merge happens, the separate issue would be created to track the work required to make smoke test work again.