index pattern aws-billing-*

[tbsoares]

The index are not created: root@d75355af3996:/usr/share/elasticsearch# curl -s '172.19.0.2:9200/_cat/indices?v' health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open .kibana 1 1 1 0 3.1kb 3.1kb

[droidlabour]

Can you provide the logs. It takes some time to download the billing reports, and index it via Logstash. Try waiting for sometime, and see if the index is created. Generally it takes around 4-5 minutes at max to index each billing report.

[tbsoares]

Hey!

In fact the error is when the awselkbilling_aws-elk-billing container is starting:

The error is: Traceback (most recent call last): File "/aws-elk-billing/orchestrate.py", line 27, in s3_dir_to_index = tools.get_s3_bucket_dir_to_index() File "/aws-elk-billing/tools/tools.py", line 100, in get_s3_bucket_dir_to_index for keys in key_names['CommonPrefixes']: KeyError: 'CommonPrefixes'

[aaronwest]

Getting the same error as @tbsoares. I tried different things to ensure I wasn't missing something such as altering the bucket permissions, changing the values in prod.env, etc. I'm gonna keep looking into it.

aws-elk-billing_1  | Checking if Kibana container has started to listen to 5160
aws-elk-billing_1  | Great Kibana is listening on 5601 :)
aws-elk-billing_1  | ES mapping created :)
aws-elk-billing_1  | Traceback (most recent call last):
aws-elk-billing_1  |   File "/aws-elk-billing/orchestrate.py", line 27, in <module>
aws-elk-billing_1  |     s3_dir_to_index = tools.get_s3_bucket_dir_to_index()
aws-elk-billing_1  |   File "/aws-elk-billing/tools/tools.py", line 100, in get_s3_bucket_dir_to_index
aws-elk-billing_1  |     for keys in key_names['CommonPrefixes']:
aws-elk-billing_1  | KeyError: 'CommonPrefixes'
awselkbilling_aws-elk-billing_1 exited with code 1

[aaronwest]

Digging into this deeper I ran into several path issues in tools.py. Some I were able to resolve by making code changes but I decided not to spend anymore time on it. I think some assumptions are made regarding S3_BILL_PATH_NAME in prod.env. I tried two different setups, one where the AWS report was configured with a path prefix and one without. In both cases there were issues.

[droidlabour]

I've added some changes with the help of @victortrac . You need to change prod.env. Please refer to the prod.sample.env for the environment variables. Instructions for finding out the values is here https://github.com/PriceBoardIn/aws-elk-billing#set-s3-credentials-and-aws-billing-bucket-and-directory-name

[aaronwest]

@droidlabour @victortrac - sweet, I'll pull here in a minute and fire up a test. I have two different AWS report configurations I can test these changes on.

[droidlabour]

@aaronwest Sorry there's still an error. If your Report Path is something like dir1/dir2 then it's not going to work. Lemme fix it.

[victortrac]

@droidlabour my original patch supports subpaths. The strip() is intentional.

[aaronwest]

Firing up the containers, retrieving the gzipped data, and creating the aws-billing-* Kibana index is working fine on the first pass. But there's still some sort of issue going on. See the error log below. I wonder if this error is also why the Kibana visualizations and dashboard aren't getting created. If you fire up the app and attempt to load http://localhost:5601/app/kibana#/dashboard/AWS-Billing-DashBoard I get a message about the dashboard not being created.

aws-elk-billing_1  | Opening billing report filename /aws-elk-billing/billing_report_2016-07.csv
aws-elk-billing_1  | Publishing record to logstash %v 0xc820012360
aws-elk-billing_1  | Publishing record to logstash %v 0xc8200127e0
aws-elk-billing_1  | Publishing record to logstash %v 0xc820012720
aws-elk-billing_1  |
aws-elk-billing_1  | panic: runtime error: index out of range
aws-elk-billing_1  |
aws-elk-billing_1  | goroutine 13 [running]:
aws-elk-billing_1  | panic(0x5bd640, 0xc820010070)
aws-elk-billing_1  |    /usr/local/go/src/runtime/panic.go:481 +0x3e6
aws-elk-billing_1  | main.ParseRecord(0xc820012660, 0xc820012780, 0xc820012600)
aws-elk-billing_1  |    /aws-elk-billing/main.go:167 +0x9de
aws-elk-billing_1  | created by main.main
aws-elk-billing_1  |    /aws-elk-billing/main.go:101 +0x483
aws-elk-billing_1  | exit status 2
aws-elk-billing_1  |
aws-elk-billing_1  | Something went wrong while getting the file reference or while talking with logstash

[droidlabour]

@victortrac I've already tested your changes. It wasn't handling the case where the Report Path is /directory. It was working fine for directory dir1/dir2. I just pushed another commit https://github.com/PriceBoardIn/aws-elk-billing/commit/2940418d06ef9cecb21de1e21ae28da2bbfd80bf that has fixed this issue.

[droidlabour]

@aaronwest Yeah the kibana index, visualization & dashboard are created after the golang code has finished indexing to ES via logstash. Can you try deleting all the ES index and retrying.

[aaronwest]

@droidlabour do you mean deleting the Kibana index and firing up the containers again with docker-compose up?

[aaronwest]

@droidlabour I deleted the Elasticsearch index with:

curl -XDELETE 'http://192.168.99.100:9200/aws-billing-2016.07'

Then, I deleted the associated Kibana index: aws-billing-*.

Finally, I restarted the app with docker-compose up. I'm still getting the same errors in main.go.

aws-elk-billing_1  | aws-billing* indice deleted or Not found, its OK main golang code will create a new one for you :)
aws-elk-billing_1  | AWS billing CSV report parser!
aws-elk-billing_1  | out after make(chan []byte) is %v 0xc820012360
aws-elk-billing_1  | Opening billing report filename /aws-elk-billing/billing_report_2016-07.csv
aws-elk-billing_1  | Publishing record to logstash %v 0xc820012360
aws-elk-billing_1  | Publishing record to logstash %v 0xc8200127e0
aws-elk-billing_1  | Publishing record to logstash %v 0xc820012720
aws-elk-billing_1  |
aws-elk-billing_1  | panic: runtime error: index out of range
aws-elk-billing_1  |
aws-elk-billing_1  | goroutine 10 [running]:
aws-elk-billing_1  | panic(0x5bd640, 0xc820010070)
aws-elk-billing_1  |    /usr/local/go/src/runtime/panic.go:481 +0x3e6
aws-elk-billing_1  | main.ParseRecord(0xc820012660, 0xc8200126c0, 0xc8200124e0)
aws-elk-billing_1  |    /aws-elk-billing/main.go:167 +0x9de
aws-elk-billing_1  | created by main.main
aws-elk-billing_1  |    /aws-elk-billing/main.go:101 +0x483
aws-elk-billing_1  | exit status 2
aws-elk-billing_1  |
aws-elk-billing_1  | Something went wrong while getting the file reference or while talking with logstash
awselkbilling_aws-elk-billing_1 exited with code 1

Elasticsearch is getting the index so I'm not sure where the breakdown is. Haven't looked much deeper yet. NOTE: I have waited more than 10 minutes after firing up the containers so I'm pretty sure this isn't a timing issue. Especially considering the errors.

curl 'http://192.168.99.100:9200/_cluster/health?level=indices&pretty=1'

{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 6,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0,
  "indices" : {
    "aws-billing-2016.07" : {
      "status" : "yellow",
      "number_of_shards" : 5,
      "number_of_replicas" : 1,
      "active_primary_shards" : 5,
      "active_shards" : 5,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 5
    },
    ".kibana" : {
      "status" : "yellow",
      "number_of_shards" : 1,
      "number_of_replicas" : 1,
      "active_primary_shards" : 1,
      "active_shards" : 1,
      "relocating_shards" : 0,
      "initializing_shards" : 0,
      "unassigned_shards" : 1
    }
  }
}

[droidlabour]

@aaronwest Can you make sure that you are using the AWS Cost and Usage Report type.

[aaronwest]

@droidlabour Certainly. Here's a screenshot of the report config.

[droidlabour]

@aaronwest Can you list all the fields in your billing report CSV file?

[aaronwest]

@droidlabour Here you go:

identity/LineItemId,identity/TimeInterval,bill/InvoiceId,bill/BillingEntity,bill/BillType,bill/PayerAccountId,bill/BillingPeriodStartDate,bill/BillingPeriodEndDate,lineItem/UsageAccountId,lineItem/LineItemType,lineItem/UsageStartDate,lineItem/UsageEndDate,lineItem/ProductCode,lineItem/UsageType,lineItem/Operation,lineItem/AvailabilityZone,lineItem/ResourceId,lineItem/UsageAmount,lineItem/CurrencyCode,lineItem/UnblendedRate,lineItem/UnblendedCost,lineItem/BlendedRate,lineItem/BlendedCost,lineItem/LineItemDescription,lineItem/TaxType,product/ProductName,product/availability,product/durability,product/fromLocation,product/fromLocationType,product/group,product/groupDescription,product/location,product/locationType,product/operation,product/productFamily,product/servicecode,product/sku,product/storageClass,product/storageMedia,product/toLocation,product/toLocationType,product/transferType,product/usagetype,product/volumeType,pricing/term

[droidlabour]

@aaronwest Oh I got it, you've created the report with Resource IDs dimension. Currently it works without Resource IDs. Trying creating one.

[aaronwest]

@droidlabour I'll try that and let you know how it works out.