search

Teevity / ice

AWS Usage Tool
2.85k stars 434 forks source link

Failed to parse billing file #304

Open sayan-rudder opened 4 years ago

sayan-rudder commented 4 years ago

Failed to parse billing data config: /-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv.zip

ice_1 | | Error unacceptable character '' (0x3) special characters are not allowed ice_1 | in "'reader'", position 2 ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.checkPrintable(StreamReader.java:93) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:192) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.(StreamReader.java:60) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLParser.(YAMLParser.java:147) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:426) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:327) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:300) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14) ice_1 | | Error at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726) ice_1 | | Error at com.netflix.ice.processor.config.BillingDataConfig.(BillingDataConfig.java:148) ice_1 | | Error at com.netflix.ice.processor.ProcessorConfig.readBillingDataConfig(ProcessorConfig.java:605) ice_1 | | Error at com.netflix.ice.processor.ProcessorConfig.processBillingDataConfig(ProcessorConfig.java:525)

Is this expected, any pointers will be helpful

nfonrose commented 4 years ago

It looks like a configuration error inside your ice.properties file.

The line of code failing is trying to read a YAML based configuration file. https://github.com/jimroth/ice/blob/master/src/java/com/netflix/ice/processor/config/BillingDataConfig.java#L148

You probably need to post the configuration here if you need more help.

Nicolas

-- Nicolas Fonrose | Teevity | Founder +33.6.61.35.43.31 https://teevity.com - Cloud Costs Analytics for AWS, GCP and Azure powered by NetflixOSS Ice twitter - @nfonrose / @teevity

On Tue, Jun 2, 2020 at 9:38 AM sayan-rudder notifications@github.com wrote:

Failed to parse billing data config:

/-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv.zip ice_1 | | Error unacceptable character '' (0x3) special characters are not allowed ice_1 | in "'reader'", position 2 ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.checkPrintable(StreamReader.java:93) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.update(StreamReader.java:192) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.snakeyaml.reader.StreamReader.(StreamReader.java:60) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLParser.(YAMLParser.java:147) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory._createParser(YAMLFactory.java:426) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:327) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:300) ice_1 | | Error at com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:14) ice_1 | | Error at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726) ice_1 | | Error at com.netflix.ice.processor.config.BillingDataConfig.(BillingDataConfig.java:148) ice_1 | | Error at com.netflix.ice.processor.ProcessorConfig.readBillingDataConfig(ProcessorConfig.java:605) ice_1 | | Error at com.netflix.ice.processor.ProcessorConfig.processBillingDataConfig(ProcessorConfig.java:525) Is this expected, any pointers will be helpful — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub , or unsubscribe .
sayan-rudder commented 4 years ago

Sorry for my low understanding.

is a config file different from the actual aws provided billing data file? The above error is requiring a config file ? Where to find it ?

or should I post the billing data file itself?

Thanks for all the help!

@nfonrose

sayan-rudder commented 4 years ago

I think, it processed one of the billing files now but NPE in saving the data :(

ice_1 | 2020-06-02 05:23:16,630 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.DetailedBillingReportProcessor - processed 500000 lines... ice_1 | 2020-06-02 05:23:17,489 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344 ice_1 | 2020-06-02 05:23:17,490 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344 ice_1 | 2020-06-02 05:23:17,490 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344 ice_1 | 2020-06-02 05:23:17,490 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344 ice_1 | 2020-06-02 05:23:17,530 [com.netflix.ice.processor.BillingFileProcessor] INFO basic.BasicLineItemProcessor - account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv Support: com.netflix.ice.processor.DetailedBillingReportLineItem@1a49e344 ice_1 | | Error 2020-06-02 05:23:17,558 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - Estimated,account_id,,Rounding,,,,,,,,,,This line contains rounding error due to the consolidated billing and hourly line item calculation processes.,,,,,0.0095870483,,0.0100143818,, ice_1 | Message: Invalid format: "" ice_1 | Line | Method ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor ice_1 | | 210 | process in '' ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor ice_1 | | 195 | processBillingFile in '' ice_1 | | 152 | processBillingZipFile . in '' ice_1 | | 128 | processReport in '' ice_1 | | 295 | downloadAndProcessReport in '' ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor ice_1 | | 81 | poll . . . . . . . . . . in '' ice_1 | | 50 | doWork in com.netflix.ice.common.Poller ice_1 | | 28 | access$000 . . . . . . . in '' ice_1 | | 88 | run in com.netflix.ice.common.Poller$1 ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread ice_1 | | Error 2020-06-02 05:23:17,653 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - Estimated,account_id,,InvoiceTotal,,,,,,,,,,Total amount for invoice Estimated,,,,,0.01,,0.01,, ice_1 | Message: Invalid format: "" ice_1 | Line | Method ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor ice_1 | | 210 | process in '' ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor ice_1 | | 195 | processBillingFile in '' ice_1 | | 152 | processBillingZipFile . in '' ice_1 | | 128 | processReport in '' ice_1 | | 295 | downloadAndProcessReport in '' ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor ice_1 | | 81 | poll . . . . . . . . . . in '' ice_1 | | 50 | doWork in com.netflix.ice.common.Poller ice_1 | | 28 | access$000 . . . . . . . in '' ice_1 | | 88 | run in com.netflix.ice.common.Poller$1 ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread ice_1 | | Error 2020-06-02 05:23:17,659 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - ,account_id,454531037350,AccountTotal,,,,,,,,,,Total for linked account# 454531037350,,,,,-6.8566690473,,-0.0000044282,, ice_1 | Message: Invalid format: "" ice_1 | Line | Method ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor ice_1 | | 210 | process in '' ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor ice_1 | | 195 | processBillingFile in '' ice_1 | | 152 | processBillingZipFile . in '' ice_1 | | 128 | processReport in '' ice_1 | | 295 | downloadAndProcessReport in '' ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor ice_1 | | 81 | poll . . . . . . . . . . in '' ice_1 | | 50 | doWork in com.netflix.ice.common.Poller ice_1 | | 28 | access$000 . . . . . . . in '' ice_1 | | 88 | run in com.netflix.ice.common.Poller$1 ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread ice_1 | | Error 2020-06-02 05:23:17,662 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - ,account_id,account_id,AccountTotal,,,,,,,,,,Total for linked account# account_id,,,,,6.8570819990,,-0.0000099536,, ice_1 | Message: Invalid format: "" ice_1 | Line | Method ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor ice_1 | | 210 | process in '' ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor ice_1 | | 195 | processBillingFile in '' ice_1 | | 152 | processBillingZipFile . in '' ice_1 | | 128 | processReport in '' ice_1 | | 295 | downloadAndProcessReport in '' ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor ice_1 | | 81 | poll . . . . . . . . . . in '' ice_1 | | 50 | doWork in com.netflix.ice.common.Poller ice_1 | | 28 | access$000 . . . . . . . in '' ice_1 | | 88 | run in com.netflix.ice.common.Poller$1 ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread ice_1 | | Error 2020-06-02 05:23:17,665 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.DetailedBillingReportProcessor - ,account_id,,StatementTotal,,,,,,,,,,Total statement amount for period 2020-05-01 00:00:00 - 2020-05-31 23:59:59,,,,,0.01,,0.01,, ice_1 | Message: Invalid format: "" ice_1 | Line | Method ice_1 | ->> 747 | parseMillis in org.joda.time.format.DateTimeFormatter ice_1 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ice_1 | | 67 | getStartMillis in com.netflix.ice.processor.DetailedBillingReportLineItem ice_1 | | 159 | getUsageInterval . . . . in com.netflix.ice.basic.BasicLineItemProcessor ice_1 | | 210 | process in '' ice_1 | | 234 | processOneLine . . . . . in com.netflix.ice.processor.DetailedBillingReportProcessor ice_1 | | 195 | processBillingFile in '' ice_1 | | 152 | processBillingZipFile . in '' ice_1 | | 128 | processReport in '' ice_1 | | 295 | downloadAndProcessReport in '' ice_1 | | 145 | processReports in com.netflix.ice.processor.BillingFileProcessor ice_1 | | 81 | poll . . . . . . . . . . in '' ice_1 | | 50 | doWork in com.netflix.ice.common.Poller ice_1 | | 28 | access$000 . . . . . . . in '' ice_1 | | 88 | run in com.netflix.ice.common.Poller$1 ice_1 | ^ 748 | run . . . . . . . . . . in java.lang.Thread ice_1 | 2020-06-02 05:23:17,669 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.DetailedBillingReportProcessor - done processing account_id-aws-billing-detailed-line-items-with-resources-and-tags-2020-05.csv.zip, end is 2020-06-01 00:00:00 ice_1 | 2020-06-02 05:23:17,669 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - Process reservations pulled from the accounts ice_1 | 2020-06-02 05:23:17,676 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.DetailedBillingReservationProcessor - ---------- Process Non-resource ice_1 | 2020-06-02 05:23:17,677 [com.netflix.ice.processor.BillingFileProcessor] INFO processor.BillingFileProcessor - adding savings data for 2020-05-01T00:00:00.000Z... ice_1 | | Error 2020-06-02 05:23:17,696 [com.netflix.ice.processor.BillingFileProcessor] ERROR processor.BillingFileProcessor - Failed to process reports: java.lang.NullPointerException ice_1 | | Error java.lang.NullPointerException ice_1 | | Error at com.netflix.ice.processor.BillingFileProcessor.addSavingsData(BillingFileProcessor.java:273) ice_1 | | Error at com.netflix.ice.processor.BillingFileProcessor.processReports(BillingFileProcessor.java:214) ice_1 | | Error at com.netflix.ice.processor.BillingFileProcessor.poll(BillingFileProcessor.java:81) ice_1 | | Error at com.netflix.ice.common.Poller.doWork(Poller.java:50) ice_1 | | Error at com.netflix.ice.common.Poller.access$000(Poller.java:28) ice_1 | | Error at com.netflix.ice.common.Poller$1.run(Poller.java:88) ice_1 | | Error at java.lang.Thread.run(Thread.java:748)