search

project-koku / masu

This is a READ ONLY repo. See https://github.com/project-koku/koku for current masu implementation
GNU Affero General Public License v3.0
5 stars 6 forks source link

ERROR: ON CONFLICT DO UPDATE command cannot affect row a second time #376

Closed dccurtis closed 5 years ago

dccurtis commented 5 years ago

Ingesting cost-usage-bucket as new provider. Expected Jan and Feb. Exception is thrown for January assembly_id f23124d1-cdb0-468d-a84a-a0adaa783280


worker_1  | [2019-02-19 21:59:10,178: WARNING/ForkPoolWorker-3] [2019-02-19 21:59:10,178] INFO in aws_report_downloader: Downloading /koku/20190201-20190301/koku-Manifest.json to /var/tmp/masu/acct10001/aws/cost-usage-bucket/koku-Manifest.json
worker_1  | [2019-02-19 21:59:10,178: INFO/ForkPoolWorker-3] Downloading /koku/20190201-20190301/koku-Manifest.json to /var/tmp/masu/acct10001/aws/cost-usage-bucket/koku-Manifest.json
worker_1  | [2019-02-19 21:59:10,384: WARNING/ForkPoolWorker-3] [2019-02-19 21:59:10,383] INFO in report_downloader_base: Inserting manifest database record for assembly_id: 6d824bc9-41a1-4d72-b090-049e88309771
worker_1  | [2019-02-19 21:59:10,383: INFO/ForkPoolWorker-3] Inserting manifest database record for assembly_id: 6d824bc9-41a1-4d72-b090-049e88309771
worker_1  | [2019-02-19 21:59:10,737: WARNING/ForkPoolWorker-3] [2019-02-19 21:59:10,737] INFO in report_downloader_base: No manifest entry found.  Adding for bill period start: 2019-02-01 00:00:00
worker_1  | [2019-02-19 21:59:10,737: INFO/ForkPoolWorker-3] No manifest entry found.  Adding for bill period start: 2019-02-01 00:00:00
worker_1  | [2019-02-19 21:59:11,011: WARNING/ForkPoolWorker-3] [2019-02-19 21:59:11,011] INFO in aws_report_downloader: Local S3 filename: 6d824bc9-41a1-4d72-b090-049e88309771-koku-1.csv.gz
worker_1  | [2019-02-19 21:59:11,011: INFO/ForkPoolWorker-3] Local S3 filename: 6d824bc9-41a1-4d72-b090-049e88309771-koku-1.csv.gz
worker_1  | [2019-02-19 21:59:11,117: WARNING/ForkPoolWorker-3] [2019-02-19 21:59:11,117] INFO in aws_report_downloader: Downloading /koku/20190201-20190301/6d824bc9-41a1-4d72-b090-049e88309771/koku-1.csv.gz to /var/tmp/masu/acct10001/aws/cost-usage-bucket/6d824bc9-41a1-4d72-b090-049e88309771-koku-1.csv.gz
worker_1  | [2019-02-19 21:59:11,117: INFO/ForkPoolWorker-3] Downloading /koku/20190201-20190301/6d824bc9-41a1-4d72-b090-049e88309771/koku-1.csv.gz to /var/tmp/masu/acct10001/aws/cost-usage-bucket/6d824bc9-41a1-4d72-b090-049e88309771-koku-1.csv.gz
worker_1  | [2019-02-19 21:59:11,415: INFO/ForkPoolWorker-3] masu.processor.tasks.get_report_files[f5fc084d-cb40-4e8c-9d34-a985b839b9b1]: Reports to be processed: [{'file': '/var/tmp/masu/acct10001/aws/cost-usage-bucket/f23124d1-cdb0-468d-a84a-a0adaa783280-koku-1.csv.gz', 'compression': 'GZIP', 'start_date': datetime.datetime(2019, 1, 1, 21, 59, 1, 1, tzinfo=<UTC>), 'assembly_id': 'f23124d1-cdb0-468d-a84a-a0adaa783280', 'manifest_id': 1, 'provider_id': 1}, {'file': '/var/tmp/masu/acct10001/aws/cost-usage-bucket/6d824bc9-41a1-4d72-b090-049e88309771-koku-1.csv.gz', 'compression': 'GZIP', 'start_date': datetime.datetime(2019, 2, 1, 21, 59, 1, 1, tzinfo=<UTC>), 'assembly_id': '6d824bc9-41a1-4d72-b090-049e88309771', 'manifest_id': 2, 'provider_id': 1}]
worker_1  | [2019-02-19 21:59:11,706: INFO/ForkPoolWorker-3] masu.processor.tasks.get_report_files[f5fc084d-cb40-4e8c-9d34-a985b839b9b1]: Processing starting - schema_name: acct10001, provider_uuid: cf7fc6fc-bd7d-486d-a7c0-5018c3ac76e6, File: /var/tmp/masu/acct10001/aws/cost-usage-bucket/f23124d1-cdb0-468d-a84a-a0adaa783280-koku-1.csv.gz
worker_1  | [2019-02-19 21:59:11,707: INFO/ForkPoolWorker-3] masu.processor.tasks.get_report_files[f5fc084d-cb40-4e8c-9d34-a985b839b9b1]: Processing Report: schema_name: acct10001, report_path: /var/tmp/masu/acct10001/aws/cost-usage-bucket/f23124d1-cdb0-468d-a84a-a0adaa783280-koku-1.csv.gz, compression: GZIP, provider: AWS, start_date: 2019-01-01 21:59:01.000001+00:00
worker_1  | [2019-02-19 21:59:11,707: INFO/ForkPoolWorker-3] masu.processor.tasks.get_report_files[f5fc084d-cb40-4e8c-9d34-a985b839b9b1]: Avaiable memory: 1589977088 bytes (14.3%)
worker_1  | [2019-02-19 21:59:12,739: WARNING/ForkPoolWorker-3] [2019-02-19 21:59:12,739] INFO in aws_report_processor: Initialized report processor for file: f23124d1-cdb0-468d-a84a-a0adaa783280-koku-1.csv.gz and schema: acct10001
worker_1  | [2019-02-19 21:59:12,739: INFO/ForkPoolWorker-3] Initialized report processor for file: f23124d1-cdb0-468d-a84a-a0adaa783280-koku-1.csv.gz and schema: acct10001
worker_1  | [2019-02-19 21:59:13,080: WARNING/ForkPoolWorker-3] [2019-02-19 21:59:13,080] INFO in aws_report_processor: File <_io.TextIOWrapper name='/var/tmp/masu/acct10001/aws/cost-usage-bucket/f23124d1-cdb0-468d-a84a-a0adaa783280-koku-1.csv.gz' encoding='UTF-8'> opened for processing
worker_1  | [2019-02-19 21:59:13,080: INFO/ForkPoolWorker-3] File <_io.TextIOWrapper name='/var/tmp/masu/acct10001/aws/cost-usage-bucket/f23124d1-cdb0-468d-a84a-a0adaa783280-koku-1.csv.gz' encoding='UTF-8'> opened for processing
worker_1  | [2019-02-19 21:59:17,349: ERROR/ForkPoolWorker-3] masu.processor.tasks.get_report_files[f5fc084d-cb40-4e8c-9d34-a985b839b9b1]: ON CONFLICT DO UPDATE command cannot affect row a second time
worker_1  | HINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.
worker_1  |
worker_1  | [2019-02-19 21:59:17,724: INFO/ForkPoolWorker-3] masu.processor.tasks.get_report_files[f5fc084d-cb40-4e8c-9d34-a985b839b9b1]: Processing starting - schema_name: acct10001, provider_uuid: cf7fc6fc-bd7d-486d-a7c0-5018c3ac76e6, File: /var/tmp/masu/acct10001/aws/cost-usage-bucket/6d824bc9-41a1-4d72-b090-049e88309771-koku-1.csv.gz
worker_1  | [2019-02-19 21:59:17,725: INFO/ForkPoolWorker-3] masu.processor.tasks.get_report_files[f5fc084d-cb40-4e8c-9d34-a985b839b9b1]: Processing Report: schema_name: acct10001, report_path: /var/tmp/masu/acct10001/aws/cost-usage-bucket/6d824bc9-41a1-4d72-b090-049e88309771-koku-1.csv.gz, compression: GZIP, provider: AWS, start_date: 2019-02-01 21:59:01.000001+00:00

postgres=# select * from reporting_common_costusagereportstatus;
 id |                    report_name                     |    last_completed_datetime    |     last_started_datetime     |                etag                | manifest_id
----+----------------------------------------------------+-------------------------------+-------------------------------+------------------------------------+-------------
  1 | f23124d1-cdb0-468d-a84a-a0adaa783280-koku-1.csv.gz |                               |                               | "60f966ee59b92566e5698fdac46a03f1" |           1
  2 | 6d824bc9-41a1-4d72-b090-049e88309771-koku-1.csv.gz | 2019-02-19 21:59:22.673719+00 | 2019-02-19 21:59:18.027479+00 | "14f48cb9597bac9652504c0d6ba7b257" |           2
(2 rows)

postgres=# select * from api_provider;
 id |                 uuid                 |    name     | type | setup_complete | authentication_id | billing_source_id | created_by_id | customer_id
----+--------------------------------------+-------------+------+----------------+-------------------+-------------------+---------------+-------------
  1 | 3573d2f0-dc2f-44de-ab57-6a5848c21117 | ProviderAWS | AWS  | t              |                 1 |                 1 |             1 |           1
(1 row)

postgres=#

db logs:
db_1      | ERROR:  ON CONFLICT DO UPDATE command cannot affect row a second time
db_1      | HINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.
db_1      | STATEMENT:
db_1      |                 INSERT INTO reporting_awscostentrylineitem (invoice_id,usage_account_id,line_item_type,usage_start,usage_end,product_code,usage_type,operation,availability_zone,usage_amount,normalization_factor,normalized_usage_amount,currency_code,unblended_rate,unblended_cost,blended_rate,blended_cost,tax_type,public_on_demand_cost,public_on_demand_rate,reservation_amortized_upfront_cost_for_usage,reservation_amortized_upfront_fee,reservation_recurring_fee_for_usage,reservation_unused_quantity,reservation_unused_recurring_fee,tags,cost_entry_id,cost_entry_bill_id,cost_entry_product_id,cost_entry_pricing_id,cost_entry_reservation_id,hash)
db_1      |                     SELECT invoice_id,usage_account_id,line_item_type,usage_start,usage_end,product_code,usage_type,operation,availability_zone,usage_amount,normalization_factor,normalized_usage_amount,currency_code,unblended_rate,unblended_cost,blended_rate,blended_cost,tax_type,public_on_demand_cost,public_on_demand_rate,reservation_amortized_upfront_cost_for_usage,reservation_amortized_upfront_fee,reservation_recurring_fee_for_usage,reservation_unused_quantity,reservation_unused_recurring_fee,tags,cost_entry_id,cost_entry_bill_id,cost_entry_product_id,cost_entry_pricing_id,cost_entry_reservation_id,hash
db_1      |                     FROM reporting_awscostentrylineitem_d38c3efc_1784_40ad_9985_e8896dfbead9
db_1      |                     WHERE invoice_id IS NOT NULL
db_1      |                     ON CONFLICT (hash,cost_entry_id) DO UPDATE
db_1      |                     SET invoice_id = excluded.invoice_id,usage_account_id = excluded.usage_account_id,line_item_type = excluded.line_item_type,usage_start = excluded.usage_start,usage_end = excluded.usage_end,product_code = excluded.product_code,usage_type = excluded.usage_type,operation = excluded.operation,availability_zone = excluded.availability_zone,usage_amount = excluded.usage_amount,normalization_factor = excluded.normalization_factor,normalized_usage_amount = excluded.normalized_usage_amount,currency_code = excluded.currency_code,unblended_rate = excluded.unblended_rate,unblended_cost = excluded.unblended_cost,blended_rate = excluded.blended_rate,blended_cost = excluded.blended_cost,tax_type = excluded.tax_type,public_on_demand_cost = excluded.public_on_demand_cost,public_on_demand_rate = excluded.public_on_demand_rate,reservation_amortized_upfront_cost_for_usage = excluded.reservation_amortized_upfront_cost_for_usage,reservation_amortized_upfront_fee = excluded.reservation_amortized_upfront_fee,reservation_recurring_fee_for_usage = excluded.reservation_recurring_fee_for_usage,reservation_unused_quantity = excluded.reservation_unused_quantity,reservation_unused_recurring_fee = excluded.reservation_unused_recurring_fee,tags = excluded.tags,cost_entry_id = excluded.cost_entry_id,cost_entry_bill_id = excluded.cost_entry_bill_id,cost_entry_product_id = excluded.cost_entry_product_id,cost_entry_pricing_id = excluded.cost_entry_pricing_id,cost_entry_reservation_id = excluded.cost_entry_reservation_id,hash = excluded.hash
db_1      |
dccurtis commented 5 years ago

Just remembered this is has been seen before. We didn’t have the cost usage report setup correctly for finalization.