search

project-koku / korekuta

Read Only Please See: https://github.com/project-koku/korekuta-operator
https://github.com/project-koku/korekuta-operator
GNU Affero General Public License v3.0
6 stars 1 forks source link

Use manifest file uuid prefix for openshift reports to match masu expectation. #87

Closed chambridge closed 5 years ago

chambridge commented 5 years ago

@adberglund found a format mismatch that was causing OpenShift uploads to be skipped in masu:

OpenShift reports were expected to have the manifest uuid as a prefix, butkorekuta was passing the cluster_id. https://github.com/project-koku/korekuta/blob/master/roles/collect/tasks/main.yml#L52 https://github.com/project-koku/masu/blob/master/masu/external/downloader/ocp/ocp_report_downloader.py#L91

Listener processes files:

[2019-07-03 19:22:33,336] INFO in kafka_msg_handler: Successfully extracted OCP for 9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801
--
  | [2019-07-03 19:22:33,361] INFO in kafka_msg_handler: Validating message: b'{"request_id": "9dfd698b4a6e45aba8e3cc048e9f983e", "validation": "success"}'
  | [2019-07-03 19:22:33,365] INFO in kafka_msg_handler: Validating message complete.
  | [2019-07-03 19:22:34,098] INFO in kafka_msg_handler: Found provider_uuid: e123457b-4eea-41ff-9fe5-3ad12fa44ae8 for cluster_id: 9a2262dd-6be5-56a2-8fa0-ebfb44c44106
  | [2019-07-03 19:22:34,471] INFO in kafka_msg_handler: Processing report for account {'authentication': '9a2262dd-6be5-56a2-8fa0-ebfb44c44106', 'customer_name': 'acct0000000', 'billing_source': '', 'provider_type': 'OCP', 'schema_name': 'acct000000', 'provider_uuid': 'e123457b-4eea-41ff-9fe5-3ad12fa44ae8'}
  | [2019-07-03 19:22:35,106] INFO in report_downloader: Attempting to get OCP manifest for 2019-06-01 19:22:01.000001+00:00...
  | [2019-07-03 19:22:35,106] INFO in ocp_report_downloader: Looking for manifest at /var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190601-20190701
  | [2019-07-03 19:22:35,107] ERROR in common: Unable to extract manifest data: [Errno 2] No such file or directory: '/var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190601-20190701/manifest.json'
  | [2019-07-03 19:22:35,107] INFO in ocp_report_downloader: Looking for manifest at /var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190601-20190701
  | [2019-07-03 19:22:35,107] ERROR in common: Unable to extract manifest data: [Errno 2] No such file or directory: '/var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190601-20190701/manifest.json'
  | [2019-07-03 19:22:35,107] INFO in ocp_report_downloader: manifest found: {}
  | [2019-07-03 19:22:35,107] ERROR in ocp_report_downloader: Current UUID for report could not be found.
  | [2019-07-03 19:22:35,107] INFO in report_downloader: Attempting to get OCP manifest for 2019-07-01 19:22:01.000001+00:00...
  | [2019-07-03 19:22:35,107] INFO in ocp_report_downloader: Looking for manifest at /var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801
  | [2019-07-03 19:22:35,108] INFO in report_downloader_base: Inserting manifest database record for assembly_id: f3a2cf65-b9a2-50dc-9d37-91091697d458
  | [2019-07-03 19:22:35,467] INFO in report_downloader_base: No manifest entry found.  Adding for bill period start: 2019-07-01 00:00:00
  | [2019-07-03 19:22:35,478] INFO in ocp_report_downloader: Looking for manifest at /var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801
  | [2019-07-03 19:22:35,478] INFO in ocp_report_downloader: manifest found: {'files': ['f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv', 'f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv'], 'date': datetime.datetime(2019, 7, 3, 19, 21, 56, tzinfo=tzlocal()), 'uuid': 'f3a2cf65-b9a2-50dc-9d37-91091697d458', 'cluster_id': '9a2262dd-6be5-56a2-8fa0-ebfb44c44106', 'manifest_path': '/var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801/manifest.json'}
  | [2019-07-03 19:22:35,479] INFO in ocp_report_downloader: Found file /var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801/f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv
  | [2019-07-03 19:22:35,479] INFO in ocp_report_downloader: Found file /var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801/f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv
  | [2019-07-03 19:22:35,846] INFO in ocp_report_downloader: Downloading /var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801/f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv to /var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv
  | [2019-07-03 19:22:36,242] INFO in ocp_report_downloader: Downloading /var/tmp/insights_local/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801/f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv to /var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv
  | [2019-07-03 19:22:38,566] INFO in ocp_report_processor: Initialized report processor for file: /var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv and schema: acct6193296
  | [2019-07-03 19:22:39,581] INFO in ocp_report_processor: File <_io.TextIOWrapper name='/var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv' mode='r' encoding='UTF-8'> opened for processing
  | [2019-07-03 19:22:45,366] INFO in ocp_report_processor: Saving report rows 0 to 32063 for 20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv
  | [2019-07-03 19:22:45,367] INFO in ocp_report_processor: Completed report processing for file: /var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv and schema: acct6193296
  | [2019-07-03 19:22:46,150] INFO in ocp_report_processor: Cleaning up temporary report files for /var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106
  | [2019-07-03 19:22:48,821] INFO in ocp_report_processor: Initialized report processor for file: /var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv and schema: acct6193296
  | [2019-07-03 19:22:49,248] INFO in ocp_report_processor: File <_io.TextIOWrapper name='/var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv' mode='r' encoding='UTF-8'> opened for processing
  | [2019-07-03 19:22:50,371] INFO in ocp_report_processor: Saving report rows 0 to 723 for 20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv
  | [2019-07-03 19:22:50,373] INFO in ocp_report_processor: Completed report processing for file: /var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/20190701-20190801_f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv and schema: acct6193296
  | [2019-07-03 19:22:51,131] INFO in ocp_report_processor: Cleaning up temporary report files for /var/tmp/masu/acct6193296/ocp/9a2262dd-6be5-56a2-8fa0-ebfb44c44106
  | [2019-07-03 19:22:51,923] INFO in kafka_msg_handler: Processing complete for account {'authentication': '9a2262dd-6be5-56a2-8fa0-ebfb44c44106', 'customer_name': 'acct000000', 'billing_source': '', 'provider_type': 'OCP', 'schema_name': 'acct000000', 'provider_uuid': 'e123457b-4eea-41ff-9fe5-3ad12fa44ae8'}
  | [2019-07-03 19:22:51,935] INFO in kafka_msg_handler: Summarization celery uuid: 58237647-0ca1-48a8-ba66-3ab13009089f
  | [2019-07-03 19:22:51,935] INFO in kafka_msg_handler: Processing: {'files': ['f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.0.csv', 'f3a2cf65-b9a2-50dc-9d37-91091697d458_openshift_usage_report.1.csv'], 'date': datetime.datetime(2019, 7, 3, 19, 21, 56, tzinfo=tzlocal()), 'uuid': 'f3a2cf65-b9a2-50dc-9d37-91091697d458', 'cluster_id': '9a2262dd-6be5-56a2-8fa0-ebfb44c44106', 'manifest_path': '/var/tmp/masu/tmpi2ulg3iv/9a2262dd-6be5-56a2-8fa0-ebfb44c44106/manifest.json'} complete.

Worker summarizes:


[2019-07-03 19:22:51,934: INFO/MainProcess] Received task: masu.processor.tasks.summarize_reports[58237647-0ca1-48a8-ba66-3ab13009089f]
--
  | Received task: masu.processor.tasks.summarize_reports[58237647-0ca1-48a8-ba66-3ab13009089f]
  | [2019-07-03 19:22:51,936: ERROR/ForkPoolWorker-1] masu.processor.tasks.summarize_reports[58237647-0ca1-48a8-ba66-3ab13009089f]: report to summarize: {'start_date': '2019-07-01T19:22:01.000001Z', 'schema_name': 'acct00000', 'provider_type': 'OCP', 'provider_uuid': 'e123457b-4eea-41ff-9fe5-3ad12fa44ae8', 'manifest_id': 3140}
  | [2019-07-03 19:22:51,949: INFO/MainProcess] Received task: masu.processor.tasks.update_summary_tables[db05109b-c8d5-4909-960d-8b863211fe92]
  | Received task: masu.processor.tasks.update_summary_tables[db05109b-c8d5-4909-960d-8b863211fe92]
  | [2019-07-03 19:22:51,949: INFO/ForkPoolWorker-1] Task masu.processor.tasks.summarize_reports[58237647-0ca1-48a8-ba66-3ab13009089f] succeeded in 0.013994443928822875s: None
  | Task masu.processor.tasks.summarize_reports[58237647-0ca1-48a8-ba66-3ab13009089f] succeeded in 0.013994443928822875s: None
  | [2019-07-03 19:22:51,950: INFO/ForkPoolWorker-1] masu.processor.tasks.update_summary_tables[db05109b-c8d5-4909-960d-8b863211fe92]: update_summary_tables called with args:
  | schema_name: acct6193296,
  | provider: OCP,
  | start_date: 2019-07-01,
  | end_date: 2019-07-03,
  | manifest_id: 3140
  | [2019-07-03 19:22:54,718: WARNING/ForkPoolWorker-1] [2019-07-03 19:22:54,718] INFO in report_summary_updater: Starting report data summarization for provider uuid: e123457b-4eea-41ff-9fe5-3ad12fa44ae8.
  | [2019-07-03 19:22:54,718] INFO in report_summary_updater: Starting report data summarization for provider uuid: e123457b-4eea-41ff-9fe5-3ad12fa44ae8.
  | [2019-07-03 19:22:54,718: INFO/ForkPoolWorker-1] Starting report data summarization for provider uuid: e123457b-4eea-41ff-9fe5-3ad12fa44ae8.
  | Starting report data summarization for provider uuid: e123457b-4eea-41ff-9fe5-3ad12fa44ae8.
  | [2019-07-03 19:22:55,271: WARNING/ForkPoolWorker-1] [2019-07-03 19:22:55,271] INFO in ocp_report_summary_updater: Overriding start and end date to process full month.
  | [2019-07-03 19:22:55,271] INFO in ocp_report_summary_updater: Overriding start and end date to process full month.
  | [2019-07-03 19:22:55,271: INFO/ForkPoolWorker-1] Overriding start and end date to process full month.
  | Overriding start and end date to process full month.