rob-baron commented 10 months ago

Robert Bartlett Baron, reported 9 months ago So I was able to load data in to the staging instance of xdmod. When I went to load the same data into a more production ready instance of xdmod, no data is appearing in the interface.

No error appeared during shredding and ingesting, and running the queries from the previous ticket yeilds:

MariaDB [modw_cloud]> SELECT from_unixtime(min(event_time_ts)), from_unixtime(max(event_time_ts)) FROM modw_cloud.openstack_staging_event; +-----------------------------------+-----------------------------------+ | from_unixtime(min(event_time_ts)) | from_unixtime(max(event_time_ts)) | +-----------------------------------+-----------------------------------+ | 2022-01-25 16:58:46.000000 | 2022-11-09 20:47:07.269951 | +-----------------------------------+-----------------------------------+ 1 row in set (0.00 sec)

MariaDB [modw_cloud]> SELECT from_unixtime(min(event_time_ts)), from_unixtime(max(event_time_ts)) FROM modw_cloud.event; +-----------------------------------+-----------------------------------+ | from_unixtime(min(event_time_ts)) | from_unixtime(max(event_time_ts)) | +-----------------------------------+-----------------------------------+ | 2018-04-17 19:23:00.418847 | 2022-11-09 20:47:07.269951 | +-----------------------------------+-----------------------------------+ 1 row in set (0.06 sec)

MariaDB [modw_cloud]> SELECT min(start_time), max(end_time) FROM modw_cloud.session_records; +---------------------+---------------------+ | min(start_time) | max(end_time) | +---------------------+---------------------+ | 2018-04-18 14:58:38 | 2018-04-30 23:00:54 | +---------------------+---------------------+ 1 row in set (0.12 sec)

Which, if I am interpreting this correctly, indicates that the data has made it into the aggregation tables.

Any suggestions? Gregary Dean , said 9 months ago Cc: msd@bu.edu, joachimw@bu.edu Ticket: https://help.xdmod.org/support/tickets/31579

Hi Rob,

It looks like your data is not making it into the tables used for populating the aggregate tables, specifically the modw.session_records table. The second column in the second and third queries should be much closer to each other. I think the results of these queries are similar to those in the previous ticket where you did not have any compute events in your log files.

Can you re-run your ingestion and aggregation commands and send me the commands you ran and the output from the logs? Thanks, Greg Robert Bartlett Baron , said 9 months ago Greg,

Am in the process of reloading the data on to that instance, though I can confirm that I am having the same issue. When this is finished I am planning to run:

xdmod-ingestor --last-modified-start-date "2018-01-01 12:30:00"

I am assuming this is what you mean by the ingestion command. I am not certain what you mean by the aggregation command. Can you clarify?

Rob.

Robert Bartlett Baron , said 9 months ago Here is the shredding command and log:

sh-4.2$ time xdmod-shredder --debug -f openstack -d /root/xdmod_data/nerc_openstack -r nerc_openstack 2022-11-18 14:09:53 [info] Command: '/usr/bin/xdmod-shredder' '--debug' '-f' 'openstack' '-d' '/root/xdmod_data/nerc_openstack' '-r' 'nerc_openstack' 2022-11-18 14:09:54 [debug] Checking for required arguments 2022-11-18 14:09:54 [notice] xdmod-shredder start (process_start_time: 2022-11-18 14:09:54) 2022-11-18 14:09:58 [debug] Creating database connection 2022-11-18 14:09:58 [debug] Creating shredder for format 'openstack' 2022-11-18 14:09:58 [debug] Setting resource to 'nerc_openstack' 2022-11-18 14:09:58 [debug] Loading configuration file /etc/xdmod/resources.json 2022-11-18 14:09:58 [debug] Parsing /etc/xdmod/resources.json 2022-11-18 14:09:58 [debug] Parsed 1 records 2022-11-18 14:09:58 [debug] Stored object Configuration\XdmodConfiguration (/etc/xdmod/resources.json) in APCu cache with key Configuration\XdmodConfiguration|/etc/xdmod/resources.json|40cd750bba9870f18aada2478b24840a in 0.001157s 2022-11-18 14:09:58 [debug] Running ETL pipeline "jobs-common, ingest-organizations, ingest-resource-types, ingest-resources, jobs-cloud-common" with parameters [] 2022-11-18 14:09:58 [debug] Loading configuration file /etc/xdmod/etl/etl.json 2022-11-18 14:09:58 [debug] Parsing /etc/xdmod/etl/etl.json 2022-11-18 14:09:58 [debug] Parsed 1 records 2022-11-18 14:09:58 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-import.json 2022-11-18 14:09:58 [debug] Parsing /etc/xdmod/etl/etl.d/acls-import.json 2022-11-18 14:09:58 [debug] Parsed 1 records 2022-11-18 14:09:58 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-11-18 14:09:58 [debug] Parsing /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-11-18 14:09:58 [debug] Parsed 1 records 2022-11-18 14:09:58 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/action_state_setup.json 2022-11-18 14:09:58 [debug] Parsing /etc/xdmod/etl/etl.d/action_state_setup.json 2022-11-18 14:09:58 [debug] Parsed 1 records 2022-11-18 14:09:58 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-11-18 14:09:58 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-11-18 14:09:58 [debug] Parsed 1 records 2022-11-18 14:09:58 [info] ETL\DataEndpoint\DirectoryScanner (name=Open Stack resource specifications, path=${CLOUD_RESOURCE_SPECS_DIRECTORY}): Relative path provided, absolute path recommended 2022-11-18 14:10:01 [debug] Qualifying relative path ${CLOUD_RESOURCE_SPECS_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-11-18 14:10:01 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-11-18 14:10:01 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-11-18 14:10:01 [debug] Parsed 1 records 2022-11-18 14:10:01 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/gateways.json 2022-11-18 14:10:01 [debug] Parsing /etc/xdmod/etl/etl.d/gateways.json 2022-11-18 14:10:01 [debug] Parsed 1 records 2022-11-18 14:10:01 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-11-18 14:10:01 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-11-18 14:10:01 [debug] Parsed 1 records 2022-11-18 14:10:01 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb.json 2022-11-18 14:10:01 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb.json 2022-11-18 14:10:01 [debug] Parsed 1 records 2022-11-18 14:10:01 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/ingest_resources.json 2022-11-18 14:10:01 [debug] Parsing /etc/xdmod/etl/etl.d/ingest_resources.json 2022-11-18 14:10:01 [debug] Parsed 1 records 2022-11-18 14:10:01 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs.json 2022-11-18 14:10:01 [debug] Parsing /etc/xdmod/etl/etl.d/jobs.json 2022-11-18 14:10:01 [debug] Parsed 1 records 2022-11-18 14:10:01 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-11-18 14:10:01 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-11-18 14:10:01 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-11-18 14:10:01 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-11-18 14:10:01 [debug] Parsed 1 records 2022-11-18 14:10:01 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-11-18 14:10:01 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-11-18 14:10:01 [debug] Parsed 1 records 2022-11-18 14:10:01 [info] ETL\DataEndpoint\DirectoryScanner (name=Generic cloud event logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-11-18 14:10:04 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-11-18 14:10:04 [info] ETL\DataEndpoint\DirectoryScanner (name=Generic volume logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-11-18 14:10:04 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-11-18 14:10:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-11-18 14:10:04 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-11-18 14:10:04 [debug] Parsed 1 records 2022-11-18 14:10:04 [info] ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-11-18 14:10:06 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-11-18 14:10:06 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_common.json 2022-11-18 14:10:06 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_common.json 2022-11-18 14:10:06 [debug] Parsed 1 records 2022-11-18 14:10:06 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/organizations.json 2022-11-18 14:10:06 [debug] Parsing /etc/xdmod/etl/etl.d/organizations.json 2022-11-18 14:10:06 [debug] Parsed 1 records 2022-11-18 14:10:06 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/resource_types.json 2022-11-18 14:10:06 [debug] Parsing /etc/xdmod/etl/etl.d/resource_types.json 2022-11-18 14:10:06 [debug] Parsed 1 records 2022-11-18 14:10:06 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/shredder.json 2022-11-18 14:10:06 [debug] Parsing /etc/xdmod/etl/etl.d/shredder.json 2022-11-18 14:10:06 [debug] Parsed 1 records 2022-11-18 14:10:06 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/staging.json 2022-11-18 14:10:06 [debug] Parsing /etc/xdmod/etl/etl.d/staging.json 2022-11-18 14:10:06 [debug] Parsed 1 records 2022-11-18 14:10:06 [info] ETL\DataEndpoint\DirectoryScanner (name=usage-directory, path=${STORAGE_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-11-18 14:10:08 [debug] Qualifying relative path ${STORAGE_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-11-18 14:10:08 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/storage.json 2022-11-18 14:10:08 [debug] Parsing /etc/xdmod/etl/etl.d/storage.json 2022-11-18 14:10:08 [debug] Parsed 1 records 2022-11-18 14:10:08 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/test_suite.json 2022-11-18 14:10:08 [debug] Parsing /etc/xdmod/etl/etl.d/test_suite.json 2022-11-18 14:10:08 [debug] Parsed 1 records 2022-11-18 14:10:08 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/verify.json 2022-11-18 14:10:08 [debug] Parsing /etc/xdmod/etl/etl.d/verify.json 2022-11-18 14:10:08 [debug] Parsed 1 records 2022-11-18 14:10:08 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdb.json 2022-11-18 14:10:08 [debug] Parsing /etc/xdmod/etl/etl.d/xdb.json 2022-11-18 14:10:08 [debug] Parsed 1 records 2022-11-18 14:10:08 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-11-18 14:10:08 [debug] Parsing /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-11-18 14:10:08 [debug] Parsed 1 records 2022-11-18 14:10:08 [debug] Stored object ETL\Configuration\EtlConfiguration (/etc/xdmod/etl/etl.json) in APCu cache with key ETL\Configuration\EtlConfiguration|/etc/xdmod/etl/etl.json|cb328615c5b43bfbe1404ea2dbb0a7fc in 9.698471s 2022-11-18 14:10:08 [debug] Running ETL pipeline with script options {"default-module-name":"xdmod","process-sections":["jobs-common","ingest-organizations","ingest-resource-types","ingest-resources","jobs-cloud-common"]} 2022-11-18 14:10:08 [info] Verifying endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:10:12 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Countable types, path=/etc/xdmod/etl/etl_data.d/jobs/countable_type.json) 2022-11-18 14:10:14 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Job record types, path=/etc/xdmod/etl/etl_data.d/jobs/job_record_type.json) 2022-11-18 14:10:15 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Job task types, path=/etc/xdmod/etl/etl_data.d/jobs/job_task_type.json) 2022-11-18 14:10:17 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Units of measure, path=/etc/xdmod/etl/etl_data.d/jobs/unit.json) 2022-11-18 14:10:20 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Submission venues, path=/etc/xdmod/etl/etl_data.d/jobs/submission_venue.json) 2022-11-18 14:10:22 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Organization configuration, path=/etc/xdmod/etl/../organization.json, 1 filters) 2022-11-18 14:10:25 [info] Verifying endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:10:29 [info] Verifying endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:10:33 [info] Verifying endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:10:34 [info] Verifying endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:10:36 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Resource types, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-11-18 14:10:37 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Resources configuration, path=/etc/xdmod/etl/../resources.json) 2022-11-18 14:10:38 [info] Verifying endpoint: ETL\DataEndpoint\ConfigurationFileEndpoint (name=Resource Types configuration, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-11-18 14:10:39 [info] Verifying endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-11-18 14:10:42 [info] Verifying endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:10:44 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud asset types, path=/etc/xdmod/etl/etl_data.d/cloud_common/asset_type.json) 2022-11-18 14:10:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud event types, path=/etc/xdmod/etl/etl_data.d/cloud_common/event_type.json) 2022-11-18 14:10:49 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/record_type.json) 2022-11-18 14:10:50 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/host.json) 2022-11-18 14:10:54 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/account.json) 2022-11-18 14:10:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance.json) 2022-11-18 14:10:57 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance_type.json) 2022-11-18 14:10:58 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/image.json) 2022-11-18 14:10:59 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=memory buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/memory_buckets.json) 2022-11-18 14:11:00 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=processor buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/processor_buckets.json) 2022-11-18 14:11:01 [info] Create action xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables) 2022-11-18 14:11:02 [info] Verifying action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables) 2022-11-18 14:11:04 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:06 [info] Source endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:08 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:10 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json' 2022-11-18 14:11:11 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json 2022-11-18 14:11:11 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json 2022-11-18 14:11:11 [debug] Parsed 1 records 2022-11-18 14:11:11 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json|40cd750bba9870f18aada2478b24840a in 0.001240s 2022-11-18 14:11:11 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json' 2022-11-18 14:11:13 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json 2022-11-18 14:11:13 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json 2022-11-18 14:11:13 [debug] Parsed 1 records 2022-11-18 14:11:13 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json|40cd750bba9870f18aada2478b24840a in 0.001448s 2022-11-18 14:11:13 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json' 2022-11-18 14:11:14 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/meta_job.json 2022-11-18 14:11:14 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/meta_job.json 2022-11-18 14:11:14 [debug] Parsed 1 records 2022-11-18 14:11:14 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json|40cd750bba9870f18aada2478b24840a in 0.001303s 2022-11-18 14:11:14 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json' 2022-11-18 14:11:15 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json 2022-11-18 14:11:15 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json 2022-11-18 14:11:15 [debug] Parsed 1 records 2022-11-18 14:11:15 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json|40cd750bba9870f18aada2478b24840a in 0.001681s 2022-11-18 14:11:15 [info] Create action xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:16 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json 2022-11-18 14:11:16 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json 2022-11-18 14:11:16 [debug] Parsed 1 records 2022-11-18 14:11:16 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/countable_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/countable_type.json' 2022-11-18 14:11:16 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001361s 2022-11-18 14:11:16 [info] Verifying action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:16 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:17 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Countable types, path=/etc/xdmod/etl/etl_data.d/jobs/countable_type.json) 2022-11-18 14:11:18 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:18 [debug] Created ETL destination table object for table definition key 'countable_type' 2022-11-18 14:11:18 [info] Create action xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:18 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json 2022-11-18 14:11:18 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json 2022-11-18 14:11:18 [debug] Parsed 1 records 2022-11-18 14:11:18 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/job_record_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/job_record_type.json' 2022-11-18 14:11:18 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001796s 2022-11-18 14:11:18 [info] Verifying action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:19 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:19 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Job record types, path=/etc/xdmod/etl/etl_data.d/jobs/job_record_type.json) 2022-11-18 14:11:21 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:21 [debug] Created ETL destination table object for table definition key 'job_record_type' 2022-11-18 14:11:21 [info] Create action xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:22 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json 2022-11-18 14:11:22 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json 2022-11-18 14:11:22 [debug] Parsed 1 records 2022-11-18 14:11:22 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/job_task_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/job_task_type.json' 2022-11-18 14:11:22 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001915s 2022-11-18 14:11:22 [info] Verifying action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:22 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:23 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Job task types, path=/etc/xdmod/etl/etl_data.d/jobs/job_task_type.json) 2022-11-18 14:11:23 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:24 [debug] Created ETL destination table object for table definition key 'job_task_type' 2022-11-18 14:11:24 [info] Create action xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:25 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/unit.json 2022-11-18 14:11:25 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/unit.json 2022-11-18 14:11:25 [debug] Parsed 1 records 2022-11-18 14:11:25 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/unit.json' to '/etc/xdmod/etl/etl_tables.d/jobs/unit.json' 2022-11-18 14:11:25 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/unit.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/unit.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001837s 2022-11-18 14:11:25 [info] Verifying action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:25 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:26 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Units of measure, path=/etc/xdmod/etl/etl_data.d/jobs/unit.json) 2022-11-18 14:11:26 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:26 [debug] Created ETL destination table object for table definition key 'unit' 2022-11-18 14:11:26 [info] Create action xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:27 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json 2022-11-18 14:11:27 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json 2022-11-18 14:11:27 [debug] Parsed 1 records 2022-11-18 14:11:27 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/submission_venue.json' to '/etc/xdmod/etl/etl_tables.d/jobs/submission_venue.json' 2022-11-18 14:11:27 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002215s 2022-11-18 14:11:27 [info] Verifying action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:27 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:27 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Submission venues, path=/etc/xdmod/etl/etl_data.d/jobs/submission_venue.json) 2022-11-18 14:11:27 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:28 [debug] Created ETL destination table object for table definition key 'submission_venue' 2022-11-18 14:11:28 [info] Create action xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:28 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json 2022-11-18 14:11:28 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json 2022-11-18 14:11:28 [debug] Parsed 1 records 2022-11-18 14:11:28 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/organization.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/organization.json' 2022-11-18 14:11:28 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001534s 2022-11-18 14:11:28 [info] Verifying action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:28 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:29 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Organization configuration, path=/etc/xdmod/etl/../organization.json, 1 filters) 2022-11-18 14:11:29 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:30 [debug] Created ETL destination table object for table definition key 'staging_organization' 2022-11-18 14:11:30 [info] Create action xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:30 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json 2022-11-18 14:11:30 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json 2022-11-18 14:11:30 [debug] Parsed 1 records 2022-11-18 14:11:30 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/organizations.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/organizations.json' 2022-11-18 14:11:30 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002181s 2022-11-18 14:11:30 [info] Verifying action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:30 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:31 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:31 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:32 [debug] Created ETL destination table object for table definition key 'hpcdb_organizations' 2022-11-18 14:11:32 [debug] Create ETL source query object 2022-11-18 14:11:32 [debug] No destination_field_map specified 2022-11-18 14:11:32 [debug] Auto-generating destination_field_map using 3 source fields: organization_id, organization_name, organization_abbrev 2022-11-18 14:11:32 [debug] Available fields for table key 'hpcdb_organizations': organization_id, organization_abbrev, organization_name 2022-11-18 14:11:32 [debug] Generated destination_field_map: Table: hpcdb_organizations organization_id -> organization_id organization_abbrev -> organization_abbrev organization_name -> organization_name 2022-11-18 14:11:32 [info] Create action xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:32 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json 2022-11-18 14:11:32 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json 2022-11-18 14:11:32 [debug] Parsed 1 records 2022-11-18 14:11:32 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/organization.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/organization.json' 2022-11-18 14:11:32 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003177s 2022-11-18 14:11:32 [info] Verifying action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:32 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:32 [info] Source endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:32 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:35 [debug] Created ETL destination table object for table definition key 'organization' 2022-11-18 14:11:35 [debug] Create ETL source query object 2022-11-18 14:11:35 [debug] No destination_field_map specified 2022-11-18 14:11:35 [debug] Auto-generating destination_field_map using 7 source fields: id, abbrev, name, short_name, long_name, order_id, organization_origin_id 2022-11-18 14:11:35 [debug] Available fields for table key 'organization': id, organizationtype_id, abbrev, name, url, phone, nsf_org_code, is_reconciled, amie_name, country_id, state_id, latitude, longitude, short_name, long_name, federation_instance_id, organization_origin_id, order_id 2022-11-18 14:11:35 [debug] Generated destination_field_map: Table: organization id -> id abbrev -> abbrev name -> name short_name -> short_name long_name -> long_name organization_origin_id -> organization_origin_id order_id -> order_id 2022-11-18 14:11:35 [info] Create action xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables) 2022-11-18 14:11:37 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables) 2022-11-18 14:11:37 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:38 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:38 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:39 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' 2022-11-18 14:11:39 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json 2022-11-18 14:11:39 [debug] Parsing /etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json 2022-11-18 14:11:39 [debug] Parsed 1 records 2022-11-18 14:11:39 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json|40cd750bba9870f18aada2478b24840a in 0.001724s 2022-11-18 14:11:39 [info] Create action xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql) 2022-11-18 14:11:39 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql) 2022-11-18 14:11:39 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:40 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:40 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:40 [info] Create action xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:40 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json 2022-11-18 14:11:40 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json 2022-11-18 14:11:40 [debug] Parsed 1 records 2022-11-18 14:11:40 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' 2022-11-18 14:11:40 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001762s 2022-11-18 14:11:40 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:41 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:42 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Resource types, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-11-18 14:11:43 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:44 [debug] Created ETL destination table object for table definition key 'staging_resource_type' 2022-11-18 14:11:44 [info] Create action xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:44 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json 2022-11-18 14:11:44 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json 2022-11-18 14:11:44 [debug] Parsed 1 records 2022-11-18 14:11:44 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-types.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-types.json' 2022-11-18 14:11:44 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002065s 2022-11-18 14:11:44 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:45 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:45 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:45 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:45 [debug] Created ETL destination table object for table definition key 'hpcdb_resource_types' 2022-11-18 14:11:45 [debug] Create ETL source query object 2022-11-18 14:11:45 [debug] No destination_field_map specified 2022-11-18 14:11:45 [debug] Auto-generating destination_field_map using 3 source fields: type_id, type_abbr, type_desc 2022-11-18 14:11:45 [debug] Available fields for table key 'hpcdb_resource_types': type_id, type_abbr, type_desc 2022-11-18 14:11:45 [debug] Generated destination_field_map: Table: hpcdb_resource_types type_id -> type_id type_abbr -> type_abbr type_desc -> type_desc 2022-11-18 14:11:45 [info] Create action xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:45 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json 2022-11-18 14:11:45 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json 2022-11-18 14:11:45 [debug] Parsed 1 records 2022-11-18 14:11:45 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-type.json' 2022-11-18 14:11:45 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001964s 2022-11-18 14:11:45 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:46 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:46 [info] Source endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:46 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:46 [debug] Created ETL destination table object for table definition key 'resourcetype' 2022-11-18 14:11:46 [debug] Create ETL source query object 2022-11-18 14:11:46 [debug] No destination_field_map specified 2022-11-18 14:11:46 [debug] Auto-generating destination_field_map using 3 source fields: id, description, abbrev 2022-11-18 14:11:46 [debug] Available fields for table key 'resourcetype': id, description, abbrev 2022-11-18 14:11:46 [debug] Generated destination_field_map: Table: resourcetype id -> id description -> description abbrev -> abbrev 2022-11-18 14:11:46 [info] Create action xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json 2022-11-18 14:11:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json 2022-11-18 14:11:47 [debug] Parsed 1 records 2022-11-18 14:11:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-config.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-config.json' 2022-11-18 14:11:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002136s 2022-11-18 14:11:47 [info] Verifying action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:49 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Resources configuration, path=/etc/xdmod/etl/../resources.json) 2022-11-18 14:11:49 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:50 [debug] Created ETL destination table object for table definition key 'staging_resource_config' 2022-11-18 14:11:50 [info] Create action xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:50 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json 2022-11-18 14:11:50 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json 2022-11-18 14:11:50 [debug] Parsed 1 records 2022-11-18 14:11:50 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource.json' 2022-11-18 14:11:50 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001694s 2022-11-18 14:11:50 [info] Verifying action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:50 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:51 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:51 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:51 [debug] Created ETL destination table object for table definition key 'staging_resource' 2022-11-18 14:11:51 [debug] Create ETL source query object 2022-11-18 14:11:51 [debug] No destination_field_map specified 2022-11-18 14:11:51 [debug] Auto-generating destination_field_map using 1 source fields: resource_name 2022-11-18 14:11:51 [debug] Available fields for table key 'staging_resource': resource_id, resource_name 2022-11-18 14:11:51 [debug] Generated destination_field_map: Table: staging_resource resource_name -> resource_name 2022-11-18 14:11:51 [info] Create action xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:51 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json 2022-11-18 14:11:51 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json 2022-11-18 14:11:51 [debug] Parsed 1 records 2022-11-18 14:11:51 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resources.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resources.json' 2022-11-18 14:11:51 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003046s 2022-11-18 14:11:51 [info] Verifying action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:52 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:52 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:53 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:53 [debug] Created ETL destination table object for table definition key 'hpcdb_resources' 2022-11-18 14:11:53 [debug] Create ETL source query object 2022-11-18 14:11:53 [debug] No destination_field_map specified 2022-11-18 14:11:53 [debug] Auto-generating destination_field_map using 8 source fields: resource_id, resource_type_id, organization_id, resource_name, resource_code, resource_description, resource_shared_jobs, resource_timezone 2022-11-18 14:11:53 [debug] Available fields for table key 'hpcdb_resources': resource_id, resource_type_id, organization_id, resource_name, resource_code, resource_description, resource_shared_jobs, resource_timezone 2022-11-18 14:11:53 [debug] Generated destination_field_map: Table: hpcdb_resources resource_id -> resource_id resource_type_id -> resource_type_id organization_id -> organization_id resource_name -> resource_name resource_code -> resource_code resource_description -> resource_description resource_shared_jobs -> resource_shared_jobs resource_timezone -> resource_timezone 2022-11-18 14:11:53 [info] Create action xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:53 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json 2022-11-18 14:11:53 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json 2022-11-18 14:11:53 [debug] Parsed 1 records 2022-11-18 14:11:53 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-fact.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-fact.json' 2022-11-18 14:11:53 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002804s 2022-11-18 14:11:53 [info] Verifying action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:54 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:55 [info] Source endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:55 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:56 [debug] Created ETL destination table object for table definition key 'resourcefact' 2022-11-18 14:11:56 [debug] Create ETL source query object 2022-11-18 14:11:56 [debug] No destination_field_map specified 2022-11-18 14:11:56 [debug] Auto-generating destination_field_map using 9 source fields: id, resourcetype_id, organization_id, name, code, description, shared_jobs, timezone, resource_origin_id 2022-11-18 14:11:56 [debug] Available fields for table key 'resourcefact': id, resourcetype_id, organization_id, name, code, description, start_date, start_date_ts, end_date, end_date_ts, shared_jobs, timezone, resource_origin_id 2022-11-18 14:11:56 [debug] Generated destination_field_map: Table: resourcefact id -> id resourcetype_id -> resourcetype_id organization_id -> organization_id name -> name code -> code description -> description shared_jobs -> shared_jobs timezone -> timezone resource_origin_id -> resource_origin_id 2022-11-18 14:11:56 [info] Create action xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:57 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json 2022-11-18 14:11:57 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json 2022-11-18 14:11:57 [debug] Parsed 1 records 2022-11-18 14:11:57 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type-realms.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type-realms.json' 2022-11-18 14:11:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002848s 2022-11-18 14:11:57 [info] Verifying action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:11:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:57 [info] Source endpoint: ETL\DataEndpoint\ConfigurationFileEndpoint (name=Resource Types configuration, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-11-18 14:11:58 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:58 [debug] Created ETL destination table object for table definition key 'staging_resource_type_realms' 2022-11-18 14:11:58 [info] Create action xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/acls/realms.json 2022-11-18 14:11:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/acls/realms.json 2022-11-18 14:11:58 [debug] Parsed 1 records 2022-11-18 14:11:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/acls/realms.json' to '/etc/xdmod/etl/etl_tables.d/acls/realms.json' 2022-11-18 14:11:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/acls/realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/acls/realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002437s 2022-11-18 14:11:58 [info] Verifying action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:11:59 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:11:59 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:00 [info] Destination endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:00 [debug] Created ETL destination table object for table definition key 'realms' 2022-11-18 14:12:00 [debug] Create ETL source query object 2022-11-18 14:12:00 [debug] No destination_field_map specified 2022-11-18 14:12:00 [debug] Auto-generating destination_field_map using 3 source fields: display, name, module_id 2022-11-18 14:12:00 [debug] Available fields for table key 'realms': realm_id, module_id, name, display 2022-11-18 14:12:00 [debug] Generated destination_field_map: Table: realms module_id -> module_id name -> name display -> display 2022-11-18 14:12:00 [info] Create action xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:12:01 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json 2022-11-18 14:12:01 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json 2022-11-18 14:12:01 [debug] Parsed 1 records 2022-11-18 14:12:01 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-type-realms.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-type-realms.json' 2022-11-18 14:12:01 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001992s 2022-11-18 14:12:01 [info] Verifying action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor) 2022-11-18 14:12:01 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:02 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:02 [info] Destination endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:03 [debug] Created ETL destination table object for table definition key 'resource_type_realms' 2022-11-18 14:12:03 [debug] Create ETL source query object 2022-11-18 14:12:03 [debug] No destination_field_map specified 2022-11-18 14:12:03 [debug] Auto-generating destination_field_map using 2 source fields: resource_type_id, realm_id 2022-11-18 14:12:03 [debug] Available fields for table key 'resource_type_realms': resource_type_realm_id, resource_type_id, realm_id 2022-11-18 14:12:03 [debug] Generated destination_field_map: Table: resource_type_realms resource_type_id -> resource_type_id realm_id -> realm_id 2022-11-18 14:12:03 [info] Create action xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables) 2022-11-18 14:12:03 [info] Verifying action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables) 2022-11-18 14:12:03 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:03 [info] Source endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:03 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:03 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/account.json' 2022-11-18 14:12:03 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/account.json 2022-11-18 14:12:03 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/account.json 2022-11-18 14:12:03 [debug] Parsed 1 records 2022-11-18 14:12:03 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/account.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/account.json|40cd750bba9870f18aada2478b24840a in 0.001847s 2022-11-18 14:12:03 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/event.json' 2022-11-18 14:12:03 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/event.json 2022-11-18 14:12:03 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/event.json 2022-11-18 14:12:03 [debug] Parsed 1 records 2022-11-18 14:12:03 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/event.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/event.json|40cd750bba9870f18aada2478b24840a in 0.001891s 2022-11-18 14:12:03 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json' 2022-11-18 14:12:03 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/asset.json 2022-11-18 14:12:03 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/asset.json 2022-11-18 14:12:03 [debug] Parsed 1 records 2022-11-18 14:12:03 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json|40cd750bba9870f18aada2478b24840a in 0.001778s 2022-11-18 14:12:03 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/host.json' 2022-11-18 14:12:04 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/host.json 2022-11-18 14:12:04 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/host.json 2022-11-18 14:12:04 [debug] Parsed 1 records 2022-11-18 14:12:04 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/host.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/host.json|40cd750bba9870f18aada2478b24840a in 0.001393s 2022-11-18 14:12:04 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json' 2022-11-18 14:12:05 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json 2022-11-18 14:12:05 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json 2022-11-18 14:12:05 [debug] Parsed 1 records 2022-11-18 14:12:05 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json|40cd750bba9870f18aada2478b24840a in 0.001598s 2022-11-18 14:12:05 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json' 2022-11-18 14:12:05 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json 2022-11-18 14:12:05 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json 2022-11-18 14:12:05 [debug] Parsed 1 records 2022-11-18 14:12:05 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json|40cd750bba9870f18aada2478b24840a in 0.001390s 2022-11-18 14:12:05 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json' 2022-11-18 14:12:05 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json 2022-11-18 14:12:05 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json 2022-11-18 14:12:05 [debug] Parsed 1 records 2022-11-18 14:12:05 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json|40cd750bba9870f18aada2478b24840a in 0.001141s 2022-11-18 14:12:05 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json' 2022-11-18 14:12:05 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json 2022-11-18 14:12:05 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json 2022-11-18 14:12:05 [debug] Parsed 1 records 2022-11-18 14:12:05 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json|40cd750bba9870f18aada2478b24840a in 0.001318s 2022-11-18 14:12:05 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json' 2022-11-18 14:12:05 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json 2022-11-18 14:12:05 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json 2022-11-18 14:12:05 [debug] Parsed 1 records 2022-11-18 14:12:05 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json|40cd750bba9870f18aada2478b24840a in 0.001599s 2022-11-18 14:12:05 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json' 2022-11-18 14:12:06 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json 2022-11-18 14:12:06 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json 2022-11-18 14:12:06 [debug] Parsed 1 records 2022-11-18 14:12:06 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json|40cd750bba9870f18aada2478b24840a in 0.002233s 2022-11-18 14:12:06 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json' 2022-11-18 14:12:06 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json 2022-11-18 14:12:06 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json 2022-11-18 14:12:06 [debug] Parsed 1 records 2022-11-18 14:12:06 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json|40cd750bba9870f18aada2478b24840a in 0.001598s 2022-11-18 14:12:06 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json' 2022-11-18 14:12:06 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json

rob-baron commented 10 months ago

2022-11-18 14:12:06 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json 2022-11-18 14:12:06 [debug] Parsed 1 records 2022-11-18 14:12:06 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json|40cd750bba9870f18aada2478b24840a in 0.001726s 2022-11-18 14:12:06 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json' 2022-11-18 14:12:06 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json 2022-11-18 14:12:06 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json 2022-11-18 14:12:06 [debug] Parsed 1 records 2022-11-18 14:12:06 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json|40cd750bba9870f18aada2478b24840a in 0.001639s 2022-11-18 14:12:06 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json' 2022-11-18 14:12:06 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json 2022-11-18 14:12:06 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json 2022-11-18 14:12:06 [debug] Parsed 1 records 2022-11-18 14:12:06 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json|40cd750bba9870f18aada2478b24840a in 0.001405s 2022-11-18 14:12:06 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json' 2022-11-18 14:12:06 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json 2022-11-18 14:12:06 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json 2022-11-18 14:12:06 [debug] Parsed 1 records 2022-11-18 14:12:06 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json|40cd750bba9870f18aada2478b24840a in 0.001805s 2022-11-18 14:12:06 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/domains.json' 2022-11-18 14:12:07 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/domains.json 2022-11-18 14:12:07 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/domains.json 2022-11-18 14:12:07 [debug] Parsed 1 records 2022-11-18 14:12:07 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/domains.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/domains.json|40cd750bba9870f18aada2478b24840a in 0.001523s 2022-11-18 14:12:07 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json' 2022-11-18 14:12:07 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json 2022-11-18 14:12:07 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json 2022-11-18 14:12:07 [debug] Parsed 1 records 2022-11-18 14:12:07 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json|40cd750bba9870f18aada2478b24840a in 0.001312s 2022-11-18 14:12:07 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json' 2022-11-18 14:12:07 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json 2022-11-18 14:12:07 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json 2022-11-18 14:12:07 [debug] Parsed 1 records 2022-11-18 14:12:07 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json|40cd750bba9870f18aada2478b24840a in 0.001226s 2022-11-18 14:12:07 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json' 2022-11-18 14:12:07 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json 2022-11-18 14:12:07 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json 2022-11-18 14:12:07 [debug] Parsed 1 records 2022-11-18 14:12:07 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json|40cd750bba9870f18aada2478b24840a in 0.001145s 2022-11-18 14:12:07 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json' 2022-11-18 14:12:07 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json 2022-11-18 14:12:07 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json 2022-11-18 14:12:07 [debug] Parsed 1 records 2022-11-18 14:12:07 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json|40cd750bba9870f18aada2478b24840a in 0.002214s 2022-11-18 14:12:07 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json' 2022-11-18 14:12:08 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json 2022-11-18 14:12:08 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json 2022-11-18 14:12:08 [debug] Parsed 1 records 2022-11-18 14:12:08 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json|40cd750bba9870f18aada2478b24840a in 0.001376s 2022-11-18 14:12:08 [info] Create action xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:08 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/asset_type.json 2022-11-18 14:12:08 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/asset_type.json 2022-11-18 14:12:08 [debug] Parsed 1 records 2022-11-18 14:12:08 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/asset_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/asset_type.json' 2022-11-18 14:12:08 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/asset_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/asset_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001826s 2022-11-18 14:12:08 [info] Verifying action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:08 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:08 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud asset types, path=/etc/xdmod/etl/etl_data.d/cloud_common/asset_type.json) 2022-11-18 14:12:08 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:08 [debug] Created ETL destination table object for table definition key 'asset_type' 2022-11-18 14:12:08 [info] Create action xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:09 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/event_type.json 2022-11-18 14:12:09 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/event_type.json 2022-11-18 14:12:09 [debug] Parsed 1 records 2022-11-18 14:12:09 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/event_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/event_type.json' 2022-11-18 14:12:09 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/event_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/event_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001305s 2022-11-18 14:12:09 [info] Verifying action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:09 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:09 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud event types, path=/etc/xdmod/etl/etl_data.d/cloud_common/event_type.json) 2022-11-18 14:12:09 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:09 [debug] Created ETL destination table object for table definition key 'event_type' 2022-11-18 14:12:09 [info] Create action xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:10 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/record_type.json 2022-11-18 14:12:10 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/record_type.json 2022-11-18 14:12:10 [debug] Parsed 1 records 2022-11-18 14:12:10 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/record_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/record_type.json' 2022-11-18 14:12:10 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/record_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/record_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001835s 2022-11-18 14:12:10 [info] Verifying action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:10 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:10 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/record_type.json) 2022-11-18 14:12:10 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:10 [debug] Created ETL destination table object for table definition key 'record_type' 2022-11-18 14:12:10 [info] Create action xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:11 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/host.json 2022-11-18 14:12:11 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/host.json 2022-11-18 14:12:11 [debug] Parsed 1 records 2022-11-18 14:12:11 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/host.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/host.json' 2022-11-18 14:12:11 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/host.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/host.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001821s 2022-11-18 14:12:11 [info] Verifying action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:11 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:11 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/host.json) 2022-11-18 14:12:11 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:11 [debug] Created ETL destination table object for table definition key 'host' 2022-11-18 14:12:11 [info] Create action xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:11 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/account.json 2022-11-18 14:12:11 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/account.json 2022-11-18 14:12:11 [debug] Parsed 1 records 2022-11-18 14:12:11 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/account.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/account.json' 2022-11-18 14:12:11 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/account.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/account.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002199s 2022-11-18 14:12:11 [info] Verifying action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:12 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:12 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/account.json) 2022-11-18 14:12:13 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:13 [debug] Created ETL destination table object for table definition key 'account' 2022-11-18 14:12:13 [info] Create action xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:13 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/instance.json 2022-11-18 14:12:13 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/instance.json 2022-11-18 14:12:13 [debug] Parsed 1 records 2022-11-18 14:12:13 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/instance.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/instance.json' 2022-11-18 14:12:13 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/instance.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/instance.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002019s 2022-11-18 14:12:13 [info] Verifying action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:13 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:14 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance.json) 2022-11-18 14:12:14 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:14 [debug] Created ETL destination table object for table definition key 'instance' 2022-11-18 14:12:14 [info] Create action xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:14 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/instance_type.json 2022-11-18 14:12:14 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/instance_type.json 2022-11-18 14:12:14 [debug] Parsed 1 records 2022-11-18 14:12:14 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/instance_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/instance_type.json' 2022-11-18 14:12:14 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/instance_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/instance_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002188s 2022-11-18 14:12:14 [info] Verifying action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:15 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:15 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance_type.json) 2022-11-18 14:12:15 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:15 [debug] Created ETL destination table object for table definition key 'instance_type' 2022-11-18 14:12:15 [info] Create action xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:16 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/image.json 2022-11-18 14:12:16 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/image.json 2022-11-18 14:12:16 [debug] Parsed 1 records 2022-11-18 14:12:16 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/image.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/image.json' 2022-11-18 14:12:16 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/image.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/image.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001962s 2022-11-18 14:12:16 [info] Verifying action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:16 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:17 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/image.json) 2022-11-18 14:12:18 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:18 [debug] Created ETL destination table object for table definition key 'image' 2022-11-18 14:12:18 [info] Create action xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:18 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/memory_buckets.json 2022-11-18 14:12:18 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/memory_buckets.json 2022-11-18 14:12:18 [debug] Parsed 1 records 2022-11-18 14:12:18 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/memory_buckets.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/memory_buckets.json' 2022-11-18 14:12:18 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/memory_buckets.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/memory_buckets.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001817s 2022-11-18 14:12:18 [info] Verifying action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:18 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:18 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=memory buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/memory_buckets.json) 2022-11-18 14:12:18 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:19 [debug] Created ETL destination table object for table definition key 'memory_buckets' 2022-11-18 14:12:19 [info] Create action xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:19 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/processor_buckets.json 2022-11-18 14:12:19 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/processor_buckets.json 2022-11-18 14:12:19 [debug] Parsed 1 records 2022-11-18 14:12:19 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/processor_buckets.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/processor_buckets.json' 2022-11-18 14:12:19 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/processor_buckets.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/processor_buckets.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001955s 2022-11-18 14:12:19 [info] Verifying action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:12:19 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:19 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=processor buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/processor_buckets.json) 2022-11-18 14:12:19 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:19 [debug] Created ETL destination table object for table definition key 'processor_buckets' 2022-11-18 14:12:19 [info] Empty lock directory specified, using temp directory: /tmp 2022-11-18 14:12:19 [info] Obtaining lock file '/tmp/etlv2_16621' 2022-11-18 14:12:19 [notice] Start processing section 'xdmod.jobs-common' 2022-11-18 14:12:20 [info] start (action_name: xdmod.jobs-common.JobsTableManagement, action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables), start_date: , end_date: ) 2022-11-18 14:12:20 [debug] Discover table 'modw.job_request_info' 2022-11-18 14:12:20 [debug] Discover table 'modw.job_task_countable' 2022-11-18 14:12:20 [debug] Discover table 'modw.meta_job' 2022-11-18 14:12:20 [debug] Discover table 'modw.meta_job_record' 2022-11-18 14:12:20 [notice] (action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables), start_time: 1668780740.2724, end_time: 1668780740.339, elapsed_time: 0.06657) 2022-11-18 14:12:20 [info] end (action_name: xdmod.jobs-common.JobsTableManagement, action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables)) 2022-11-18 14:12:20 [info] start (action_name: xdmod.jobs-common.CountableTypeIngestor, action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:12:20 [info] Truncate destination table: modw.countable_type 2022-11-18 14:12:20 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.countable_type 2022-11-18 14:12:22 [debug] Discover table 'modw.countable_type' 2022-11-18 14:12:22 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:12:22 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:12:22 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/countable_type.json 2022-11-18 14:12:22 [debug] Parsed 6 records 2022-11-18 14:12:22 [debug] Requested 5 record fields: countable_type_id, unit_id, countable_type, display, description 2022-11-18 14:12:22 [debug] No destination_field_map specified 2022-11-18 14:12:22 [debug] Auto-generating destination_field_map using 5 source fields: countable_type_id, unit_id, countable_type, display, description 2022-11-18 14:12:22 [debug] Available fields for table key 'countable_type': countable_type_id, unit_id, countable_type, display, description 2022-11-18 14:12:22 [debug] Generated destination_field_map: Table: countable_type countable_type_id -> countable_type_id unit_id -> unit_id countable_type -> countable_type display -> display description -> description 2022-11-18 14:12:22 [debug] Processing destination field map 2022-11-18 14:12:22 [debug] Insert SQL for table key 'countable_type': INSERT INTO modw.countable_type (countable_type_id, unit_id, countable_type, display, description) VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE countable_type_id = COALESCE(VALUES(countable_type_id), countable_type_id), unit_id = COALESCE(VALUES(unit_id), unit_id), countable_type = COALESCE(VALUES(countable_type), countable_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-11-18 14:12:22 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:22 [debug] ANALYZE TABLE modw.countable_type 2022-11-18 14:12:22 [debug] Completed in 0.026310s 2022-11-18 14:12:22 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 5 records (Time Taken: 1.60 s) 2022-11-18 14:12:22 [notice] (action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780740.8253, end_time: 1668780742.4267, elapsed_time: 1.60136, records_examined: 5, records_loaded: 5) 2022-11-18 14:12:22 [info] end (action_name: xdmod.jobs-common.CountableTypeIngestor, action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:12:23 [info] start (action_name: xdmod.jobs-common.JobRecordTypeIngestor, action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:12:23 [info] Truncate destination table: modw.job_record_type 2022-11-18 14:12:23 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.job_record_type 2022-11-18 14:12:25 [debug] Discover table 'modw.job_record_type' 2022-11-18 14:12:25 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:12:25 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:12:25 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/job_record_type.json 2022-11-18 14:12:25 [debug] Parsed 6 records 2022-11-18 14:12:25 [debug] Requested 4 record fields: job_record_type_id, job_record_type, display, description 2022-11-18 14:12:25 [debug] No destination_field_map specified 2022-11-18 14:12:25 [debug] Auto-generating destination_field_map using 4 source fields: job_record_type_id, job_record_type, display, description 2022-11-18 14:12:25 [debug] Available fields for table key 'job_record_type': job_record_type_id, job_record_type, display, description 2022-11-18 14:12:25 [debug] Generated destination_field_map: Table: job_record_type job_record_type_id -> job_record_type_id job_record_type -> job_record_type display -> display description -> description 2022-11-18 14:12:25 [debug] Processing destination field map 2022-11-18 14:12:25 [debug] Insert SQL for table key 'job_record_type': INSERT INTO modw.job_record_type (job_record_type_id, job_record_type, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE job_record_type_id = COALESCE(VALUES(job_record_type_id), job_record_type_id), job_record_type = COALESCE(VALUES(job_record_type), job_record_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-11-18 14:12:25 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:25 [debug] ANALYZE TABLE modw.job_record_type 2022-11-18 14:12:25 [debug] Completed in 0.025360s 2022-11-18 14:12:25 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 5 records (Time Taken: 2.23 s) 2022-11-18 14:12:25 [notice] (action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780743.27, end_time: 1668780745.5003, elapsed_time: 2.23028, records_examined: 5, records_loaded: 5) 2022-11-18 14:12:26 [info] end (action_name: xdmod.jobs-common.JobRecordTypeIngestor, action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:12:26 [info] start (action_name: xdmod.jobs-common.JobTaskTypeIngestor, action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:12:27 [info] Truncate destination table: modw.job_task_type 2022-11-18 14:12:27 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.job_task_type 2022-11-18 14:12:28 [debug] Discover table 'modw.job_task_type' 2022-11-18 14:12:28 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:12:28 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:12:28 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/job_task_type.json 2022-11-18 14:12:28 [debug] Parsed 6 records 2022-11-18 14:12:28 [debug] Requested 4 record fields: job_task_type_id, job_task_type, display, description 2022-11-18 14:12:28 [debug] No destination_field_map specified 2022-11-18 14:12:28 [debug] Auto-generating destination_field_map using 4 source fields: job_task_type_id, job_task_type, display, description 2022-11-18 14:12:28 [debug] Available fields for table key 'job_task_type': job_task_type_id, job_record_type_id, job_task_type, display, description 2022-11-18 14:12:28 [debug] Generated destination_field_map: Table: job_task_type job_task_type_id -> job_task_type_id job_task_type -> job_task_type display -> display description -> description 2022-11-18 14:12:28 [debug] Processing destination field map 2022-11-18 14:12:28 [debug] Insert SQL for table key 'job_task_type': INSERT INTO modw.job_task_type (job_task_type_id, job_task_type, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE job_task_type_id = COALESCE(VALUES(job_task_type_id), job_task_type_id), job_task_type = COALESCE(VALUES(job_task_type), job_task_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-11-18 14:12:29 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:29 [debug] ANALYZE TABLE modw.job_task_type 2022-11-18 14:12:29 [debug] Completed in 0.025410s 2022-11-18 14:12:29 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 5 records (Time Taken: 2.84 s) 2022-11-18 14:12:30 [notice] (action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780747.0102, end_time: 1668780749.8483, elapsed_time: 2.83808, records_examined: 5, records_loaded: 5) 2022-11-18 14:12:30 [info] end (action_name: xdmod.jobs-common.JobTaskTypeIngestor, action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:12:31 [info] start (action_name: xdmod.jobs-common.UnitIngestor, action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:12:31 [info] Truncate destination table: modw.unit 2022-11-18 14:12:31 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.unit 2022-11-18 14:12:32 [debug] Discover table 'modw.unit' 2022-11-18 14:12:32 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:12:32 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:12:32 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/unit.json 2022-11-18 14:12:32 [debug] Parsed 7 records 2022-11-18 14:12:32 [debug] Requested 4 record fields: unit_id, unit, display, description 2022-11-18 14:12:32 [debug] No destination_field_map specified 2022-11-18 14:12:32 [debug] Auto-generating destination_field_map using 4 source fields: unit_id, unit, display, description 2022-11-18 14:12:32 [debug] Available fields for table key 'unit': unit_id, unit, display, description 2022-11-18 14:12:32 [debug] Generated destination_field_map: Table: unit unit_id -> unit_id unit -> unit display -> display description -> description 2022-11-18 14:12:32 [debug] Processing destination field map 2022-11-18 14:12:32 [debug] Insert SQL for table key 'unit': INSERT INTO modw.unit (unit_id, unit, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE unit_id = COALESCE(VALUES(unit_id), unit_id), unit = COALESCE(VALUES(unit), unit), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-11-18 14:12:32 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:32 [debug] ANALYZE TABLE modw.unit 2022-11-18 14:12:32 [debug] Completed in 0.040480s 2022-11-18 14:12:32 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 6 records (Time Taken: 1.72 s) 2022-11-18 14:12:33 [notice] (action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780751.2743, end_time: 1668780752.9932, elapsed_time: 1.71889, records_examined: 6, records_loaded: 6) 2022-11-18 14:12:33 [info] end (action_name: xdmod.jobs-common.UnitIngestor, action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:12:33 [info] start (action_name: xdmod.jobs-common.SubmissionVenueIngestor, action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:12:33 [info] Truncate destination table: modw.submission_venue 2022-11-18 14:12:33 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.submission_venue 2022-11-18 14:12:39 [debug] Discover table 'modw.submission_venue' 2022-11-18 14:12:39 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:12:39 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:12:39 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/submission_venue.json 2022-11-18 14:12:39 [debug] Parsed 5 records 2022-11-18 14:12:39 [debug] Requested 5 record fields: submission_venue_id, submission_venue, display, description, order_id 2022-11-18 14:12:39 [debug] No destination_field_map specified 2022-11-18 14:12:39 [debug] Auto-generating destination_field_map using 5 source fields: submission_venue_id, submission_venue, display, description, order_id 2022-11-18 14:12:39 [debug] Available fields for table key 'submission_venue': submission_venue_id, submission_venue, display, description, order_id 2022-11-18 14:12:39 [debug] Generated destination_field_map: Table: submission_venue submission_venue_id -> submission_venue_id submission_venue -> submission_venue display -> display description -> description order_id -> order_id 2022-11-18 14:12:39 [debug] Processing destination field map 2022-11-18 14:12:39 [debug] Insert SQL for table key 'submission_venue': INSERT INTO modw.submission_venue (submission_venue_id, submission_venue, display, description, order_id) VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE submission_venue_id = COALESCE(VALUES(submission_venue_id), submission_venue_id), submission_venue = COALESCE(VALUES(submission_venue), submission_venue), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description), order_id = COALESCE(VALUES(order_id), order_id) 2022-11-18 14:12:40 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:40 [debug] ANALYZE TABLE modw.submission_venue 2022-11-18 14:12:41 [debug] Completed in 0.827520s 2022-11-18 14:12:41 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 4 records (Time Taken: 8.52 s) 2022-11-18 14:12:41 [notice] (action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780753.1441, end_time: 1668780761.6688, elapsed_time: 8.52463, records_examined: 4, records_loaded: 4) 2022-11-18 14:12:41 [info] end (action_name: xdmod.jobs-common.SubmissionVenueIngestor, action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:12:41 [notice] Finished processing section 'xdmod.jobs-common' 2022-11-18 14:12:41 [notice] Start processing section 'xdmod.ingest-organizations' 2022-11-18 14:12:42 [info] start (action_name: xdmod.ingest-organizations.IngestOrganizationConfig, action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:12:42 [debug] Discover table 'mod_shredder.staging_organization' 2022-11-18 14:12:42 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:12:43 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:12:43 [debug] Parsing /etc/xdmod/etl/../organization.json 2022-11-18 14:12:43 [debug] Registering filter: xdmod.external_process 2022-11-18 14:12:43 [debug] Creating filter xdmod.external_process: jq '{id:1,name,abbrev}' 2022-11-18 14:12:43 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:12:43 [debug] Parsed 1 records 2022-11-18 14:12:43 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/common/organization.schema.json 2022-11-18 14:12:43 [debug] Requested 3 record fields: id, name, abbrev 2022-11-18 14:12:43 [debug] Processing destination field map 2022-11-18 14:12:43 [debug] Insert SQL for table key 'staging_organization': INSERT INTO mod_shredder.staging_organization (organization_id, organization_name, organization_abbrev) VALUES (?, ?, ?) ON DUPLICATE KEY UPDATE organization_id = COALESCE(VALUES(organization_id), organization_id), organization_name = COALESCE(VALUES(organization_name), organization_name), organization_abbrev = COALESCE(VALUES(organization_abbrev), organization_abbrev) 2022-11-18 14:12:43 [info] Execute Post-execute tasks: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:43 [debug] ANALYZE TABLE mod_shredder.staging_organization 2022-11-18 14:12:44 [debug] Completed in 0.267090s 2022-11-18 14:12:44 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 1.13 s) 2022-11-18 14:12:44 [notice] (action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780762.9408, end_time: 1668780764.0698, elapsed_time: 1.12903, records_examined: 1, records_loaded: 1) 2022-11-18 14:12:44 [info] end (action_name: xdmod.ingest-organizations.IngestOrganizationConfig, action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:12:45 [info] start (action_name: xdmod.ingest-organizations.IngestOrganizationHpcdb, action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:12:45 [debug] Discover table 'mod_hpcdb.hpcdb_organizations' 2022-11-18 14:12:45 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:12:46 [debug] Available Variables: DESTINATION_SCHEMA='mod_hpcdb', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:12:46 [debug] Allowing same-server SQL optimizations 2022-11-18 14:12:46 [info] Single-database ingest into ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:47 [debug] INSERT INTO mod_hpcdb.hpcdb_organizations (organization_id,organization_name,organization_abbrev) SELECT o.organization_id AS organization_id, o.organization_name AS organization_name, o.organization_abbrev AS organization_abbrev FROM mod_shredder.staging_organization AS o ON DUPLICATE KEY UPDATE organization_id=VALUES(organization_id),organization_name=VALUES(organization_name),organization_abbrev=VALUES(organization_abbrev) 2022-11-18 14:12:47 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-11-18 14:12:47 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 1.45 s) 2022-11-18 14:12:48 [notice] (action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor), start_time: 1668780765.9716, end_time: 1668780767.4186, elapsed_time: 1.44703, records_examined: 0, records_loaded: 0) 2022-11-18 14:12:48 [info] end (action_name: xdmod.ingest-organizations.IngestOrganizationHpcdb, action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:12:48 [info] start (action_name: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse, action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:12:49 [debug] Discover table 'modw.organization' 2022-11-18 14:12:50 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:12:50 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_hpcdb', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:12:50 [debug] Query optimization disabled 2022-11-18 14:12:50 [debug] Using multi-database ingest 2022-11-18 14:12:50 [debug] Using temporary file '/tmp/modw.organization.data.ts_1668780770.1680160760jOyvrF' for destination table key 'organization' 2022-11-18 14:12:50 [debug] LOAD statement for destination table key 'organization' ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): CREATE TABLE modw.tmp_organization_1668780770 LIKE modw.organization; ALTER TABLE modw.tmp_organization_1668780770 DISABLE KEYS; LOAD DATA LOCAL INFILE '/tmp/modw.organization.data.ts_1668780770.1680160760jOyvrF' INTO TABLE modw.tmp_organization_1668780770 FIELDS TERMINATED BY 0x1e OPTIONALLY ENCLOSED BY 0x1f ESCAPED BY 0x5c LINES TERMINATED BY 0x1d (id,abbrev,name,short_name,long_name,organization_origin_id,order_id); SHOW WARNINGS; INSERT INTO modw.organization (id,abbrev,name,short_name,long_name,organization_origin_id,order_id) SELECT id,abbrev,name,short_name,long_name,organization_origin_id,order_id FROM modw.tmp_organization_1668780770 ON DUPLICATE KEY UPDATE id=VALUES(id),abbrev=VALUES(abbrev),name=VALUES(name),short_name=VALUES(short_name),long_name=VALUES(long_name),organization_origin_id=VALUES(organization_origin_id),order_id=VALUES(order_id); DROP TABLE modw.tmp_organization_1668780770; 2022-11-18 14:12:50 [info] Multi-database ingest into ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:51 [info] Switching to un-buffered query mode 2022-11-18 14:12:51 [debug] Current net_write_timeout = 60 2022-11-18 14:12:51 [info] ETL\Ingestor\DatabaseIngestor: Querying ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:52 [debug] Source query ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod): SELECT organization_id AS id, organization_abbrev AS abbrev, organization_name AS name, COALESCE(organization_abbrev, organization_name) AS short_name, CASE WHEN ISNULL(organization_abbrev) THEN organization_name ELSE CONCAT(organization_abbrev, ' - ', organization_name) END AS long_name, NULL AS order_id, organization_id AS organization_origin_id FROM mod_hpcdb.hpcdb_organizations 2022-11-18 14:12:53 [debug] Loaded 1 records into 'organization' 2022-11-18 14:12:53 [debug] Loaded 1 files in 1s 2022-11-18 14:12:53 [info] ETL\Ingestor\DatabaseIngestor: Processed 1 records (1 source records) in 2s 2022-11-18 14:12:54 [info] Returning buffered query mode to: true 2022-11-18 14:12:54 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:12:55 [debug] ANALYZE TABLE modw.organization 2022-11-18 14:12:56 [debug] Completed in 0.694600s 2022-11-18 14:12:56 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 1 records (Time Taken: 6.63 s) 2022-11-18 14:12:58 [notice] (action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_time: 1668780769.8684, end_time: 1668780776.5008, elapsed_time: 6.63245, records_examined: 1, records_loaded: 1) 2022-11-18 14:12:58 [info] end (action_name: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse, action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:13:00 [notice] Finished processing section 'xdmod.ingest-organizations' 2022-11-18 14:13:00 [notice] Start processing section 'xdmod.ingest-resource-types' 2022-11-18 14:13:01 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement, action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables), start_date: , end_date: ) 2022-11-18 14:13:01 [debug] Discover table 'modw.staging_resource_type' 2022-11-18 14:13:01 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables), start_time: 1668780781.2142, end_time: 1668780781.2385, elapsed_time: 0.02425) 2022-11-18 14:13:01 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement, action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables)) 2022-11-18 14:13:03 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesStagingUnknown, action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_date: , end_date: ) 2022-11-18 14:13:04 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-11-18 14:13:05 [info] Executing statement ( 1 / 2) (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)-sql-1, endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod), sql: SET SESSION sql_mode='NO_AUTO_VALUE_ON_ZERO') 2022-11-18 14:13:05 [info] Finished executing statement ( 1 / 2) (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)-sql-1, rows: 0, start_time: 1668780785.3132, end_time: 1668780785.9753, elapsed_time: 0.66217494010925) 2022-11-18 14:13:06 [info] Executing statement ( 2 / 2) (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)-sql-2, endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod), sql: INSERT IGNORE INTO mod_shredder.staging_resource_type (resource_type_id, resource_type_description, resource_type_abbrev) VALUES ('0', 'Unknown Resource Type', 'UNK')) 2022-11-18 14:13:06 [info] Finished executing statement ( 2 / 2) (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)-sql-2, rows: 0, start_time: 1668780786.212, end_time: 1668780786.6015, elapsed_time: 0.38950705528259) 2022-11-18 14:13:06 [notice] Finished Processing 2 SQL statements 2022-11-18 14:13:07 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1668780784.3898, end_time: 1668780787.0849, elapsed_time: 2.69516) 2022-11-18 14:13:07 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesStagingUnknown, action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)) 2022-11-18 14:13:08 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesStaging, action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:13:08 [debug] Discover table 'mod_shredder.staging_resource_type' 2022-11-18 14:13:08 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:08 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:08 [debug] Parsing /etc/xdmod/etl/../resource_types.json 2022-11-18 14:13:08 [debug] Registering filter: xdmod.external_process 2022-11-18 14:13:08 [debug] Creating filter xdmod.external_process: jq '[ .resource_types | to_entries[]| {abbrev: .key, description:.value.description} ] ' 2022-11-18 14:13:08 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:13:09 [debug] Parsed 12 records 2022-11-18 14:13:09 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/common/resource-types.schema.json 2022-11-18 14:13:09 [debug] Requested 2 record fields: abbrev, description 2022-11-18 14:13:09 [debug] Processing destination field map 2022-11-18 14:13:09 [debug] Insert SQL for table key 'staging_resource_type': INSERT INTO mod_shredder.staging_resource_type (resource_type_description, resource_type_abbrev) VALUES (?, ?) ON DUPLICATE KEY UPDATE resource_type_description = COALESCE(VALUES(resource_type_description), resource_type_description), resource_type_abbrev = COALESCE(VALUES(resource_type_abbrev), resource_type_abbrev) 2022-11-18 14:13:09 [info] Execute Post-execute tasks: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:09 [debug] ANALYZE TABLE mod_shredder.staging_resource_type 2022-11-18 14:13:09 [debug] Completed in 0.063820s 2022-11-18 14:13:09 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 12 records (Time Taken: 0.34 s) 2022-11-18 14:13:09 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780788.8001, end_time: 1668780789.1384, elapsed_time: 0.33821, records_examined: 12, records_loaded: 12) 2022-11-18 14:13:10 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesStaging, action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:13:10 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesHpcdb, action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:13:10 [debug] Discover table 'mod_hpcdb.hpcdb_resource_types' 2022-11-18 14:13:10 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:10 [debug] Available Variables: DESTINATION_SCHEMA='mod_hpcdb', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:10 [debug] Allowing same-server SQL optimizations 2022-11-18 14:13:10 [info] Single-database ingest into ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:11 [debug] INSERT INTO mod_hpcdb.hpcdb_resource_types (type_id,type_abbr,type_desc) SELECT rt.resource_type_id AS type_id, rt.resource_type_abbrev AS type_abbr, rt.resource_type_description AS type_desc FROM mod_shredder.staging_resource_type AS rt ON DUPLICATE KEY UPDATE type_id=VALUES(type_id),type_abbr=VALUES(type_abbr),type_desc=VALUES(type_desc) 2022-11-18 14:13:11 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-11-18 14:13:12 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 2.39 s) 2022-11-18 14:13:13 [notice] (action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor), start_time: 1668780790.2384, end_time: 1668780792.6324, elapsed_time: 2.39402, records_examined: 0, records_loaded: 0) 2022-11-18 14:13:13 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesHpcdb, action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:13:14 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesDatawarehouse, action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:13:14 [debug] Discover table 'modw.resourcetype' 2022-11-18 14:13:14 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:14 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_hpcdb', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:14 [debug] Allowing same-server SQL optimizations 2022-11-18 14:13:14 [info] Single-database ingest into ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:15 [debug] INSERT INTO modw.resourcetype (id,description,abbrev) SELECT rtf.type_id AS id, rtf.type_desc AS description, rtf.type_abbr AS abbrev FROM mod_hpcdb.hpcdb_resource_types AS rtf ON DUPLICATE KEY UPDATE id=VALUES(id),description=VALUES(description),abbrev=VALUES(abbrev) 2022-11-18 14:13:15 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-11-18 14:13:15 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.83 s) 2022-11-18 14:13:15 [notice] (action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_time: 1668780794.4197, end_time: 1668780795.2495, elapsed_time: 0.82979, records_examined: 0, records_loaded: 0) 2022-11-18 14:13:15 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesDatawarehouse, action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:13:15 [notice] Finished processing section 'xdmod.ingest-resource-types' 2022-11-18 14:13:15 [notice] Start processing section 'xdmod.ingest-resources' 2022-11-18 14:13:15 [info] start (action_name: xdmod.ingest-resources.IngestResourceConfig, action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:13:16 [debug] Discover table 'mod_shredder.staging_resource_config' 2022-11-18 14:13:16 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:17 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:17 [debug] Parsing /etc/xdmod/etl/../resources.json 2022-11-18 14:13:17 [debug] Parsed 4 records 2022-11-18 14:13:17 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/common/resources.schema.json 2022-11-18 14:13:17 [debug] Requested 7 record fields: resource, name, description, resource_type, pi_column, shared_jobs, timezone 2022-11-18 14:13:17 [debug] Processing destination field map 2022-11-18 14:13:17 [debug] Insert SQL for table key 'staging_resource_config': INSERT INTO mod_shredder.staging_resource_config (resource, name, description, shared_jobs, timezone, type_abbrev) VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE resource = COALESCE(VALUES(resource), resource), name = COALESCE(VALUES(name), name), description = COALESCE(VALUES(description), description), shared_jobs = COALESCE(VALUES(shared_jobs), shared_jobs), timezone = COALESCE(VALUES(timezone), timezone), type_abbrev = COALESCE(VALUES(type_abbrev), type_abbrev) 2022-11-18 14:13:17 [info] Execute Post-execute tasks: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:17 [debug] ANALYZE TABLE mod_shredder.staging_resource_config 2022-11-18 14:13:17 [debug] Completed in 0.188050s 2022-11-18 14:13:17 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 4 records (Time Taken: 0.77 s) 2022-11-18 14:13:18 [notice] (action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780796.7846, end_time: 1668780797.5525, elapsed_time: 0.76791, records_examined: 4, records_loaded: 4) 2022-11-18 14:13:18 [info] end (action_name: xdmod.ingest-resources.IngestResourceConfig, action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:13:18 [info] start (action_name: xdmod.ingest-resources.IngestResourcesStaging, action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:13:18 [debug] Discover table 'mod_shredder.staging_resource' 2022-11-18 14:13:18 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:18 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:18 [debug] Allowing same-server SQL optimizations

rob-baron commented 10 months ago

2022-11-18 14:13:18 [info] Single-database ingest into ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:18 [debug] INSERT INTO mod_shredder.staging_resource (resource_name) SELECT DISTINCT rc.resource AS resource_name FROM mod_shredder.staging_resource_config AS rc ON DUPLICATE KEY UPDATE resource_name=VALUES(resource_name) 2022-11-18 14:13:18 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-11-18 14:13:18 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.35 s) 2022-11-18 14:13:19 [notice] (action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_time: 1668780798.6032, end_time: 1668780798.9503, elapsed_time: 0.34713, records_examined: 0, records_loaded: 0) 2022-11-18 14:13:20 [info] end (action_name: xdmod.ingest-resources.IngestResourcesStaging, action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:13:20 [info] start (action_name: xdmod.ingest-resources.HpcdbIngestResources, action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:13:20 [debug] Discover table 'mod_hpcdb.hpcdb_resources' 2022-11-18 14:13:20 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:21 [debug] Available Variables: DESTINATION_SCHEMA='mod_hpcdb', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:21 [debug] Allowing same-server SQL optimizations 2022-11-18 14:13:21 [info] Single-database ingest into ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:21 [debug] INSERT INTO mod_hpcdb.hpcdb_resources (resource_id,resource_type_id,organization_id,resource_name,resource_code,resource_description,resource_shared_jobs,resource_timezone) SELECT r.resource_id AS resource_id, COALESCE(srt.resource_type_id, 0) AS resource_type_id, 1 AS organization_id, rc.name AS resource_name, rc.resource AS resource_code, rc.description AS resource_description, COALESCE(rc.shared_jobs, FALSE) AS resource_shared_jobs, COALESCE(rc.timezone, 'UTC') AS resource_timezone FROM mod_shredder.staging_resource AS r LEFT OUTER JOIN mod_shredder.staging_resource_config AS rc ON r.resource_name = rc.resource LEFT JOIN mod_shredder.staging_resource_type AS srt ON rc.type_abbrev = srt.resource_type_abbrev ON DUPLICATE KEY UPDATE resource_id=VALUES(resource_id),resource_type_id=VALUES(resource_type_id),organization_id=VALUES(organization_id),resource_name=VALUES(resource_name),resource_code=VALUES(resource_code),resource_description=VALUES(resource_description),resource_shared_jobs=VALUES(resource_shared_jobs),resource_timezone=VALUES(resource_timezone) 2022-11-18 14:13:21 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-11-18 14:13:21 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.84 s) 2022-11-18 14:13:21 [notice] (action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor), start_time: 1668780800.5355, end_time: 1668780801.3793, elapsed_time: 0.84384, records_examined: 0, records_loaded: 0) 2022-11-18 14:13:21 [info] end (action_name: xdmod.ingest-resources.HpcdbIngestResources, action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:13:22 [info] start (action_name: xdmod.ingest-resources.IngestResourcefact, action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:13:22 [debug] Discover table 'modw.resourcefact' 2022-11-18 14:13:22 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:23 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_hpcdb', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:23 [debug] Allowing same-server SQL optimizations 2022-11-18 14:13:23 [info] Single-database ingest into ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:23 [debug] INSERT INTO modw.resourcefact (id,resourcetype_id,organization_id,name,code,description,shared_jobs,timezone,resource_origin_id) SELECT r.resource_id AS id, r.resource_type_id AS resourcetype_id, r.organization_id AS organization_id, r.resource_name AS name, r.resource_code AS code, r.resource_description AS description, r.resource_shared_jobs AS shared_jobs, r.resource_timezone AS timezone, r.resource_id AS resource_origin_id FROM mod_hpcdb.hpcdb_resources AS r ORDER BY r.resource_id ON DUPLICATE KEY UPDATE id=VALUES(id),resourcetype_id=VALUES(resourcetype_id),organization_id=VALUES(organization_id),name=VALUES(name),code=VALUES(code),description=VALUES(description),shared_jobs=VALUES(shared_jobs),timezone=VALUES(timezone),resource_origin_id=VALUES(resource_origin_id) 2022-11-18 14:13:23 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-11-18 14:13:23 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.95 s) 2022-11-18 14:13:23 [notice] (action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor), start_time: 1668780802.5436, end_time: 1668780803.4925, elapsed_time: 0.94887, records_examined: 0, records_loaded: 0) 2022-11-18 14:13:23 [info] end (action_name: xdmod.ingest-resources.IngestResourcefact, action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:13:24 [info] start (action_name: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging, action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:13:24 [debug] Discover table 'mod_shredder.staging_resource_type_realms' 2022-11-18 14:13:24 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:25 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:25 [debug] Loading configuration file /etc/xdmod/resource_types.json 2022-11-18 14:13:25 [debug] Parsing /etc/xdmod/resource_types.json 2022-11-18 14:13:25 [debug] Parsed 1 records 2022-11-18 14:13:25 [debug] Stored object Configuration\XdmodConfiguration (/etc/xdmod/etl/../resource_types.json) in APCu cache with key Configuration\XdmodConfiguration|/etc/xdmod/etl/../resource_types.json|40cd750bba9870f18aada2478b24840a in 0.001655s 2022-11-18 14:13:25 [debug] Parsing /tmp/etl-xdmod-config-jsonRnzK18 2022-11-18 14:13:25 [debug] Registering filter: xdmod.external_process 2022-11-18 14:13:25 [debug] Creating filter xdmod.external_process: jq '[(.resource_types | keys) as $a | .resource_types as $rt | $a[] | {abbrev: . , realm: $rt[.].realms[]}]' 2022-11-18 14:13:25 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:13:25 [debug] Parsed 9 records 2022-11-18 14:13:25 [debug] Requested 2 record fields: abbrev, realm 2022-11-18 14:13:25 [debug] No destination_field_map specified 2022-11-18 14:13:25 [debug] Auto-generating destination_field_map using 2 source fields: abbrev, realm 2022-11-18 14:13:25 [debug] Available fields for table key 'staging_resource_type_realms': staging_resource_type_realm_id, abbrev, realm 2022-11-18 14:13:25 [debug] Generated destination_field_map: Table: staging_resource_type_realms abbrev -> abbrev realm -> realm 2022-11-18 14:13:25 [debug] Processing destination field map 2022-11-18 14:13:25 [debug] Insert SQL for table key 'staging_resource_type_realms': INSERT INTO mod_shredder.staging_resource_type_realms (abbrev, realm) VALUES (?, ?) ON DUPLICATE KEY UPDATE abbrev = COALESCE(VALUES(abbrev), abbrev), realm = COALESCE(VALUES(realm), realm) 2022-11-18 14:13:26 [info] Execute Post-execute tasks: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:27 [debug] ANALYZE TABLE mod_shredder.staging_resource_type_realms 2022-11-18 14:13:27 [debug] Completed in 0.056420s 2022-11-18 14:13:27 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 9 records (Time Taken: 2.45 s) 2022-11-18 14:13:27 [notice] (action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780804.6306, end_time: 1668780807.0785, elapsed_time: 2.44789, records_examined: 9, records_loaded: 9) 2022-11-18 14:13:27 [info] end (action_name: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging, action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:13:27 [info] start (action_name: xdmod.ingest-resources.IngestInitialRealms, action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:13:27 [debug] Discover table 'moddb.realms' 2022-11-18 14:13:27 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:28 [debug] Available Variables: DESTINATION_SCHEMA='moddb', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:28 [debug] Allowing same-server SQL optimizations 2022-11-18 14:13:28 [info] Single-database ingest into ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:28 [debug] INSERT INTO moddb.realms (display,name,module_id) SELECT DISTINCT srtr.realm AS display, LOWER(srtr.realm) AS name, 1 AS module_id FROM mod_shredder.staging_resource_type_realms AS srtr ON DUPLICATE KEY UPDATE display=VALUES(display),name=VALUES(name),module_id=VALUES(module_id) 2022-11-18 14:13:28 [info] ETL\Ingestor\DatabaseIngestor: Processed 5 records 2022-11-18 14:13:28 [info] Execute Post-execute tasks: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:28 [debug] ANALYZE TABLE moddb.realms 2022-11-18 14:13:28 [debug] Completed in 0.024810s 2022-11-18 14:13:28 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 5 records (Time Taken: 0.95 s) 2022-11-18 14:13:28 [notice] (action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor), start_time: 1668780807.9649, end_time: 1668780808.9155, elapsed_time: 0.95059, records_examined: 5, records_loaded: 5) 2022-11-18 14:13:29 [info] end (action_name: xdmod.ingest-resources.IngestInitialRealms, action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:13:31 [info] start (action_name: xdmod.ingest-resources.IngestResourcefactRealmRelations, action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-11-18 14:13:31 [debug] Discover table 'moddb.resource_type_realms' 2022-11-18 14:13:31 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:32 [debug] Available Variables: DESTINATION_SCHEMA='moddb', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:32 [debug] Allowing same-server SQL optimizations 2022-11-18 14:13:32 [info] Single-database ingest into ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:32 [debug] INSERT INTO moddb.resource_type_realms (resource_type_id,realm_id) SELECT rt.id AS resource_type_id, r.realm_id AS realm_id FROM mod_shredder.staging_resource_type_realms AS srtr JOIN modw.resourcetype AS rt ON rt.abbrev = srtr.abbrev JOIN moddb.realms AS r ON r.display = srtr.realm ON DUPLICATE KEY UPDATE resource_type_id=VALUES(resource_type_id),realm_id=VALUES(realm_id) 2022-11-18 14:13:33 [info] ETL\Ingestor\DatabaseIngestor: Processed 198 records 2022-11-18 14:13:33 [info] Execute Post-execute tasks: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:33 [debug] ANALYZE TABLE moddb.resource_type_realms 2022-11-18 14:13:34 [debug] Completed in 0.199050s 2022-11-18 14:13:34 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 198 records (Time Taken: 2.26 s) 2022-11-18 14:13:34 [notice] (action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor), start_time: 1668780811.8043, end_time: 1668780814.0629, elapsed_time: 2.25856, records_examined: 198, records_loaded: 198) 2022-11-18 14:13:34 [info] end (action_name: xdmod.ingest-resources.IngestResourcefactRealmRelations, action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor)) 2022-11-18 14:13:35 [notice] Finished processing section 'xdmod.ingest-resources' 2022-11-18 14:13:35 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:13:35 [info] start (action_name: xdmod.jobs-cloud-common.CloudTableManagement, action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_date: , end_date: ) 2022-11-18 14:13:36 [debug] Discover table 'modw_cloud.account' 2022-11-18 14:13:36 [debug] Discover table 'modw_cloud.event' 2022-11-18 14:13:36 [debug] Column last_modified: values for "timestamp" differ ("CURRENT_TIMESTAMP on update current_timestamp" != "current_timestamp() on update current_timestamp()") 2022-11-18 14:13:36 [notice] Altering table modw_cloud.event 2022-11-18 14:13:37 [debug] Alter table SQL ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod): ALTER TABLE modw_cloud.event CHANGE COLUMN last_modified last_modified timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP on update current_timestamp ; 2022-11-18 14:13:37 [debug] Discover table 'modw_cloud.asset' 2022-11-18 14:13:37 [debug] Discover table 'modw_cloud.host' 2022-11-18 14:13:37 [debug] Discover table 'modw_cloud.instance_data' 2022-11-18 14:13:37 [debug] Discover table 'modw_cloud.event_asset' 2022-11-18 14:13:37 [debug] Discover table 'modw_cloud.raw_resource_specs' 2022-11-18 14:13:37 [debug] Discover table 'modw_cloud.staging_resource_specifications' 2022-11-18 14:13:37 [debug] Discover table 'modw_cloud.cloud_resource_specs' 2022-11-18 14:13:37 [debug] Column last_modified: values for "timestamp" differ ("CURRENT_TIMESTAMP on update current_timestamp" != "current_timestamp() on update current_timestamp()") 2022-11-18 14:13:37 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 14:13:39 [debug] Alter table SQL ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod): ALTER TABLE modw_cloud.cloud_resource_specs CHANGE COLUMN last_modified last_modified timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP on update current_timestamp ; 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.openstack_raw_event' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.openstack_raw_instance_type' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.openstack_raw_volume' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.generic_cloud_raw_event' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.generic_cloud_raw_instance_type' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.generic_cloud_raw_volume' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.domains' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.domain_submission_venues_staging' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.domain_submission_venues' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.staging_pi_to_project' 2022-11-18 14:13:39 [debug] Discover table 'modw_cloud.session_records' 2022-11-18 14:13:39 [debug] Column last_modified: values for "timestamp" differ ("CURRENT_TIMESTAMP on update current_timestamp" != "current_timestamp() on update current_timestamp()") 2022-11-18 14:13:39 [notice] Altering table modw_cloud.session_records 2022-11-18 14:13:40 [debug] Alter table SQL ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod): ALTER TABLE modw_cloud.session_records CHANGE COLUMN last_modified last_modified timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP on update current_timestamp ; 2022-11-18 14:13:40 [debug] Discover table 'modw_cloud.cloudfact_by_day_sessionlist' 2022-11-18 14:13:40 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1668780816.0473, end_time: 1668780820.2509, elapsed_time: 4.20365) 2022-11-18 14:13:40 [info] end (action_name: xdmod.jobs-cloud-common.CloudTableManagement, action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables)) 2022-11-18 14:13:41 [info] start (action_name: xdmod.jobs-cloud-common.CloudAssetTypeInitializer, action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:13:42 [debug] Discover table 'modw_cloud.asset_type' 2022-11-18 14:13:42 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:43 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:43 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/asset_type.json 2022-11-18 14:13:43 [debug] Parsed 6 records 2022-11-18 14:13:43 [debug] Requested 5 record fields: asset_type_id, asset_type, display, description, unit_id 2022-11-18 14:13:43 [debug] No destination_field_map specified 2022-11-18 14:13:43 [debug] Auto-generating destination_field_map using 5 source fields: asset_type_id, asset_type, display, description, unit_id 2022-11-18 14:13:43 [debug] Available fields for table key 'asset_type': asset_type_id, asset_type, display, description, unit_id 2022-11-18 14:13:43 [debug] Generated destination_field_map: Table: asset_type asset_type_id -> asset_type_id asset_type -> asset_type display -> display description -> description unit_id -> unit_id 2022-11-18 14:13:43 [debug] Processing destination field map 2022-11-18 14:13:43 [debug] Insert SQL for table key 'asset_type': INSERT INTO modw_cloud.asset_type (asset_type_id, asset_type, display, description, unit_id) VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE asset_type_id = COALESCE(VALUES(asset_type_id), asset_type_id), asset_type = COALESCE(VALUES(asset_type), asset_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description), unit_id = COALESCE(VALUES(unit_id), unit_id) 2022-11-18 14:13:43 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:43 [debug] ANALYZE TABLE modw_cloud.asset_type 2022-11-18 14:13:43 [debug] Completed in 0.035820s 2022-11-18 14:13:43 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 5 records (Time Taken: 1.60 s) 2022-11-18 14:13:44 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780822.2813, end_time: 1668780823.8816, elapsed_time: 1.60038, records_examined: 5, records_loaded: 5) 2022-11-18 14:13:44 [info] end (action_name: xdmod.jobs-cloud-common.CloudAssetTypeInitializer, action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:13:44 [info] start (action_name: xdmod.jobs-cloud-common.CloudEventTypeInitializer, action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:13:44 [debug] Discover table 'modw_cloud.event_type' 2022-11-18 14:13:44 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:44 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:44 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/event_type.json 2022-11-18 14:13:44 [debug] Parsed 66 records 2022-11-18 14:13:44 [debug] Requested 4 record fields: event_type_id, event_type, display, description 2022-11-18 14:13:44 [debug] No destination_field_map specified 2022-11-18 14:13:44 [debug] Auto-generating destination_field_map using 4 source fields: event_type_id, event_type, display, description 2022-11-18 14:13:44 [debug] Available fields for table key 'event_type': event_type_id, event_type, display, description 2022-11-18 14:13:44 [debug] Generated destination_field_map: Table: event_type event_type_id -> event_type_id event_type -> event_type display -> display description -> description 2022-11-18 14:13:44 [debug] Processing destination field map 2022-11-18 14:13:44 [debug] Insert SQL for table key 'event_type': INSERT INTO modw_cloud.event_type (event_type_id, event_type, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE event_type_id = COALESCE(VALUES(event_type_id), event_type_id), event_type = COALESCE(VALUES(event_type), event_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-11-18 14:13:44 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:44 [debug] ANALYZE TABLE modw_cloud.event_type 2022-11-18 14:13:44 [debug] Completed in 0.079390s 2022-11-18 14:13:44 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 65 records (Time Taken: 0.20 s) 2022-11-18 14:13:45 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780824.7962, end_time: 1668780824.9953, elapsed_time: 0.19905, records_examined: 65, records_loaded: 65) 2022-11-18 14:13:46 [info] end (action_name: xdmod.jobs-cloud-common.CloudEventTypeInitializer, action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:13:46 [info] start (action_name: xdmod.jobs-cloud-common.CloudRecordTypeInitializer, action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:13:46 [debug] Discover table 'modw_cloud.record_type' 2022-11-18 14:13:47 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:47 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:47 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/record_type.json 2022-11-18 14:13:47 [debug] Parsed 5 records 2022-11-18 14:13:47 [debug] Requested 4 record fields: record_type_id, record_type, display, description 2022-11-18 14:13:47 [debug] No destination_field_map specified 2022-11-18 14:13:47 [debug] Auto-generating destination_field_map using 4 source fields: record_type_id, record_type, display, description 2022-11-18 14:13:47 [debug] Available fields for table key 'record_type': record_type_id, record_type, display, description 2022-11-18 14:13:47 [debug] Generated destination_field_map: Table: record_type record_type_id -> record_type_id record_type -> record_type display -> display description -> description 2022-11-18 14:13:47 [debug] Processing destination field map 2022-11-18 14:13:47 [debug] Insert SQL for table key 'record_type': INSERT INTO modw_cloud.record_type (record_type_id, record_type, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE record_type_id = COALESCE(VALUES(record_type_id), record_type_id), record_type = COALESCE(VALUES(record_type), record_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-11-18 14:13:47 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:48 [debug] ANALYZE TABLE modw_cloud.record_type 2022-11-18 14:13:49 [debug] Completed in 1.346260s 2022-11-18 14:13:49 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 4 records (Time Taken: 2.46 s) 2022-11-18 14:13:49 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780826.9825, end_time: 1668780829.4393, elapsed_time: 2.45675, records_examined: 4, records_loaded: 4) 2022-11-18 14:13:51 [info] end (action_name: xdmod.jobs-cloud-common.CloudRecordTypeInitializer, action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:13:52 [info] start (action_name: xdmod.jobs-cloud-common.CloudHostUnknownInitializer, action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:13:53 [debug] Discover table 'modw_cloud.host' 2022-11-18 14:13:53 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:54 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:54 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/host.json 2022-11-18 14:13:54 [debug] Parsed 2 records 2022-11-18 14:13:54 [debug] Requested 3 record fields: resource_id, host_id, hostname 2022-11-18 14:13:54 [debug] No destination_field_map specified 2022-11-18 14:13:54 [debug] Auto-generating destination_field_map using 3 source fields: resource_id, host_id, hostname 2022-11-18 14:13:54 [debug] Available fields for table key 'host': resource_id, host_id, hostname 2022-11-18 14:13:54 [debug] Generated destination_field_map: Table: host resource_id -> resource_id host_id -> host_id hostname -> hostname 2022-11-18 14:13:54 [debug] Processing destination field map 2022-11-18 14:13:54 [debug] Insert SQL for table key 'host': INSERT INTO modw_cloud.host (resource_id, host_id, hostname) VALUES (?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), host_id = COALESCE(VALUES(host_id), host_id), hostname = COALESCE(VALUES(hostname), hostname) 2022-11-18 14:13:54 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:54 [debug] ANALYZE TABLE modw_cloud.host 2022-11-18 14:13:55 [debug] Completed in 0.773040s 2022-11-18 14:13:55 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 2.24 s) 2022-11-18 14:13:55 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780833.122, end_time: 1668780835.3613, elapsed_time: 2.23933, records_examined: 1, records_loaded: 1) 2022-11-18 14:13:55 [info] end (action_name: xdmod.jobs-cloud-common.CloudHostUnknownInitializer, action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:13:56 [info] start (action_name: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer, action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:13:57 [debug] Discover table 'modw_cloud.account' 2022-11-18 14:13:57 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:13:58 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:13:58 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/account.json 2022-11-18 14:13:58 [debug] Parsed 2 records 2022-11-18 14:13:58 [debug] Requested 4 record fields: resource_id, account_id, provider_account, display 2022-11-18 14:13:58 [debug] No destination_field_map specified 2022-11-18 14:13:58 [debug] Auto-generating destination_field_map using 4 source fields: resource_id, account_id, provider_account, display 2022-11-18 14:13:58 [debug] Available fields for table key 'account': resource_id, account_id, provider_account, display, principalinvestigator_person_id, fos_id 2022-11-18 14:13:58 [debug] Generated destination_field_map: Table: account resource_id -> resource_id account_id -> account_id provider_account -> provider_account display -> display 2022-11-18 14:13:58 [debug] Processing destination field map 2022-11-18 14:13:58 [debug] Insert SQL for table key 'account': INSERT INTO modw_cloud.account (resource_id, account_id, provider_account, display) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), account_id = COALESCE(VALUES(account_id), account_id), provider_account = COALESCE(VALUES(provider_account), provider_account), display = COALESCE(VALUES(display), display) 2022-11-18 14:13:58 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:13:58 [debug] ANALYZE TABLE modw_cloud.account 2022-11-18 14:13:58 [debug] Completed in 0.262380s 2022-11-18 14:13:58 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 0.99 s) 2022-11-18 14:14:00 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780837.8918, end_time: 1668780838.88, elapsed_time: 0.98816, records_examined: 1, records_loaded: 1) 2022-11-18 14:14:00 [info] end (action_name: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer, action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:14:00 [info] start (action_name: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer, action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:14:00 [debug] Discover table 'modw_cloud.instance' 2022-11-18 14:14:00 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:14:01 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:14:01 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/instance.json 2022-11-18 14:14:01 [debug] Parsed 2 records 2022-11-18 14:14:01 [debug] Requested 5 record fields: resource_id, instance_id, account_id, provider_identifier, person_id 2022-11-18 14:14:01 [debug] No destination_field_map specified 2022-11-18 14:14:01 [debug] Auto-generating destination_field_map using 5 source fields: resource_id, instance_id, account_id, provider_identifier, person_id 2022-11-18 14:14:01 [debug] Available fields for table key 'instance': resource_id, instance_id, account_id, provider_identifier, person_id, service_provider 2022-11-18 14:14:01 [debug] Generated destination_field_map: Table: instance resource_id -> resource_id instance_id -> instance_id account_id -> account_id provider_identifier -> provider_identifier person_id -> person_id 2022-11-18 14:14:01 [debug] Processing destination field map 2022-11-18 14:14:01 [debug] Insert SQL for table key 'instance': INSERT INTO modw_cloud.instance (resource_id, instance_id, account_id, provider_identifier, person_id) VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), instance_id = COALESCE(VALUES(instance_id), instance_id), account_id = COALESCE(VALUES(account_id), account_id), provider_identifier = COALESCE(VALUES(provider_identifier), provider_identifier), person_id = COALESCE(VALUES(person_id), person_id) 2022-11-18 14:14:01 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:01 [debug] ANALYZE TABLE modw_cloud.instance 2022-11-18 14:14:03 [debug] Completed in 1.652230s 2022-11-18 14:14:03 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 3.08 s) 2022-11-18 14:14:04 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780840.4366, end_time: 1668780843.5179, elapsed_time: 3.08135, records_examined: 1, records_loaded: 1) 2022-11-18 14:14:05 [info] end (action_name: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer, action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:14:05 [info] start (action_name: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer, action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:14:06 [debug] Discover table 'modw_cloud.instance_type' 2022-11-18 14:14:06 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:14:06 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:14:06 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/instance_type.json 2022-11-18 14:14:06 [debug] Parsed 2 records 2022-11-18 14:14:06 [debug] Requested 10 record fields: resource_id, instance_type_id, instance_type, display, description, num_cores, memory_mb, disk_gb, start_time, end_time 2022-11-18 14:14:06 [debug] No destination_field_map specified 2022-11-18 14:14:06 [debug] Auto-generating destination_field_map using 10 source fields: resource_id, instance_type_id, instance_type, display, description, num_cores, memory_mb, disk_gb, start_time, end_time 2022-11-18 14:14:06 [debug] Available fields for table key 'instance_type': resource_id, instance_type_id, instance_type, display, description, num_cores, memory_mb, disk_gb, start_time, end_time 2022-11-18 14:14:06 [debug] Generated destination_field_map: Table: instance_type resource_id -> resource_id instance_type_id -> instance_type_id instance_type -> instance_type display -> display description -> description num_cores -> num_cores memory_mb -> memory_mb disk_gb -> disk_gb start_time -> start_time end_time -> end_time 2022-11-18 14:14:06 [debug] Processing destination field map 2022-11-18 14:14:06 [debug] Insert SQL for table key 'instance_type': INSERT INTO modw_cloud.instance_type (resource_id, instance_type_id, instance_type, display, description, num_cores, memory_mb, disk_gb, start_time, end_time) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), instance_type_id = COALESCE(VALUES(instance_type_id), instance_type_id), instance_type = COALESCE(VALUES(instance_type), instance_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description), num_cores = COALESCE(VALUES(num_cores), num_cores), memory_mb = COALESCE(VALUES(memory_mb), memory_mb), disk_gb = COALESCE(VALUES(disk_gb), disk_gb), start_time = COALESCE(VALUES(start_time), start_time), end_time = COALESCE(VALUES(end_time), end_time) 2022-11-18 14:14:06 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:07 [debug] ANALYZE TABLE modw_cloud.instance_type 2022-11-18 14:14:08 [debug] Completed in 0.485370s 2022-11-18 14:14:08 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 1.56 s) 2022-11-18 14:14:08 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780846.6474, end_time: 1668780848.2059, elapsed_time: 1.55848, records_examined: 1, records_loaded: 1) 2022-11-18 14:14:09 [info] end (action_name: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer, action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:14:10 [info] start (action_name: xdmod.jobs-cloud-common.CloudImageUnknownInitializer, action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:14:11 [debug] Discover table 'modw_cloud.image' 2022-11-18 14:14:11 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:14:11 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:14:11 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/image.json 2022-11-18 14:14:11 [debug] Parsed 2 records 2022-11-18 14:14:11 [debug] Requested 3 record fields: resource_id, image_id, image 2022-11-18 14:14:11 [debug] No destination_field_map specified 2022-11-18 14:14:11 [debug] Auto-generating destination_field_map using 3 source fields: resource_id, image_id, image 2022-11-18 14:14:11 [debug] Available fields for table key 'image': resource_id, image_id, image 2022-11-18 14:14:11 [debug] Generated destination_field_map: Table: image resource_id -> resource_id image_id -> image_id image -> image 2022-11-18 14:14:11 [debug] Processing destination field map 2022-11-18 14:14:11 [debug] Insert SQL for table key 'image': INSERT INTO modw_cloud.image (resource_id, image_id, image) VALUES (?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), image_id = COALESCE(VALUES(image_id), image_id), image = COALESCE(VALUES(image), image) 2022-11-18 14:14:11 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:12 [debug] ANALYZE TABLE modw_cloud.image 2022-11-18 14:14:13 [debug] Completed in 1.102820s 2022-11-18 14:14:13 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 2.37 s) 2022-11-18 14:14:14 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780851.0774, end_time: 1668780853.4469, elapsed_time: 2.36946, records_examined: 1, records_loaded: 1) 2022-11-18 14:14:16 [info] end (action_name: xdmod.jobs-cloud-common.CloudImageUnknownInitializer, action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:14:16 [info] start (action_name: xdmod.jobs-cloud-common.MemoryBucketGenerator, action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:14:17 [debug] Discover table 'modw_cloud.memory_buckets' 2022-11-18 14:14:17 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:14:18 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:14:18 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/memory_buckets.json 2022-11-18 14:14:18 [debug] Parsed 9 records 2022-11-18 14:14:18 [debug] Requested 4 record fields: id, min_memory, max_memory, description 2022-11-18 14:14:18 [debug] No destination_field_map specified 2022-11-18 14:14:18 [debug] Auto-generating destination_field_map using 4 source fields: id, min_memory, max_memory, description 2022-11-18 14:14:18 [debug] Available fields for table key 'memory_buckets': id, min_memory, max_memory, description 2022-11-18 14:14:18 [debug] Generated destination_field_map: Table: memory_buckets id -> id min_memory -> min_memory max_memory -> max_memory description -> description 2022-11-18 14:14:18 [debug] Processing destination field map 2022-11-18 14:14:18 [debug] Insert SQL for table key 'memory_buckets': INSERT INTO modw_cloud.memory_buckets (id, min_memory, max_memory, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE id = COALESCE(VALUES(id), id), min_memory = COALESCE(VALUES(min_memory), min_memory), max_memory = COALESCE(VALUES(max_memory), max_memory), description = COALESCE(VALUES(description), description) 2022-11-18 14:14:18 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:20 [debug] ANALYZE TABLE modw_cloud.memory_buckets 2022-11-18 14:14:20 [debug] Completed in 0.089850s 2022-11-18 14:14:20 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 8 records (Time Taken: 3.10 s) 2022-11-18 14:14:21 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780857.1893, end_time: 1668780860.2895, elapsed_time: 3.10028, records_examined: 8, records_loaded: 8) 2022-11-18 14:14:23 [info] end (action_name: xdmod.jobs-cloud-common.MemoryBucketGenerator, action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:14:24 [info] start (action_name: xdmod.jobs-cloud-common.ProcessorBucketGenerator, action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:14:25 [debug] Discover table 'modw_cloud.processor_buckets' 2022-11-18 14:14:25 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:14:26 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:14:26 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/processor_buckets.json 2022-11-18 14:14:26 [debug] Parsed 8 records 2022-11-18 14:14:26 [debug] Requested 4 record fields: id, min_processors, max_processors, description 2022-11-18 14:14:26 [debug] No destination_field_map specified 2022-11-18 14:14:26 [debug] Auto-generating destination_field_map using 4 source fields: id, min_processors, max_processors, description 2022-11-18 14:14:26 [debug] Available fields for table key 'processor_buckets': id, min_processors, max_processors, description 2022-11-18 14:14:26 [debug] Generated destination_field_map: Table: processor_buckets id -> id min_processors -> min_processors max_processors -> max_processors description -> description 2022-11-18 14:14:26 [debug] Processing destination field map 2022-11-18 14:14:26 [debug] Insert SQL for table key 'processor_buckets': INSERT INTO modw_cloud.processor_buckets (id, min_processors, max_processors, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE id = COALESCE(VALUES(id), id), min_processors = COALESCE(VALUES(min_processors), min_processors), max_processors = COALESCE(VALUES(max_processors), max_processors), description = COALESCE(VALUES(description), description) 2022-11-18 14:14:26 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:26 [debug] ANALYZE TABLE modw_cloud.processor_buckets 2022-11-18 14:14:27 [debug] Completed in 0.616780s 2022-11-18 14:14:27 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 7 records (Time Taken: 2.25 s) 2022-11-18 14:14:27 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668780865.3171, end_time: 1668780867.5712, elapsed_time: 2.25407, records_examined: 7, records_loaded: 7) 2022-11-18 14:14:28 [info] end (action_name: xdmod.jobs-cloud-common.ProcessorBucketGenerator, action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor)) 2022-11-18 14:14:28 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:14:29 [info] Releasing lock file '/tmp/etlv2_16621' 2022-11-18 14:14:30 [debug] Running ETL pipeline "jobs-cloud-ingest-openstack" with parameters {"include-only-resource-codes":"nerc_openstack","variable-overrides":{"CLOUD_EVENT_LOG_DIRECTORY":"\/root\/xdmod_data\/nerc_openstack"}} 2022-11-18 14:14:30 [debug] Loading configuration file /etc/xdmod/etl/etl.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-import.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/acls-import.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/action_state_setup.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/action_state_setup.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [info] ETL\DataEndpoint\DirectoryScanner (name=Open Stack resource specifications, path=${CLOUD_RESOURCE_SPECS_DIRECTORY}): Relative path provided, absolute path recommended 2022-11-18 14:14:30 [debug] Qualifying relative path ${CLOUD_RESOURCE_SPECS_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/gateways.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/gateways.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/ingest_resources.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/ingest_resources.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/jobs.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-11-18 14:14:30 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_common.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_common.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/organizations.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/organizations.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/resource_types.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/resource_types.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/shredder.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/shredder.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/staging.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/staging.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [info] ETL\DataEndpoint\DirectoryScanner (name=usage-directory, path=${STORAGE_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-11-18 14:14:30 [debug] Qualifying relative path ${STORAGE_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/storage.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/storage.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/test_suite.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/test_suite.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/verify.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/verify.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdb.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/xdb.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-11-18 14:14:30 [debug] Parsing /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-11-18 14:14:30 [debug] Parsed 1 records 2022-11-18 14:14:30 [debug] Stored object ETL\Configuration\EtlConfiguration (/etc/xdmod/etl/etl.json) in APCu cache with key ETL\Configuration\EtlConfiguration|/etc/xdmod/etl/etl.json|e65721c298ce9ee9c23d1ed4b2822db1 in 0.247407s 2022-11-18 14:14:30 [debug] Running ETL pipeline with script options {"default-module-name":"xdmod","process-sections":["jobs-cloud-ingest-openstack"],"include-only-resource-codes":"nerc_openstack","variable-overrides":{"CLOUD_EVENT_LOG_DIRECTORY":"\/root\/xdmod_data\/nerc_openstack"}} 2022-11-18 14:14:30 [debug] Loading resource code to id map ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): SELECT id, code from modw.resourcefact 2022-11-18 14:14:30 [info] Verifying endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:32 [info] Verifying endpoint: ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=/root/xdmod_data/nerc_openstack, handler=jsonfile) 2022-11-18 14:14:34 [info] Verifying endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:34 [info] Create action xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:14:34 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_openstack/raw_cloud_job_logs.json 2022-11-18 14:14:34 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_openstack/raw_cloud_job_logs.json 2022-11-18 14:14:34 [debug] Parsed 1 records 2022-11-18 14:14:34 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json' to '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json' 2022-11-18 14:14:34 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json' 2022-11-18 14:14:34 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_openstack/raw_cloud_job_logs.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_openstack/raw_cloud_job_logs.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.004337s 2022-11-18 14:14:34 [info] Verifying action: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-11-18 14:14:34 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:34 [info] Source endpoint: ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=/root/xdmod_data/nerc_openstack, handler=jsonfile) 2022-11-18 14:14:34 [debug] Connecting directory scanner to /root/xdmod_data/nercopenstack 2022-11-18 14:14:34 [debug] Set max recursion depth: 1 2022-11-18 14:14:34 [info] Applying pattern filters: (directory: null, file: /[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.json/) 2022-11-18 14:14:34 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-11-18 14:14:35 [debug] Created ETL destination table object for table definition key 'openstack_raw_event' 2022-11-18 14:14:35 [debug] Created ETL destination table object for table definition key 'openstack_raw_instance_type' 2022-11-18 14:14:35 [info] Empty lock directory specified, using temp directory: /tmp 2022-11-18 14:14:35 [info] Obtaining lock file '/tmp/etlv2_16621' 2022-11-18 14:14:35 [notice] Start processing section 'xdmod.jobs-cloud-ingest-openstack' 2022-11-18 14:14:36 [info] start (action_name: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor, action: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-11-18 14:14:36 [debug] Discover table 'modw_cloud.openstack_raw_event' 2022-11-18 14:14:36 [debug] Discover table 'modw_cloud.openstack_raw_instance_type' 2022-11-18 14:14:36 [info] Process date interval (1/1) (start: none, end: none) 2022-11-18 14:14:36 [debug] Available Variables: CLOUD_EVENT_LOG_DIRECTORY='/root/xdmod_data/nerc_openstack', DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='nobody@massopen.cloud', RESOURCE='nerc_openstack', RESOURCE_ID='2', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-11-18 14:14:36 [debug] Initializing first file iterator: /root/xdmod_data/nerc_openstack/2022-05-01T00:00:00_2022-05-02T00:00:00.json 2022-11-18 14:14:36 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-01T00:00:00_2022-05-02T00:00:00.json 2022-11-18 14:14:36 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-01T00:00:00_2022-05-02T00:00:00.json 2022-11-18 14:14:36 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:36 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:36 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:37 [debug] Parsed 1 records 2022-11-18 14:14:37 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:37 [info] Found 1 records 2022-11-18 14:14:37 [debug] Requested 22 record fields: domain, event_type, generated, host, instance_id, instance_type, project_name, user_name, user_id, resource_id, project_id, request_id, vcpus, memory_mb, disk_gb, state, service, record_type, size, created_at, volume_id, service_provider 2022-11-18 14:14:37 [debug] Processing destination field map 2022-11-18 14:14:37 [debug] Insert SQL for table key 'openstack_raw_event': INSERT INTO modw_cloud.openstack_raw_event (domain, event_type, record_type, event_time_utc, create_time_utc, provider_instance_identifier, hostname, user_name, user_id, resource_id, instance_type, provider_account, project_name, request_id, project_id, openstack_resource_id, disk_gb, num_cores, memory_mb, size, volume_id, state, service_provider) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE domain = COALESCE(VALUES(domain), domain), event_type = COALESCE(VALUES(event_type), event_type), record_type = COALESCE(VALUES(record_type), record_type), event_time_utc = COALESCE(VALUES(event_time_utc), event_time_utc), create_time_utc = COALESCE(VALUES(create_time_utc), create_time_utc), provider_instance_identifier = COALESCE(VALUES(provider_instance_identifier), provider_instance_identifier), hostname = COALESCE(VALUES(hostname), hostname), user_name = COALESCE(VALUES(user_name), user_name), user_id = COALESCE(VALUES(user_id), user_id), resource_id = COALESCE(VALUES(resource_id), resource_id), instance_type = COALESCE(VALUES(instance_type), instance_type), provider_account = COALESCE(VALUES(provider_account), provider_account), project_name = COALESCE(VALUES(project_name), project_name), request_id = COALESCE(VALUES(request_id), request_id), project_id = COALESCE(VALUES(project_id), project_id), openstack_resource_id = COALESCE(VALUES(openstack_resource_id), openstack_resource_id), disk_gb = COALESCE(VALUES(disk_gb), disk_gb), num_cores = COALESCE(VALUES(num_cores), num_cores), memory_mb = COALESCE(VALUES(memory_mb), memory_mb), size = COALESCE(VALUES(size), size), volume_id = COALESCE(VALUES(volume_id), volume_id), state = COALESCE(VALUES(state), state), service_provider = COALESCE(VALUES(service_provider), service_provider)

rob-baron commented 10 months ago

2022-11-18 14:14:37 [debug] Insert SQL for table key 'openstack_raw_instance_type': INSERT INTO modw_cloud.openstack_raw_instance_type (instance_type, resource_id, display, num_cores, memory_mb, disk_gb, start_time) VALUES (?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE instance_type = COALESCE(VALUES(instance_type), instance_type), resource_id = COALESCE(VALUES(resource_id), resource_id), display = COALESCE(VALUES(display), display), num_cores = COALESCE(VALUES(num_cores), num_cores), memory_mb = COALESCE(VALUES(memory_mb), memory_mb), disk_gb = COALESCE(VALUES(disk_gb), disk_gb), start_time = COALESCE(VALUES(start_time), start_time) 2022-11-18 14:14:37 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:37 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-18T00:00:00_2022-10-19T00:00:00.json 2022-11-18 14:14:37 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-18T00:00:00_2022-10-19T00:00:00.json 2022-11-18 14:14:37 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:37 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:37 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:37 [debug] Parsed 12 records 2022-11-18 14:14:37 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:37 [info] Found 12 records 2022-11-18 14:14:40 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:40 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-09T00:00:00_2022-09-10T00:00:00.json 2022-11-18 14:14:40 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-09T00:00:00_2022-09-10T00:00:00.json 2022-11-18 14:14:40 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:40 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:40 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:40 [debug] Parsed 2 records 2022-11-18 14:14:40 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:40 [info] Found 2 records 2022-11-18 14:14:41 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:41 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-02T00:00:00_2022-11-03T00:00:00.json 2022-11-18 14:14:42 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-02T00:00:00_2022-11-03T00:00:00.json 2022-11-18 14:14:42 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:42 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:42 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:42 [debug] Parsed 32 records 2022-11-18 14:14:42 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:42 [info] Found 32 records 2022-11-18 14:14:50 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:50 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-06-01T00:00:00_2022-06-02T00:00:00.json 2022-11-18 14:14:51 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-06-01T00:00:00_2022-06-02T00:00:00.json 2022-11-18 14:14:51 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:51 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:51 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:51 [debug] Parsed 1 records 2022-11-18 14:14:51 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:51 [info] Found 1 records 2022-11-18 14:14:51 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:51 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-04-23T00:00:00_2022-04-24T00:00:00.json 2022-11-18 14:14:52 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-04-23T00:00:00_2022-04-24T00:00:00.json 2022-11-18 14:14:52 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:52 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:52 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:52 [debug] Parsed 1 records 2022-11-18 14:14:52 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:52 [info] Found 1 records 2022-11-18 14:14:52 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:52 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-04-12T00:00:00_2022-04-13T00:00:00.json 2022-11-18 14:14:52 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-04-12T00:00:00_2022-04-13T00:00:00.json 2022-11-18 14:14:52 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:52 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:52 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:52 [debug] Parsed 31 records 2022-11-18 14:14:52 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:52 [info] Found 31 records 2022-11-18 14:14:58 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:58 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-05T00:00:00_2022-10-06T00:00:00.json 2022-11-18 14:14:58 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-05T00:00:00_2022-10-06T00:00:00.json 2022-11-18 14:14:58 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:58 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:58 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:58 [debug] Parsed 10 records 2022-11-18 14:14:58 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:58 [info] Found 10 records 2022-11-18 14:14:59 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:59 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-06-18T00:00:00_2022-06-19T00:00:00.json 2022-11-18 14:14:59 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-06-18T00:00:00_2022-06-19T00:00:00.json 2022-11-18 14:14:59 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:59 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:59 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:00 [debug] Parsed 2 records 2022-11-18 14:15:00 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:00 [info] Found 2 records 2022-11-18 14:15:00 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:00 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-09T00:00:00_2022-03-10T00:00:00.json 2022-11-18 14:15:00 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-09T00:00:00_2022-03-10T00:00:00.json 2022-11-18 14:15:00 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:00 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:00 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:00 [debug] Parsed 4 records 2022-11-18 14:15:00 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:00 [info] Found 4 records 2022-11-18 14:15:04 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:04 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-04T00:00:00_2022-08-05T00:00:00.json 2022-11-18 14:15:06 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-04T00:00:00_2022-08-05T00:00:00.json 2022-11-18 14:15:06 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:06 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:06 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:06 [debug] Parsed 6 records 2022-11-18 14:15:06 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:06 [info] Found 6 records 2022-11-18 14:15:18 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:18 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-26T00:00:00_2022-10-27T00:00:00.json 2022-11-18 14:15:20 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-26T00:00:00_2022-10-27T00:00:00.json 2022-11-18 14:15:20 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:20 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:20 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:20 [debug] Parsed 11 records 2022-11-18 14:15:20 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:20 [info] Found 11 records 2022-11-18 14:15:38 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:38 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-01T00:00:00_2022-09-02T00:00:00.json 2022-11-18 14:15:39 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-01T00:00:00_2022-09-02T00:00:00.json 2022-11-18 14:15:39 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:39 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:39 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:39 [debug] Parsed 2 records 2022-11-18 14:15:39 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:39 [info] Found 2 records 2022-11-18 14:15:46 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:46 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-22T00:00:00_2022-05-23T00:00:00.json 2022-11-18 14:15:47 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-22T00:00:00_2022-05-23T00:00:00.json 2022-11-18 14:15:47 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:47 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:47 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:47 [debug] Parsed 4 records 2022-11-18 14:15:47 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:47 [info] Found 4 records 2022-11-18 14:15:53 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:53 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-17T00:00:00_2022-10-18T00:00:00.json 2022-11-18 14:15:54 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-17T00:00:00_2022-10-18T00:00:00.json 2022-11-18 14:15:54 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:54 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:54 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:54 [debug] Parsed 14 records 2022-11-18 14:15:54 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:54 [info] Found 14 records 2022-11-18 14:16:22 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:16:22 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-05T00:00:00_2022-09-06T00:00:00.json 2022-11-18 14:16:22 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-05T00:00:00_2022-09-06T00:00:00.json 2022-11-18 14:16:22 [debug] Registering filter: xdmod.external_process 2022-11-18 14:16:22 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:16:22 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:16:22 [debug] Parsed 4 records 2022-11-18 14:16:22 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:16:22 [info] Found 4 records 2022-11-18 14:16:30 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:16:30 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-11T00:00:00_2022-02-12T00:00:00.json 2022-11-18 14:16:30 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-11T00:00:00_2022-02-12T00:00:00.json 2022-11-18 14:16:30 [debug] Registering filter: xdmod.external_process 2022-11-18 14:16:30 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:16:30 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:16:31 [debug] Parsed 2 records 2022-11-18 14:16:31 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:16:31 [info] Found 2 records 2022-11-18 14:16:35 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:16:35 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-09T00:00:00_2022-05-10T00:00:00.json 2022-11-18 14:16:37 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-09T00:00:00_2022-05-10T00:00:00.json 2022-11-18 14:16:37 [debug] Registering filter: xdmod.external_process 2022-11-18 14:16:37 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:16:37 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:16:37 [debug] Parsed 1 records 2022-11-18 14:16:37 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:16:37 [info] Found 1 records 2022-11-18 14:16:44 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:16:44 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-22T00:00:00_2022-03-23T00:00:00.json 2022-11-18 14:16:45 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-22T00:00:00_2022-03-23T00:00:00.json 2022-11-18 14:16:45 [debug] Registering filter: xdmod.external_process 2022-11-18 14:16:45 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:16:45 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:16:45 [debug] Parsed 12 records 2022-11-18 14:16:45 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:16:45 [info] Found 12 records 2022-11-18 14:17:06 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:06 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-19T00:00:00_2022-05-20T00:00:00.json 2022-11-18 14:17:07 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-19T00:00:00_2022-05-20T00:00:00.json 2022-11-18 14:17:07 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:07 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:07 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:07 [debug] Parsed 6 records 2022-11-18 14:17:07 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:07 [info] Found 6 records 2022-11-18 14:17:24 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:24 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-25T00:00:00_2022-10-26T00:00:00.json 2022-11-18 14:17:25 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-25T00:00:00_2022-10-26T00:00:00.json 2022-11-18 14:17:25 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:25 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:25 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:25 [debug] Parsed 8 records 2022-11-18 14:17:25 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:25 [info] Found 8 records 2022-11-18 14:17:32 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:32 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-27T00:00:00_2022-10-28T00:00:00.json 2022-11-18 14:17:32 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-27T00:00:00_2022-10-28T00:00:00.json 2022-11-18 14:17:32 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:32 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:32 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:32 [debug] Parsed 13 records 2022-11-18 14:17:32 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:32 [info] Found 13 records 2022-11-18 14:17:45 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:45 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-06-27T00:00:00_2022-06-28T00:00:00.json 2022-11-18 14:17:45 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-06-27T00:00:00_2022-06-28T00:00:00.json 2022-11-18 14:17:45 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:45 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:45 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:45 [debug] Parsed 4 records 2022-11-18 14:17:45 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:45 [info] Found 4 records 2022-11-18 14:17:47 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:47 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-16T00:00:00_2022-07-17T00:00:00.json 2022-11-18 14:17:49 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-16T00:00:00_2022-07-17T00:00:00.json 2022-11-18 14:17:49 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:49 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:49 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:49 [debug] Parsed 6 records 2022-11-18 14:17:49 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:49 [info] Found 6 records 2022-11-18 14:17:53 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:53 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-24T00:00:00_2022-03-25T00:00:00.json 2022-11-18 14:17:53 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-24T00:00:00_2022-03-25T00:00:00.json 2022-11-18 14:17:53 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:53 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:53 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:53 [debug] Parsed 5 records 2022-11-18 14:17:53 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:53 [info] Found 5 records 2022-11-18 14:17:59 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:59 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-01T00:00:00_2022-11-02T00:00:00.json 2022-11-18 14:17:59 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-01T00:00:00_2022-11-02T00:00:00.json 2022-11-18 14:17:59 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:59 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:59 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:59 [debug] Parsed 2 records 2022-11-18 14:17:59 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:59 [info] Found 2 records 2022-11-18 14:18:01 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:01 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-02T00:00:00_2022-05-03T00:00:00.json 2022-11-18 14:18:02 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-02T00:00:00_2022-05-03T00:00:00.json 2022-11-18 14:18:02 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:02 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:02 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:02 [debug] Parsed 21 records 2022-11-18 14:18:02 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:02 [info] Found 21 records 2022-11-18 14:18:09 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:09 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-12T00:00:00_2022-07-13T00:00:00.json 2022-11-18 14:18:10 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-12T00:00:00_2022-07-13T00:00:00.json 2022-11-18 14:18:10 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:10 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:10 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:11 [debug] Parsed 70 records 2022-11-18 14:18:11 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:11 [info] Found 70 records 2022-11-18 14:18:33 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:33 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-14T00:00:00_2022-07-15T00:00:00.json 2022-11-18 14:18:33 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-14T00:00:00_2022-07-15T00:00:00.json 2022-11-18 14:18:33 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:33 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:33 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:33 [debug] Parsed 2 records 2022-11-18 14:18:33 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:33 [info] Found 2 records 2022-11-18 14:18:35 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:35 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-07T00:00:00_2022-10-08T00:00:00.json 2022-11-18 14:18:36 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-07T00:00:00_2022-10-08T00:00:00.json 2022-11-18 14:18:36 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:36 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:36 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:36 [debug] Parsed 6 records 2022-11-18 14:18:36 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:36 [info] Found 6 records 2022-11-18 14:18:38 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:38 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-07T00:00:00_2022-11-08T00:00:00.json 2022-11-18 14:18:40 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-07T00:00:00_2022-11-08T00:00:00.json 2022-11-18 14:18:40 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:40 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:40 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:40 [debug] Parsed 96 records 2022-11-18 14:18:40 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:40 [info] Found 96 records 2022-11-18 14:19:09 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:09 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-29T00:00:00_2022-07-30T00:00:00.json 2022-11-18 14:19:09 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-29T00:00:00_2022-07-30T00:00:00.json 2022-11-18 14:19:09 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:09 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:09 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:09 [debug] Parsed 10 records 2022-11-18 14:19:09 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:09 [info] Found 10 records 2022-11-18 14:19:17 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:17 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-08T00:00:00_2022-02-09T00:00:00.json 2022-11-18 14:19:18 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-08T00:00:00_2022-02-09T00:00:00.json 2022-11-18 14:19:18 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:18 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:18 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:18 [debug] Parsed 12 records 2022-11-18 14:19:18 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:18 [info] Found 12 records 2022-11-18 14:19:25 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:25 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-30T00:00:00_2022-08-31T00:00:00.json 2022-11-18 14:19:26 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-30T00:00:00_2022-08-31T00:00:00.json 2022-11-18 14:19:26 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:26 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:26 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:26 [debug] Parsed 1 records 2022-11-18 14:19:26 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:26 [info] Found 1 records 2022-11-18 14:19:27 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:27 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-18T00:00:00_2022-03-19T00:00:00.json 2022-11-18 14:19:27 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-18T00:00:00_2022-03-19T00:00:00.json 2022-11-18 14:19:27 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:27 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:27 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:27 [debug] Parsed 1 records 2022-11-18 14:19:27 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:27 [info] Found 1 records 2022-11-18 14:19:30 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:30 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-20T00:00:00_2022-07-21T00:00:00.json 2022-11-18 14:19:30 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-20T00:00:00_2022-07-21T00:00:00.json 2022-11-18 14:19:30 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:30 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:30 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:31 [debug] Parsed 2 records 2022-11-18 14:19:31 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:31 [info] Found 2 records 2022-11-18 14:19:33 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:33 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-23T00:00:00_2022-09-24T00:00:00.json 2022-11-18 14:19:33 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-23T00:00:00_2022-09-24T00:00:00.json 2022-11-18 14:19:33 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:33 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:33 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:33 [debug] Parsed 6 records 2022-11-18 14:19:33 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:33 [info] Found 6 records 2022-11-18 14:19:37 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:37 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-04-11T00:00:00_2022-04-12T00:00:00.json 2022-11-18 14:19:37 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-04-11T00:00:00_2022-04-12T00:00:00.json 2022-11-18 14:19:37 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:37 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:37 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:37 [debug] Parsed 12 records 2022-11-18 14:19:37 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:37 [info] Found 12 records 2022-11-18 14:19:50 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:50 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-18T00:00:00_2022-07-19T00:00:00.json 2022-11-18 14:19:50 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-18T00:00:00_2022-07-19T00:00:00.json 2022-11-18 14:19:50 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:50 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:50 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:50 [debug] Parsed 2 records 2022-11-18 14:19:50 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:50 [info] Found 2 records 2022-11-18 14:19:52 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:52 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-19T00:00:00_2022-07-20T00:00:00.json 2022-11-18 14:19:53 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-19T00:00:00_2022-07-20T00:00:00.json 2022-11-18 14:19:53 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:53 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:53 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:53 [debug] Parsed 4 records 2022-11-18 14:19:53 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:53 [info] Found 4 records 2022-11-18 14:19:55 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:55 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-29T00:00:00_2022-09-30T00:00:00.json 2022-11-18 14:19:56 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-29T00:00:00_2022-09-30T00:00:00.json 2022-11-18 14:19:56 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:56 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:56 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:56 [debug] Parsed 18 records 2022-11-18 14:19:56 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:56 [info] Found 18 records 2022-11-18 14:20:14 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:14 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-25T00:00:00_2022-07-26T00:00:00.json 2022-11-18 14:20:15 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-25T00:00:00_2022-07-26T00:00:00.json 2022-11-18 14:20:15 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:15 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:15 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:16 [debug] Parsed 8 records 2022-11-18 14:20:16 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:16 [info] Found 8 records 2022-11-18 14:20:25 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:25 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-04T00:00:00_2022-03-05T00:00:00.json 2022-11-18 14:20:25 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-04T00:00:00_2022-03-05T00:00:00.json 2022-11-18 14:20:25 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:25 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:25 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:25 [debug] Parsed 4 records 2022-11-18 14:20:25 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:25 [info] Found 4 records 2022-11-18 14:20:28 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:28 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-05T00:00:00_2022-08-06T00:00:00.json 2022-11-18 14:20:29 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-05T00:00:00_2022-08-06T00:00:00.json 2022-11-18 14:20:29 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:29 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:29 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:29 [debug] Parsed 4 records 2022-11-18 14:20:29 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:29 [info] Found 4 records 2022-11-18 14:20:31 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:31 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-04-06T00:00:00_2022-04-07T00:00:00.json 2022-11-18 14:20:32 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-04-06T00:00:00_2022-04-07T00:00:00.json 2022-11-18 14:20:32 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:32 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:32 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:32 [debug] Parsed 6 records 2022-11-18 14:20:32 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:32 [info] Found 6 records 2022-11-18 14:20:35 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:35 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-20T00:00:00_2022-10-21T00:00:00.json 2022-11-18 14:20:36 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-20T00:00:00_2022-10-21T00:00:00.json 2022-11-18 14:20:36 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:36 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:36 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:36 [debug] Parsed 2 records 2022-11-18 14:20:36 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:36 [info] Found 2 records 2022-11-18 14:20:37 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:37 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-28T00:00:00_2022-10-29T00:00:00.json 2022-11-18 14:20:39 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-28T00:00:00_2022-10-29T00:00:00.json 2022-11-18 14:20:39 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:39 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:39 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:39 [debug] Parsed 6 records 2022-11-18 14:20:39 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:39 [info] Found 6 records 2022-11-18 14:20:41 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:41 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-05T00:00:00_2022-07-06T00:00:00.json 2022-11-18 14:20:41 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-05T00:00:00_2022-07-06T00:00:00.json 2022-11-18 14:20:41 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:41 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:41 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:41 [debug] Parsed 2 records 2022-11-18 14:20:41 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:41 [info] Found 2 records 2022-11-18 14:20:42 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:42 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-24T00:00:00_2022-08-25T00:00:00.json 2022-11-18 14:20:43 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-24T00:00:00_2022-08-25T00:00:00.json 2022-11-18 14:20:43 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:43 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:43 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:43 [debug] Parsed 87 records 2022-11-18 14:20:43 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:43 [info] Found 87 records 2022-11-18 14:21:08 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:21:08 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-27T00:00:00_2022-05-28T00:00:00.json 2022-11-18 14:21:08 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-27T00:00:00_2022-05-28T00:00:00.json 2022-11-18 14:21:08 [debug] Registering filter: xdmod.external_process 2022-11-18 14:21:08 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:21:08 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:21:08 [debug] Parsed 2 records 2022-11-18 14:21:08 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:21:08 [info] Found 2 records 2022-11-18 14:21:09 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:21:09 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-08T00:00:00_2022-11-09T00:00:00.json 2022-11-18 14:21:11 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-08T00:00:00_2022-11-09T00:00:00.json 2022-11-18 14:21:11 [debug] Registering filter: xdmod.external_process 2022-11-18 14:21:11 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:21:11 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:21:11 [debug] Parsed 30 records 2022-11-18 14:21:11 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:21:11 [info] Found 30 records 2022-11-18 14:21:49 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:21:49 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-10T00:00:00_2022-02-11T00:00:00.json 2022-11-18 14:21:51 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-10T00:00:00_2022-02-11T00:00:00.json 2022-11-18 14:21:51 [debug] Registering filter: xdmod.external_process 2022-11-18 14:21:51 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:21:51 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:21:51 [debug] Parsed 6 records 2022-11-18 14:21:51 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:21:51 [info] Found 6 records 2022-11-18 14:22:00 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:00 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-24T00:00:00_2022-02-25T00:00:00.json 2022-11-18 14:22:01 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-24T00:00:00_2022-02-25T00:00:00.json 2022-11-18 14:22:01 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:01 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:01 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:01 [debug] Parsed 10 records 2022-11-18 14:22:01 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:01 [info] Found 10 records 2022-11-18 14:22:20 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:20 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-17T00:00:00_2022-07-18T00:00:00.json 2022-11-18 14:22:21 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-17T00:00:00_2022-07-18T00:00:00.json 2022-11-18 14:22:21 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:21 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:21 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:21 [debug] Parsed 2 records 2022-11-18 14:22:21 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:21 [info] Found 2 records 2022-11-18 14:22:25 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:25 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-30T00:00:00_2022-03-31T00:00:00.json 2022-11-18 14:22:26 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-30T00:00:00_2022-03-31T00:00:00.json 2022-11-18 14:22:26 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:26 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:26 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:26 [debug] Parsed 4 records 2022-11-18 14:22:26 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:26 [info] Found 4 records 2022-11-18 14:22:34 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:34 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-23T00:00:00_2022-02-24T00:00:00.json 2022-11-18 14:22:35 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-23T00:00:00_2022-02-24T00:00:00.json 2022-11-18 14:22:35 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:35 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:35 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:35 [debug] Parsed 13 records 2022-11-18 14:22:35 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:35 [info] Found 13 records 2022-11-18 14:22:56 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:56 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-17T00:00:00_2022-03-18T00:00:00.json 2022-11-18 14:22:58 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-17T00:00:00_2022-03-18T00:00:00.json 2022-11-18 14:22:58 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:58 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:58 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:58 [debug] Parsed 2 records 2022-11-18 14:22:58 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:58 [info] Found 2 records 2022-11-18 14:23:00 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:00 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-25T00:00:00_2022-03-26T00:00:00.json 2022-11-18 14:23:00 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-25T00:00:00_2022-03-26T00:00:00.json 2022-11-18 14:23:00 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:00 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:00 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:00 [debug] Parsed 1 records 2022-11-18 14:23:00 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:00 [info] Found 1 records 2022-11-18 14:23:02 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:02 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-23T00:00:00_2022-03-24T00:00:00.json 2022-11-18 14:23:02 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-23T00:00:00_2022-03-24T00:00:00.json 2022-11-18 14:23:02 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:02 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:02 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:02 [debug] Parsed 14 records 2022-11-18 14:23:02 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:02 [info] Found 14 records 2022-11-18 14:23:19 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:19 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-22T00:00:00_2022-02-23T00:00:00.json 2022-11-18 14:23:19 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-22T00:00:00_2022-02-23T00:00:00.json 2022-11-18 14:23:19 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:19 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:19 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:19 [debug] Parsed 6 records 2022-11-18 14:23:19 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:19 [info] Found 6 records 2022-11-18 14:23:22 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:22 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-23T00:00:00_2022-07-24T00:00:00.json 2022-11-18 14:23:23 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-23T00:00:00_2022-07-24T00:00:00.json 2022-11-18 14:23:23 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:23 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:23 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:23 [debug] Parsed 16 records 2022-11-18 14:23:23 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:23 [info] Found 16 records 2022-11-18 14:23:29 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:29 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-04T00:00:00_2022-11-05T00:00:00.json 2022-11-18 14:23:30 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-04T00:00:00_2022-11-05T00:00:00.json 2022-11-18 14:23:30 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:30 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:30 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:30 [debug] Parsed 10 records 2022-11-18 14:23:30 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:30 [info] Found 10 records 2022-11-18 14:23:34 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:34 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-04T00:00:00_2022-10-05T00:00:00.json 2022-11-18 14:23:35 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-04T00:00:00_2022-10-05T00:00:00.json 2022-11-18 14:23:35 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:35 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:35 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:35 [debug] Parsed 14 records 2022-11-18 14:23:35 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:35 [info] Found 14 records 2022-11-18 14:23:43 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:43 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-08T00:00:00_2022-09-09T00:00:00.json 2022-11-18 14:23:44 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-08T00:00:00_2022-09-09T00:00:00.json 2022-11-18 14:23:44 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:44 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:44 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:44 [debug] Parsed 2 records 2022-11-18 14:23:44 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:44 [info] Found 2 records 2022-11-18 14:23:44 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:44 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-13T00:00:00_2022-08-14T00:00:00.json 2022-11-18 14:23:45 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-13T00:00:00_2022-08-14T00:00:00.json 2022-11-18 14:23:45 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:45 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:45 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:45 [debug] Parsed 2 records 2022-11-18 14:23:45 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:45 [info] Found 2 records 2022-11-18 14:23:47 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:47 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-01-26T00:00:00_2022-01-27T00:00:00.json 2022-11-18 14:23:47 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-01-26T00:00:00_2022-01-27T00:00:00.json 2022-11-18 14:23:47 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:47 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:47 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:47 [debug] Parsed 4 records 2022-11-18 14:23:47 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:47 [info] Found 4 records 2022-11-18 14:23:48 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:48 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-11T00:00:00_2022-08-12T00:00:00.json 2022-11-18 14:23:49 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-11T00:00:00_2022-08-12T00:00:00.json 2022-11-18 14:23:49 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:49 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:49 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:49 [debug] Parsed 6 records 2022-11-18 14:23:49 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:49 [info] Found 6 records 2022-11-18 14:23:50 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:50 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-01T00:00:00_2022-07-02T00:00:00.json 2022-11-18 14:23:50 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-01T00:00:00_2022-07-02T00:00:00.json 2022-11-18 14:23:50 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:50 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:50 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:50 [debug] Parsed 2 records 2022-11-18 14:23:50 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:50 [info] Found 2 records 2022-11-18 14:23:51 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:51 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-13T00:00:00_2022-07-14T00:00:00.json 2022-11-18 14:23:53 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-13T00:00:00_2022-07-14T00:00:00.json 2022-11-18 14:23:53 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:53 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:53 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:53 [debug] Parsed 76 records 2022-11-18 14:23:53 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:53 [info] Found 76 records 2022-11-18 14:24:34 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:24:34 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-14T00:00:00_2022-10-15T00:00:00.json 2022-11-18 14:24:35 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-14T00:00:00_2022-10-15T00:00:00.json 2022-11-18 14:24:35 [debug] Registering filter: xdmod.external_process

rob-baron commented 10 months ago

2022-11-18 14:14:37 [debug] Insert SQL for table key 'openstack_raw_instance_type': INSERT INTO modw_cloud.openstack_raw_instance_type (instance_type, resource_id, display, num_cores, memory_mb, disk_gb, start_time) VALUES (?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE instance_type = COALESCE(VALUES(instance_type), instance_type), resource_id = COALESCE(VALUES(resource_id), resource_id), display = COALESCE(VALUES(display), display), num_cores = COALESCE(VALUES(num_cores), num_cores), memory_mb = COALESCE(VALUES(memory_mb), memory_mb), disk_gb = COALESCE(VALUES(disk_gb), disk_gb), start_time = COALESCE(VALUES(start_time), start_time) 2022-11-18 14:14:37 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:37 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-18T00:00:00_2022-10-19T00:00:00.json 2022-11-18 14:14:37 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-18T00:00:00_2022-10-19T00:00:00.json 2022-11-18 14:14:37 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:37 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:37 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:37 [debug] Parsed 12 records 2022-11-18 14:14:37 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:37 [info] Found 12 records 2022-11-18 14:14:40 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:40 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-09T00:00:00_2022-09-10T00:00:00.json 2022-11-18 14:14:40 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-09T00:00:00_2022-09-10T00:00:00.json 2022-11-18 14:14:40 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:40 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:40 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:40 [debug] Parsed 2 records 2022-11-18 14:14:40 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:40 [info] Found 2 records 2022-11-18 14:14:41 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:41 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-02T00:00:00_2022-11-03T00:00:00.json 2022-11-18 14:14:42 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-02T00:00:00_2022-11-03T00:00:00.json 2022-11-18 14:14:42 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:42 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:42 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:42 [debug] Parsed 32 records 2022-11-18 14:14:42 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:42 [info] Found 32 records 2022-11-18 14:14:50 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:50 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-06-01T00:00:00_2022-06-02T00:00:00.json 2022-11-18 14:14:51 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-06-01T00:00:00_2022-06-02T00:00:00.json 2022-11-18 14:14:51 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:51 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:51 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:51 [debug] Parsed 1 records 2022-11-18 14:14:51 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:51 [info] Found 1 records 2022-11-18 14:14:51 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:51 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-04-23T00:00:00_2022-04-24T00:00:00.json 2022-11-18 14:14:52 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-04-23T00:00:00_2022-04-24T00:00:00.json 2022-11-18 14:14:52 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:52 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:52 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:52 [debug] Parsed 1 records 2022-11-18 14:14:52 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:52 [info] Found 1 records 2022-11-18 14:14:52 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:52 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-04-12T00:00:00_2022-04-13T00:00:00.json 2022-11-18 14:14:52 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-04-12T00:00:00_2022-04-13T00:00:00.json 2022-11-18 14:14:52 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:52 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:52 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:52 [debug] Parsed 31 records 2022-11-18 14:14:52 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:52 [info] Found 31 records 2022-11-18 14:14:58 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:58 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-05T00:00:00_2022-10-06T00:00:00.json 2022-11-18 14:14:58 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-05T00:00:00_2022-10-06T00:00:00.json 2022-11-18 14:14:58 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:58 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:58 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:14:58 [debug] Parsed 10 records 2022-11-18 14:14:58 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:14:58 [info] Found 10 records 2022-11-18 14:14:59 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:14:59 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-06-18T00:00:00_2022-06-19T00:00:00.json 2022-11-18 14:14:59 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-06-18T00:00:00_2022-06-19T00:00:00.json 2022-11-18 14:14:59 [debug] Registering filter: xdmod.external_process 2022-11-18 14:14:59 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:14:59 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:00 [debug] Parsed 2 records 2022-11-18 14:15:00 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:00 [info] Found 2 records 2022-11-18 14:15:00 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:00 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-09T00:00:00_2022-03-10T00:00:00.json 2022-11-18 14:15:00 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-09T00:00:00_2022-03-10T00:00:00.json 2022-11-18 14:15:00 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:00 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:00 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:00 [debug] Parsed 4 records 2022-11-18 14:15:00 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:00 [info] Found 4 records 2022-11-18 14:15:04 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:04 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-04T00:00:00_2022-08-05T00:00:00.json 2022-11-18 14:15:06 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-04T00:00:00_2022-08-05T00:00:00.json 2022-11-18 14:15:06 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:06 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:06 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:06 [debug] Parsed 6 records 2022-11-18 14:15:06 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:06 [info] Found 6 records 2022-11-18 14:15:18 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:18 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-26T00:00:00_2022-10-27T00:00:00.json 2022-11-18 14:15:20 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-26T00:00:00_2022-10-27T00:00:00.json 2022-11-18 14:15:20 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:20 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:20 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:20 [debug] Parsed 11 records 2022-11-18 14:15:20 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:20 [info] Found 11 records 2022-11-18 14:15:38 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:38 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-01T00:00:00_2022-09-02T00:00:00.json 2022-11-18 14:15:39 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-01T00:00:00_2022-09-02T00:00:00.json 2022-11-18 14:15:39 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:39 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:39 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:39 [debug] Parsed 2 records 2022-11-18 14:15:39 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:39 [info] Found 2 records 2022-11-18 14:15:46 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:46 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-22T00:00:00_2022-05-23T00:00:00.json 2022-11-18 14:15:47 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-22T00:00:00_2022-05-23T00:00:00.json 2022-11-18 14:15:47 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:47 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:47 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:47 [debug] Parsed 4 records 2022-11-18 14:15:47 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:47 [info] Found 4 records 2022-11-18 14:15:53 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:15:53 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-17T00:00:00_2022-10-18T00:00:00.json 2022-11-18 14:15:54 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-17T00:00:00_2022-10-18T00:00:00.json 2022-11-18 14:15:54 [debug] Registering filter: xdmod.external_process 2022-11-18 14:15:54 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:15:54 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:15:54 [debug] Parsed 14 records 2022-11-18 14:15:54 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:15:54 [info] Found 14 records 2022-11-18 14:16:22 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:16:22 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-05T00:00:00_2022-09-06T00:00:00.json 2022-11-18 14:16:22 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-05T00:00:00_2022-09-06T00:00:00.json 2022-11-18 14:16:22 [debug] Registering filter: xdmod.external_process 2022-11-18 14:16:22 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:16:22 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:16:22 [debug] Parsed 4 records 2022-11-18 14:16:22 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:16:22 [info] Found 4 records 2022-11-18 14:16:30 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:16:30 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-11T00:00:00_2022-02-12T00:00:00.json 2022-11-18 14:16:30 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-11T00:00:00_2022-02-12T00:00:00.json 2022-11-18 14:16:30 [debug] Registering filter: xdmod.external_process 2022-11-18 14:16:30 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:16:30 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:16:31 [debug] Parsed 2 records 2022-11-18 14:16:31 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:16:31 [info] Found 2 records 2022-11-18 14:16:35 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:16:35 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-09T00:00:00_2022-05-10T00:00:00.json 2022-11-18 14:16:37 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-09T00:00:00_2022-05-10T00:00:00.json 2022-11-18 14:16:37 [debug] Registering filter: xdmod.external_process 2022-11-18 14:16:37 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:16:37 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:16:37 [debug] Parsed 1 records 2022-11-18 14:16:37 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:16:37 [info] Found 1 records 2022-11-18 14:16:44 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:16:44 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-22T00:00:00_2022-03-23T00:00:00.json 2022-11-18 14:16:45 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-22T00:00:00_2022-03-23T00:00:00.json 2022-11-18 14:16:45 [debug] Registering filter: xdmod.external_process 2022-11-18 14:16:45 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:16:45 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:16:45 [debug] Parsed 12 records 2022-11-18 14:16:45 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:16:45 [info] Found 12 records 2022-11-18 14:17:06 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:06 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-19T00:00:00_2022-05-20T00:00:00.json 2022-11-18 14:17:07 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-19T00:00:00_2022-05-20T00:00:00.json 2022-11-18 14:17:07 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:07 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:07 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:07 [debug] Parsed 6 records 2022-11-18 14:17:07 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:07 [info] Found 6 records 2022-11-18 14:17:24 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:24 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-25T00:00:00_2022-10-26T00:00:00.json 2022-11-18 14:17:25 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-25T00:00:00_2022-10-26T00:00:00.json 2022-11-18 14:17:25 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:25 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:25 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:25 [debug] Parsed 8 records 2022-11-18 14:17:25 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:25 [info] Found 8 records 2022-11-18 14:17:32 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:32 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-27T00:00:00_2022-10-28T00:00:00.json 2022-11-18 14:17:32 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-27T00:00:00_2022-10-28T00:00:00.json 2022-11-18 14:17:32 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:32 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:32 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:32 [debug] Parsed 13 records 2022-11-18 14:17:32 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:32 [info] Found 13 records 2022-11-18 14:17:45 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:45 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-06-27T00:00:00_2022-06-28T00:00:00.json 2022-11-18 14:17:45 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-06-27T00:00:00_2022-06-28T00:00:00.json 2022-11-18 14:17:45 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:45 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:45 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:45 [debug] Parsed 4 records 2022-11-18 14:17:45 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:45 [info] Found 4 records 2022-11-18 14:17:47 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:47 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-16T00:00:00_2022-07-17T00:00:00.json 2022-11-18 14:17:49 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-16T00:00:00_2022-07-17T00:00:00.json 2022-11-18 14:17:49 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:49 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:49 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:49 [debug] Parsed 6 records 2022-11-18 14:17:49 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:49 [info] Found 6 records 2022-11-18 14:17:53 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:53 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-24T00:00:00_2022-03-25T00:00:00.json 2022-11-18 14:17:53 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-24T00:00:00_2022-03-25T00:00:00.json 2022-11-18 14:17:53 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:53 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:53 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:53 [debug] Parsed 5 records 2022-11-18 14:17:53 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:53 [info] Found 5 records 2022-11-18 14:17:59 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:17:59 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-01T00:00:00_2022-11-02T00:00:00.json 2022-11-18 14:17:59 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-01T00:00:00_2022-11-02T00:00:00.json 2022-11-18 14:17:59 [debug] Registering filter: xdmod.external_process 2022-11-18 14:17:59 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:17:59 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:17:59 [debug] Parsed 2 records 2022-11-18 14:17:59 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:17:59 [info] Found 2 records 2022-11-18 14:18:01 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:01 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-02T00:00:00_2022-05-03T00:00:00.json 2022-11-18 14:18:02 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-02T00:00:00_2022-05-03T00:00:00.json 2022-11-18 14:18:02 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:02 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:02 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:02 [debug] Parsed 21 records 2022-11-18 14:18:02 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:02 [info] Found 21 records 2022-11-18 14:18:09 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:09 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-12T00:00:00_2022-07-13T00:00:00.json 2022-11-18 14:18:10 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-12T00:00:00_2022-07-13T00:00:00.json 2022-11-18 14:18:10 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:10 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:10 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:11 [debug] Parsed 70 records 2022-11-18 14:18:11 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:11 [info] Found 70 records 2022-11-18 14:18:33 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:33 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-14T00:00:00_2022-07-15T00:00:00.json 2022-11-18 14:18:33 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-14T00:00:00_2022-07-15T00:00:00.json 2022-11-18 14:18:33 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:33 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:33 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:33 [debug] Parsed 2 records 2022-11-18 14:18:33 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:33 [info] Found 2 records 2022-11-18 14:18:35 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:35 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-07T00:00:00_2022-10-08T00:00:00.json 2022-11-18 14:18:36 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-07T00:00:00_2022-10-08T00:00:00.json 2022-11-18 14:18:36 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:36 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:36 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:36 [debug] Parsed 6 records 2022-11-18 14:18:36 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:36 [info] Found 6 records 2022-11-18 14:18:38 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:18:38 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-07T00:00:00_2022-11-08T00:00:00.json 2022-11-18 14:18:40 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-07T00:00:00_2022-11-08T00:00:00.json 2022-11-18 14:18:40 [debug] Registering filter: xdmod.external_process 2022-11-18 14:18:40 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:18:40 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:18:40 [debug] Parsed 96 records 2022-11-18 14:18:40 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:18:40 [info] Found 96 records 2022-11-18 14:19:09 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:09 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-29T00:00:00_2022-07-30T00:00:00.json 2022-11-18 14:19:09 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-29T00:00:00_2022-07-30T00:00:00.json 2022-11-18 14:19:09 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:09 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:09 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:09 [debug] Parsed 10 records 2022-11-18 14:19:09 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:09 [info] Found 10 records 2022-11-18 14:19:17 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:17 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-08T00:00:00_2022-02-09T00:00:00.json 2022-11-18 14:19:18 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-08T00:00:00_2022-02-09T00:00:00.json 2022-11-18 14:19:18 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:18 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:18 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:18 [debug] Parsed 12 records 2022-11-18 14:19:18 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:18 [info] Found 12 records 2022-11-18 14:19:25 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:25 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-30T00:00:00_2022-08-31T00:00:00.json 2022-11-18 14:19:26 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-30T00:00:00_2022-08-31T00:00:00.json 2022-11-18 14:19:26 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:26 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:26 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:26 [debug] Parsed 1 records 2022-11-18 14:19:26 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:26 [info] Found 1 records 2022-11-18 14:19:27 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:27 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-18T00:00:00_2022-03-19T00:00:00.json 2022-11-18 14:19:27 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-18T00:00:00_2022-03-19T00:00:00.json 2022-11-18 14:19:27 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:27 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:27 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:27 [debug] Parsed 1 records 2022-11-18 14:19:27 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:27 [info] Found 1 records 2022-11-18 14:19:30 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:30 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-20T00:00:00_2022-07-21T00:00:00.json 2022-11-18 14:19:30 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-20T00:00:00_2022-07-21T00:00:00.json 2022-11-18 14:19:30 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:30 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:30 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:31 [debug] Parsed 2 records 2022-11-18 14:19:31 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:31 [info] Found 2 records 2022-11-18 14:19:33 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:33 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-23T00:00:00_2022-09-24T00:00:00.json 2022-11-18 14:19:33 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-23T00:00:00_2022-09-24T00:00:00.json 2022-11-18 14:19:33 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:33 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:33 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:33 [debug] Parsed 6 records 2022-11-18 14:19:33 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:33 [info] Found 6 records 2022-11-18 14:19:37 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:37 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-04-11T00:00:00_2022-04-12T00:00:00.json 2022-11-18 14:19:37 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-04-11T00:00:00_2022-04-12T00:00:00.json 2022-11-18 14:19:37 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:37 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:37 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:37 [debug] Parsed 12 records 2022-11-18 14:19:37 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:37 [info] Found 12 records 2022-11-18 14:19:50 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:50 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-18T00:00:00_2022-07-19T00:00:00.json 2022-11-18 14:19:50 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-18T00:00:00_2022-07-19T00:00:00.json 2022-11-18 14:19:50 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:50 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:50 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:50 [debug] Parsed 2 records 2022-11-18 14:19:50 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:50 [info] Found 2 records 2022-11-18 14:19:52 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:52 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-19T00:00:00_2022-07-20T00:00:00.json 2022-11-18 14:19:53 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-19T00:00:00_2022-07-20T00:00:00.json 2022-11-18 14:19:53 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:53 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:53 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:53 [debug] Parsed 4 records 2022-11-18 14:19:53 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:53 [info] Found 4 records 2022-11-18 14:19:55 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:19:55 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-29T00:00:00_2022-09-30T00:00:00.json 2022-11-18 14:19:56 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-29T00:00:00_2022-09-30T00:00:00.json 2022-11-18 14:19:56 [debug] Registering filter: xdmod.external_process 2022-11-18 14:19:56 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:19:56 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:19:56 [debug] Parsed 18 records 2022-11-18 14:19:56 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:19:56 [info] Found 18 records 2022-11-18 14:20:14 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:14 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-25T00:00:00_2022-07-26T00:00:00.json 2022-11-18 14:20:15 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-25T00:00:00_2022-07-26T00:00:00.json 2022-11-18 14:20:15 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:15 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:15 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:16 [debug] Parsed 8 records 2022-11-18 14:20:16 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:16 [info] Found 8 records 2022-11-18 14:20:25 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:25 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-04T00:00:00_2022-03-05T00:00:00.json 2022-11-18 14:20:25 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-04T00:00:00_2022-03-05T00:00:00.json 2022-11-18 14:20:25 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:25 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:25 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:25 [debug] Parsed 4 records 2022-11-18 14:20:25 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:25 [info] Found 4 records 2022-11-18 14:20:28 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:28 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-05T00:00:00_2022-08-06T00:00:00.json 2022-11-18 14:20:29 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-05T00:00:00_2022-08-06T00:00:00.json 2022-11-18 14:20:29 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:29 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:29 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:29 [debug] Parsed 4 records 2022-11-18 14:20:29 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:29 [info] Found 4 records 2022-11-18 14:20:31 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:31 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-04-06T00:00:00_2022-04-07T00:00:00.json 2022-11-18 14:20:32 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-04-06T00:00:00_2022-04-07T00:00:00.json 2022-11-18 14:20:32 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:32 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:32 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:32 [debug] Parsed 6 records 2022-11-18 14:20:32 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:32 [info] Found 6 records 2022-11-18 14:20:35 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:35 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-20T00:00:00_2022-10-21T00:00:00.json 2022-11-18 14:20:36 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-20T00:00:00_2022-10-21T00:00:00.json 2022-11-18 14:20:36 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:36 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:36 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:36 [debug] Parsed 2 records 2022-11-18 14:20:36 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:36 [info] Found 2 records 2022-11-18 14:20:37 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:37 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-28T00:00:00_2022-10-29T00:00:00.json 2022-11-18 14:20:39 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-28T00:00:00_2022-10-29T00:00:00.json 2022-11-18 14:20:39 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:39 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:39 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:39 [debug] Parsed 6 records 2022-11-18 14:20:39 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:39 [info] Found 6 records 2022-11-18 14:20:41 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:41 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-05T00:00:00_2022-07-06T00:00:00.json 2022-11-18 14:20:41 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-05T00:00:00_2022-07-06T00:00:00.json 2022-11-18 14:20:41 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:41 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:41 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:41 [debug] Parsed 2 records 2022-11-18 14:20:41 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:41 [info] Found 2 records 2022-11-18 14:20:42 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:20:42 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-24T00:00:00_2022-08-25T00:00:00.json 2022-11-18 14:20:43 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-24T00:00:00_2022-08-25T00:00:00.json 2022-11-18 14:20:43 [debug] Registering filter: xdmod.external_process 2022-11-18 14:20:43 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:20:43 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:20:43 [debug] Parsed 87 records 2022-11-18 14:20:43 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:20:43 [info] Found 87 records 2022-11-18 14:21:08 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:21:08 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-05-27T00:00:00_2022-05-28T00:00:00.json 2022-11-18 14:21:08 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-05-27T00:00:00_2022-05-28T00:00:00.json 2022-11-18 14:21:08 [debug] Registering filter: xdmod.external_process 2022-11-18 14:21:08 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:21:08 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:21:08 [debug] Parsed 2 records 2022-11-18 14:21:08 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:21:08 [info] Found 2 records 2022-11-18 14:21:09 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:21:09 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-08T00:00:00_2022-11-09T00:00:00.json 2022-11-18 14:21:11 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-08T00:00:00_2022-11-09T00:00:00.json 2022-11-18 14:21:11 [debug] Registering filter: xdmod.external_process 2022-11-18 14:21:11 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:21:11 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:21:11 [debug] Parsed 30 records 2022-11-18 14:21:11 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:21:11 [info] Found 30 records 2022-11-18 14:21:49 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:21:49 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-10T00:00:00_2022-02-11T00:00:00.json 2022-11-18 14:21:51 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-10T00:00:00_2022-02-11T00:00:00.json 2022-11-18 14:21:51 [debug] Registering filter: xdmod.external_process 2022-11-18 14:21:51 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:21:51 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:21:51 [debug] Parsed 6 records 2022-11-18 14:21:51 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:21:51 [info] Found 6 records 2022-11-18 14:22:00 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:00 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-24T00:00:00_2022-02-25T00:00:00.json 2022-11-18 14:22:01 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-24T00:00:00_2022-02-25T00:00:00.json 2022-11-18 14:22:01 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:01 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:01 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:01 [debug] Parsed 10 records 2022-11-18 14:22:01 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:01 [info] Found 10 records 2022-11-18 14:22:20 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:20 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-17T00:00:00_2022-07-18T00:00:00.json 2022-11-18 14:22:21 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-17T00:00:00_2022-07-18T00:00:00.json 2022-11-18 14:22:21 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:21 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:21 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:21 [debug] Parsed 2 records 2022-11-18 14:22:21 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:21 [info] Found 2 records 2022-11-18 14:22:25 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:25 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-30T00:00:00_2022-03-31T00:00:00.json 2022-11-18 14:22:26 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-30T00:00:00_2022-03-31T00:00:00.json 2022-11-18 14:22:26 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:26 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:26 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:26 [debug] Parsed 4 records 2022-11-18 14:22:26 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:26 [info] Found 4 records 2022-11-18 14:22:34 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:34 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-23T00:00:00_2022-02-24T00:00:00.json 2022-11-18 14:22:35 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-23T00:00:00_2022-02-24T00:00:00.json 2022-11-18 14:22:35 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:35 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:35 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:35 [debug] Parsed 13 records 2022-11-18 14:22:35 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:35 [info] Found 13 records 2022-11-18 14:22:56 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:22:56 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-17T00:00:00_2022-03-18T00:00:00.json 2022-11-18 14:22:58 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-17T00:00:00_2022-03-18T00:00:00.json 2022-11-18 14:22:58 [debug] Registering filter: xdmod.external_process 2022-11-18 14:22:58 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:22:58 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:22:58 [debug] Parsed 2 records 2022-11-18 14:22:58 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:22:58 [info] Found 2 records 2022-11-18 14:23:00 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:00 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-25T00:00:00_2022-03-26T00:00:00.json 2022-11-18 14:23:00 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-25T00:00:00_2022-03-26T00:00:00.json 2022-11-18 14:23:00 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:00 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:00 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:00 [debug] Parsed 1 records 2022-11-18 14:23:00 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:00 [info] Found 1 records 2022-11-18 14:23:02 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:02 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-03-23T00:00:00_2022-03-24T00:00:00.json 2022-11-18 14:23:02 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-03-23T00:00:00_2022-03-24T00:00:00.json 2022-11-18 14:23:02 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:02 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:02 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:02 [debug] Parsed 14 records 2022-11-18 14:23:02 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:02 [info] Found 14 records 2022-11-18 14:23:19 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:19 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-02-22T00:00:00_2022-02-23T00:00:00.json 2022-11-18 14:23:19 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-02-22T00:00:00_2022-02-23T00:00:00.json 2022-11-18 14:23:19 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:19 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:19 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:19 [debug] Parsed 6 records 2022-11-18 14:23:19 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:19 [info] Found 6 records 2022-11-18 14:23:22 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:22 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-23T00:00:00_2022-07-24T00:00:00.json 2022-11-18 14:23:23 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-23T00:00:00_2022-07-24T00:00:00.json 2022-11-18 14:23:23 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:23 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:23 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:23 [debug] Parsed 16 records 2022-11-18 14:23:23 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:23 [info] Found 16 records 2022-11-18 14:23:29 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:29 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-11-04T00:00:00_2022-11-05T00:00:00.json 2022-11-18 14:23:30 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-11-04T00:00:00_2022-11-05T00:00:00.json 2022-11-18 14:23:30 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:30 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:30 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:30 [debug] Parsed 10 records 2022-11-18 14:23:30 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:30 [info] Found 10 records 2022-11-18 14:23:34 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:34 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-04T00:00:00_2022-10-05T00:00:00.json 2022-11-18 14:23:35 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-04T00:00:00_2022-10-05T00:00:00.json 2022-11-18 14:23:35 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:35 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:35 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:35 [debug] Parsed 14 records 2022-11-18 14:23:35 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:35 [info] Found 14 records 2022-11-18 14:23:43 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:43 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-09-08T00:00:00_2022-09-09T00:00:00.json 2022-11-18 14:23:44 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-09-08T00:00:00_2022-09-09T00:00:00.json 2022-11-18 14:23:44 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:44 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:44 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:44 [debug] Parsed 2 records 2022-11-18 14:23:44 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:44 [info] Found 2 records 2022-11-18 14:23:44 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:44 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-13T00:00:00_2022-08-14T00:00:00.json 2022-11-18 14:23:45 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-13T00:00:00_2022-08-14T00:00:00.json 2022-11-18 14:23:45 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:45 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:45 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:45 [debug] Parsed 2 records 2022-11-18 14:23:45 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:45 [info] Found 2 records 2022-11-18 14:23:47 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:47 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-01-26T00:00:00_2022-01-27T00:00:00.json 2022-11-18 14:23:47 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-01-26T00:00:00_2022-01-27T00:00:00.json 2022-11-18 14:23:47 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:47 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:47 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:47 [debug] Parsed 4 records 2022-11-18 14:23:47 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:47 [info] Found 4 records 2022-11-18 14:23:48 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:48 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-08-11T00:00:00_2022-08-12T00:00:00.json 2022-11-18 14:23:49 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-08-11T00:00:00_2022-08-12T00:00:00.json 2022-11-18 14:23:49 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:49 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:49 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:49 [debug] Parsed 6 records 2022-11-18 14:23:49 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:49 [info] Found 6 records 2022-11-18 14:23:50 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:50 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-01T00:00:00_2022-07-02T00:00:00.json 2022-11-18 14:23:50 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-01T00:00:00_2022-07-02T00:00:00.json 2022-11-18 14:23:50 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:50 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:50 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:50 [debug] Parsed 2 records 2022-11-18 14:23:50 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:50 [info] Found 2 records 2022-11-18 14:23:51 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:23:51 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-07-13T00:00:00_2022-07-14T00:00:00.json 2022-11-18 14:23:53 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-07-13T00:00:00_2022-07-14T00:00:00.json 2022-11-18 14:23:53 [debug] Registering filter: xdmod.external_process 2022-11-18 14:23:53 [debug] Creating filter xdmod.external_process: jq -c '.[] | .["record_type"] += if .["event_type"] == "compute.instance.exists" then "ADMINISTRATIVE" else "ACCOUNTING" end | .["service_provider"] += "${SERVICE_PROVIDER}"' 2022-11-18 14:23:53 [debug] Adding filter xdmod.external_process to stream: jq 2022-11-18 14:23:53 [debug] Parsed 76 records 2022-11-18 14:23:53 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/cloud_openstack/event.schema.json 2022-11-18 14:23:53 [info] Found 76 records 2022-11-18 14:24:34 [debug] Current file iterator no longer valid, checking next iterator. 2022-11-18 14:24:34 [info] Scanning file: /root/xdmod_data/nerc_openstack/2022-10-14T00:00:00_2022-10-15T00:00:00.json 2022-11-18 14:24:35 [debug] Parsing /root/xdmod_data/nerc_openstack/2022-10-14T00:00:00_2022-10-15T00:00:00.json 2022-11-18 14:24:35 [debug] Registering filter: xdmod.external_process

rob-baron commented 10 months ago

2022-11-18 14:40:49 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782449.1124, end_time: 1668782449.3668, elapsed_time: 0.25442, records_examined: 65, records_loaded: 65) 2022-11-18 14:40:50 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782449.5527, end_time: 1668782450.6202, elapsed_time: 1.06752, records_examined: 4, records_loaded: 4) 2022-11-18 14:40:51 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782450.872, end_time: 1668782450.987, elapsed_time: 0.11499, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:51 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782451.1728, end_time: 1668782451.3106, elapsed_time: 0.13779, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:51 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782451.4301, end_time: 1668782451.5324, elapsed_time: 0.10227, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:51 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782451.663, end_time: 1668782451.844, elapsed_time: 0.18104, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:52 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782451.9628, end_time: 1668782452.1627, elapsed_time: 0.19993, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:52 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782452.2759, end_time: 1668782452.3984, elapsed_time: 0.12254, records_examined: 8, records_loaded: 8) 2022-11-18 14:40:52 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782452.7365, end_time: 1668782452.8947, elapsed_time: 0.15821, records_examined: 7, records_loaded: 7) 2022-11-18 14:40:53 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:40:53 [notice] Start processing section 'xdmod.jobs-cloud-import-users-generic' 2022-11-18 14:40:53 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudStagingUserIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782453.2051, end_time: 1668782453.3236, elapsed_time: 0.1184, records_examined: 0, records_loaded: 0) 2022-11-18 14:40:53 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudStagingUserResourceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782453.4501, end_time: 1668782453.6152, elapsed_time: 0.16515, records_examined: 0, records_loaded: 0) 2022-11-18 14:40:54 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingHpcdbPeopleIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782453.7356, end_time: 1668782454.3419, elapsed_time: 0.6063, records_examined: 0, records_loaded: 0) 2022-11-18 14:40:57 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudloudStagingHpcdbSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782455.2347, end_time: 1668782457.0196, elapsed_time: 1.78485, records_examined: 100, records_loaded: 100) 2022-11-18 14:40:58 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782458.2435, end_time: 1668782458.9109, elapsed_time: 0.66746, records_examined: 100, records_loaded: 100) 2022-11-18 14:41:04 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingPersonIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782459.7932, end_time: 1668782464.3235, elapsed_time: 4.53034, records_examined: 50, records_loaded: 50) 2022-11-18 14:41:04 [notice] Finished processing section 'xdmod.jobs-cloud-import-users-generic' 2022-11-18 14:41:04 [notice] Start processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:41:06 [notice] (action: xdmod.hpcdb-ingest-common.resource-types (ETL\Ingestor\DatabaseIngestor), start_time: 1668782465.2803, end_time: 1668782466.71, elapsed_time: 1.42976, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:08 [notice] (action: xdmod.hpcdb-ingest-common.organizations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782467.4046, end_time: 1668782468.1126, elapsed_time: 0.708, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:10 [notice] (action: xdmod.hpcdb-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782469.3205, end_time: 1668782470.1014, elapsed_time: 0.78088, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:11 [notice] (action: xdmod.hpcdb-ingest-common.resource-specs (ETL\Ingestor\DatabaseIngestor), start_time: 1668782470.6531, end_time: 1668782471.3775, elapsed_time: 0.72441, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:13 [notice] (action: xdmod.hpcdb-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782472.0661, end_time: 1668782473.213, elapsed_time: 1.14696, records_examined: 4, records_loaded: 4) 2022-11-18 14:41:15 [notice] (action: xdmod.hpcdb-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782474.4145, end_time: 1668782474.8578, elapsed_time: 0.44332, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:16 [notice] (action: xdmod.hpcdb-ingest-common.accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782476.3728, end_time: 1668782476.6071, elapsed_time: 0.23432, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:18 [notice] (action: xdmod.hpcdb-ingest-common.allocations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782477.4697, end_time: 1668782478.0964, elapsed_time: 0.62668, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:20 [notice] (action: xdmod.hpcdb-ingest-common.people (ETL\Ingestor\DatabaseIngestor), start_time: 1668782478.7544, end_time: 1668782479.852, elapsed_time: 1.09758, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:22 [notice] (action: xdmod.hpcdb-ingest-common.requests (ETL\Ingestor\DatabaseIngestor), start_time: 1668782481.749, end_time: 1668782482.6878, elapsed_time: 0.93879, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:24 [notice] (action: xdmod.hpcdb-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782483.6428, end_time: 1668782484.2517, elapsed_time: 0.60881, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:27 [notice] (action: xdmod.hpcdb-ingest-common.allocations-on-resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782485.4036, end_time: 1668782487.5777, elapsed_time: 2.17416, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:29 [notice] (action: xdmod.hpcdb-ingest-common.email-addresses (ETL\Ingestor\DatabaseIngestor), start_time: 1668782487.7596, end_time: 1668782488.9533, elapsed_time: 1.19364, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:30 [notice] (action: xdmod.hpcdb-ingest-common.principal-investigators (ETL\Ingestor\DatabaseIngestor), start_time: 1668782489.7231, end_time: 1668782490.6485, elapsed_time: 0.92543, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:35 [notice] (action: xdmod.hpcdb-ingest-common.system-accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782492.6003, end_time: 1668782495.019, elapsed_time: 2.41876, records_examined: 100, records_loaded: 100) 2022-11-18 14:41:36 [notice] Finished processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:41:36 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:41:38 [notice] (action: xdmod.hpcdb-xdw-ingest-common.account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782496.6293, end_time: 1668782498.2751, elapsed_time: 1.64585, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:41 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782500.8061, end_time: 1668782501.8378, elapsed_time: 1.03172, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:44 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation (ETL\Ingestor\DatabaseIngestor), start_time: 1668782502.8287, end_time: 1668782503.0462, elapsed_time: 0.21753, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:51 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782507.3606, end_time: 1668782510.0521, elapsed_time: 2.69147, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:54 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-on-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782513.1627, end_time: 1668782514.2513, elapsed_time: 1.08864, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:55 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science (ETL\Ingestor\DatabaseIngestor), start_time: 1668782514.7511, end_time: 1668782515.2682, elapsed_time: 0.51712, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:03 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science-hierarchy (ETL\Ingestor\DatabaseIngestor), start_time: 1668782516.427, end_time: 1668782523.3488, elapsed_time: 6.92174, records_examined: 1, records_loaded: 1) 2022-11-18 14:42:10 [notice] (action: xdmod.hpcdb-xdw-ingest-common.organization (ETL\Ingestor\DatabaseIngestor), start_time: 1668782524.2033, end_time: 1668782530.0566, elapsed_time: 5.85333, records_examined: 1, records_loaded: 1) 2022-11-18 14:42:14 [notice] (action: xdmod.hpcdb-xdw-ingest-common.pi-person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782531.2807, end_time: 1668782533.475, elapsed_time: 2.19429, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:19 [notice] (action: xdmod.hpcdb-xdw-ingest-common.person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782536.3265, end_time: 1668782539.4257, elapsed_time: 3.09918, records_examined: 50, records_loaded: 50) 2022-11-18 14:42:21 [notice] (action: xdmod.hpcdb-xdw-ingest-common.people-under-pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782540.6028, end_time: 1668782540.9104, elapsed_time: 0.30758, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:25 [notice] (action: xdmod.hpcdb-xdw-ingest-common.principal-investigator (ETL\Ingestor\DatabaseIngestor), start_time: 1668782543.7748, end_time: 1668782545.22, elapsed_time: 1.44525, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:31 [notice] (action: xdmod.hpcdb-xdw-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782547.5542, end_time: 1668782551.1251, elapsed_time: 3.57094, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:34 [notice] (action: xdmod.hpcdb-xdw-ingest-common.request (ETL\Ingestor\DatabaseIngestor), start_time: 1668782551.8925, end_time: 1668782554.4559, elapsed_time: 2.56346, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:38 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-type (ETL\Ingestor\DatabaseIngestor), start_time: 1668782555.7907, end_time: 1668782558.2351, elapsed_time: 2.44436, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:41 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-spec (ETL\Ingestor\DatabaseIngestor), start_time: 1668782559.7698, end_time: 1668782561.2622, elapsed_time: 1.49238, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:44 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782563.1296, end_time: 1668782564.1205, elapsed_time: 0.9909, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:46 [notice] (action: xdmod.hpcdb-xdw-ingest-common.service-provider (ETL\Ingestor\DatabaseIngestor), start_time: 1668782564.947, end_time: 1668782566.4064, elapsed_time: 1.45937, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:49 [notice] (action: xdmod.hpcdb-xdw-ingest-common.system-account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782567.6409, end_time: 1668782569.4468, elapsed_time: 1.80593, records_examined: 100, records_loaded: 100) 2022-11-18 14:42:49 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:42:50 [notice] Start processing section 'xdmod.jobs-cloud-extract-generic' 2022-11-18 14:42:52 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782572.2487, end_time: 1668782572.8575, elapsed_time: 0.60877, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:53 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudHostIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782573.3821, end_time: 1668782573.485, elapsed_time: 0.10297, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:55 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudImageIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782574.9145, end_time: 1668782575.0533, elapsed_time: 0.1388, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:56 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceTypeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782575.4014, end_time: 1668782575.5932, elapsed_time: 0.19184, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:57 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782576.6259, end_time: 1668782576.8593, elapsed_time: 0.2333, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:58 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudVolumeAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782577.433, end_time: 1668782578.0495, elapsed_time: 0.61649, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:58 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_generic/post_ingest_updates.sql' using delimiter '//' containing 4 statements 2022-11-18 14:43:00 [notice] Finished Processing 4 SQL statements 2022-11-18 14:43:00 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudPostIngestUpdates (ETL\Maintenance\ExecuteSql), start_time: 1668782578.6532, end_time: 1668782580.5485, elapsed_time: 1.89526) 2022-11-18 14:43:03 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudStagingEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782581.0771, end_time: 1668782583.1575, elapsed_time: 2.08042, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:04 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudRootVolumeAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782583.6902, end_time: 1668782583.8285, elapsed_time: 0.13829, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:04 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudIpAddressAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782584.3776, end_time: 1668782584.9215, elapsed_time: 0.54398, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:05 [notice] Altering table modw_cloud.event 2022-11-18 14:43:06 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782585.3975, end_time: 1668782586.5012, elapsed_time: 1.10374, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:10 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceDataIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782587.5314, end_time: 1668782590.0665, elapsed_time: 2.53516, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:12 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventAssetRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782590.2058, end_time: 1668782590.7573, elapsed_time: 0.55142, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:13 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782592.812, end_time: 1668782592.9826, elapsed_time: 0.17058, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:13 [notice] Finished processing section 'xdmod.jobs-cloud-extract-generic' 2022-11-18 14:43:14 [notice] Start processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 14:43:15 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782595.1651, end_time: 1668782595.617, elapsed_time: 0.45187, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:16 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782595.8594, end_time: 1668782596.121, elapsed_time: 0.26158, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:17 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782596.638, end_time: 1668782596.8652, elapsed_time: 0.22719, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:17 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi-resource--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782597.3687, end_time: 1668782597.9494, elapsed_time: 0.58064, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:19 [notice] (action: xdmod.jobs-cloud-ingest-pi.user-pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782598.5894, end_time: 1668782598.9271, elapsed_time: 0.33766, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:19 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_common/pi_to_account.sql' using delimiter '//' containing 1 statements 2022-11-18 14:43:21 [notice] Finished Processing 1 SQL statements 2022-11-18 14:43:21 [notice] (action: xdmod.jobs-cloud-ingest-pi.UpdateCloudProjectToPI (ETL\Maintenance\ExecuteSql), start_time: 1668782599.9832, end_time: 1668782601.8728, elapsed_time: 1.88961) 2022-11-18 14:43:22 [notice] Finished processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 14:43:23 [notice] Ingesting cloud resource specs 2022-11-18 14:43:24 [notice] Start processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 14:43:26 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 14:43:27 [notice] (action: xdmod.ingest-cloud-resource-specs.CloudResourceSpecsReconstructor (ETL\Ingestor\CloudResourceSpecsStateTransformIngestor), start_time: 1668782606.2335, end_time: 1668782607.0264, elapsed_time: 0.79289, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:27 [notice] Finished processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 14:43:27 [notice] Ingesting OpenStack event log data 2022-11-18 14:44:33 [notice] Start processing section 'xdmod.staging-ingest-common' 2022-11-18 14:44:37 [notice] (action: xdmod.staging-ingest-common.organization (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782673.9396, end_time: 1668782675.83, elapsed_time: 1.89038, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:42 [notice] (action: xdmod.staging-ingest-common.resource-types (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782678.8296, end_time: 1668782681.8446, elapsed_time: 3.01499, records_examined: 12, records_loaded: 12) 2022-11-18 14:44:43 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-11-18 14:44:44 [notice] Finished Processing 2 SQL statements 2022-11-18 14:44:44 [notice] (action: xdmod.staging-ingest-common.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1668782683.362, end_time: 1668782684.4577, elapsed_time: 1.09578) 2022-11-18 14:44:49 [notice] (action: xdmod.staging-ingest-common.resource-config (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782686.0769, end_time: 1668782689.377, elapsed_time: 3.30007, records_examined: 4, records_loaded: 4) 2022-11-18 14:44:50 [notice] (action: xdmod.staging-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782689.5291, end_time: 1668782690.0714, elapsed_time: 0.54228, records_examined: 0, records_loaded: 0) 2022-11-18 14:44:51 [notice] (action: xdmod.staging-ingest-common.resource-specs (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782690.3026, end_time: 1668782691.7531, elapsed_time: 1.45047, records_examined: 4, records_loaded: 4) 2022-11-18 14:44:51 [notice] Finished processing section 'xdmod.staging-ingest-common' 2022-11-18 14:44:51 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:44:52 [notice] Altering table modw_cloud.event 2022-11-18 14:44:52 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 14:44:52 [notice] Altering table modw_cloud.session_records 2022-11-18 14:44:52 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1668782692.2298, end_time: 1668782692.7671, elapsed_time: 0.53733) 2022-11-18 14:44:53 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782693.1563, end_time: 1668782693.3195, elapsed_time: 0.16317, records_examined: 5, records_loaded: 5) 2022-11-18 14:44:53 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782693.4744, end_time: 1668782693.679, elapsed_time: 0.20459, records_examined: 65, records_loaded: 65) 2022-11-18 14:44:54 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782693.7979, end_time: 1668782694.1205, elapsed_time: 0.32265, records_examined: 4, records_loaded: 4) 2022-11-18 14:44:54 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782694.3415, end_time: 1668782694.4854, elapsed_time: 0.14392, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:55 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782694.6187, end_time: 1668782694.9508, elapsed_time: 0.33214, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:55 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782695.1663, end_time: 1668782695.2688, elapsed_time: 0.10253, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:55 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782695.5323, end_time: 1668782695.9617, elapsed_time: 0.42932, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:56 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782696.2516, end_time: 1668782696.3805, elapsed_time: 0.12893, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:56 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782696.5123, end_time: 1668782696.8918, elapsed_time: 0.37943, records_examined: 8, records_loaded: 8) 2022-11-18 14:44:57 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782697.1418, end_time: 1668782697.3181, elapsed_time: 0.17621, records_examined: 7, records_loaded: 7) 2022-11-18 14:44:57 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:44:57 [notice] Start processing section 'xdmod.jobs-cloud-import-users-openstack' 2022-11-18 14:44:58 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackStagingUserIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782698.0389, end_time: 1668782698.3302, elapsed_time: 0.29125, records_examined: 0, records_loaded: 0) 2022-11-18 14:44:58 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingUserResourceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782698.4677, end_time: 1668782698.6172, elapsed_time: 0.14952, records_examined: 0, records_loaded: 0) 2022-11-18 14:44:59 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingHpcdbPeopleIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782699.0507, end_time: 1668782699.1934, elapsed_time: 0.14267, records_examined: 0, records_loaded: 0) 2022-11-18 14:44:59 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingHpcdbSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782699.3924, end_time: 1668782699.6192, elapsed_time: 0.22678, records_examined: 100, records_loaded: 100) 2022-11-18 14:45:00 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782699.7739, end_time: 1668782700.264, elapsed_time: 0.49006, records_examined: 100, records_loaded: 100) 2022-11-18 14:45:04 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingPersonIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782700.8139, end_time: 1668782704.6441, elapsed_time: 3.83022, records_examined: 50, records_loaded: 50) 2022-11-18 14:45:04 [notice] Finished processing section 'xdmod.jobs-cloud-import-users-openstack' 2022-11-18 14:45:05 [notice] Start processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:45:06 [notice] (action: xdmod.hpcdb-ingest-common.resource-types (ETL\Ingestor\DatabaseIngestor), start_time: 1668782705.2362, end_time: 1668782705.9958, elapsed_time: 0.75967, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:06 [notice] (action: xdmod.hpcdb-ingest-common.organizations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782706.1973, end_time: 1668782706.8457, elapsed_time: 0.64839, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:07 [notice] (action: xdmod.hpcdb-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782707.2484, end_time: 1668782707.6471, elapsed_time: 0.39872, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:10 [notice] (action: xdmod.hpcdb-ingest-common.resource-specs (ETL\Ingestor\DatabaseIngestor), start_time: 1668782707.9986, end_time: 1668782710.0964, elapsed_time: 2.09778, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:11 [notice] (action: xdmod.hpcdb-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782710.7515, end_time: 1668782711.329, elapsed_time: 0.57747, records_examined: 4, records_loaded: 4) 2022-11-18 14:45:12 [notice] (action: xdmod.hpcdb-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782711.8736, end_time: 1668782712.4016, elapsed_time: 0.52794, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:13 [notice] (action: xdmod.hpcdb-ingest-common.accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782712.8052, end_time: 1668782713.379, elapsed_time: 0.57381, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:14 [notice] (action: xdmod.hpcdb-ingest-common.allocations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782713.9358, end_time: 1668782714.6091, elapsed_time: 0.6733, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:17 [notice] (action: xdmod.hpcdb-ingest-common.people (ETL\Ingestor\DatabaseIngestor), start_time: 1668782716.0495, end_time: 1668782716.9634, elapsed_time: 0.91392, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:18 [notice] (action: xdmod.hpcdb-ingest-common.requests (ETL\Ingestor\DatabaseIngestor), start_time: 1668782717.1298, end_time: 1668782718.1101, elapsed_time: 0.98031, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:20 [notice] (action: xdmod.hpcdb-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782719.2262, end_time: 1668782720.4221, elapsed_time: 1.19594, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:22 [notice] (action: xdmod.hpcdb-ingest-common.allocations-on-resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782721.534, end_time: 1668782722.0449, elapsed_time: 0.51085, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:24 [notice] (action: xdmod.hpcdb-ingest-common.email-addresses (ETL\Ingestor\DatabaseIngestor), start_time: 1668782722.2805, end_time: 1668782723.5511, elapsed_time: 1.27054, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:25 [notice] (action: xdmod.hpcdb-ingest-common.principal-investigators (ETL\Ingestor\DatabaseIngestor), start_time: 1668782724.8294, end_time: 1668782725.5623, elapsed_time: 0.73289, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:26 [notice] (action: xdmod.hpcdb-ingest-common.system-accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782726.0158, end_time: 1668782726.3895, elapsed_time: 0.37369, records_examined: 100, records_loaded: 100) 2022-11-18 14:45:27 [notice] Finished processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:45:27 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:45:28 [notice] (action: xdmod.hpcdb-xdw-ingest-common.account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782727.5059, end_time: 1668782728.1756, elapsed_time: 0.66976, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:29 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782728.7051, end_time: 1668782729.1014, elapsed_time: 0.39634, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:30 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation (ETL\Ingestor\DatabaseIngestor), start_time: 1668782729.6778, end_time: 1668782730.8554, elapsed_time: 1.17767, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:31 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782730.9811, end_time: 1668782731.4124, elapsed_time: 0.43129, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:32 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-on-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782731.9401, end_time: 1668782732.5155, elapsed_time: 0.57539, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:33 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science (ETL\Ingestor\DatabaseIngestor), start_time: 1668782732.7102, end_time: 1668782733.4382, elapsed_time: 0.72802, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:36 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science-hierarchy (ETL\Ingestor\DatabaseIngestor), start_time: 1668782733.7564, end_time: 1668782736.414, elapsed_time: 2.65761, records_examined: 1, records_loaded: 1) 2022-11-18 14:45:42 [notice] (action: xdmod.hpcdb-xdw-ingest-common.organization (ETL\Ingestor\DatabaseIngestor), start_time: 1668782739.3389, end_time: 1668782742.2525, elapsed_time: 2.9136, records_examined: 1, records_loaded: 1) 2022-11-18 14:45:43 [notice] (action: xdmod.hpcdb-xdw-ingest-common.pi-person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782743.3588, end_time: 1668782743.655, elapsed_time: 0.29617, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:46 [notice] (action: xdmod.hpcdb-xdw-ingest-common.person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782743.8195, end_time: 1668782746.0366, elapsed_time: 2.21704, records_examined: 50, records_loaded: 50) 2022-11-18 14:45:50 [notice] (action: xdmod.hpcdb-xdw-ingest-common.people-under-pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782748.5951, end_time: 1668782749.5405, elapsed_time: 0.94539, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:51 [notice] (action: xdmod.hpcdb-xdw-ingest-common.principal-investigator (ETL\Ingestor\DatabaseIngestor), start_time: 1668782750.2334, end_time: 1668782751.2954, elapsed_time: 1.06197, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:52 [notice] (action: xdmod.hpcdb-xdw-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782751.974, end_time: 1668782752.1218, elapsed_time: 0.14788, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:53 [notice] (action: xdmod.hpcdb-xdw-ingest-common.request (ETL\Ingestor\DatabaseIngestor), start_time: 1668782752.2676, end_time: 1668782753.2369, elapsed_time: 0.96931, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:54 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-type (ETL\Ingestor\DatabaseIngestor), start_time: 1668782753.511, end_time: 1668782754.3598, elapsed_time: 0.84879, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:56 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-spec (ETL\Ingestor\DatabaseIngestor), start_time: 1668782755.3038, end_time: 1668782755.7395, elapsed_time: 0.43567, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:57 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782756.8636, end_time: 1668782757.1858, elapsed_time: 0.3222, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:58 [notice] (action: xdmod.hpcdb-xdw-ingest-common.service-provider (ETL\Ingestor\DatabaseIngestor), start_time: 1668782757.5362, end_time: 1668782758.3492, elapsed_time: 0.81301, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:59 [notice] (action: xdmod.hpcdb-xdw-ingest-common.system-account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782758.5511, end_time: 1668782759.6849, elapsed_time: 1.13377, records_examined: 100, records_loaded: 100) 2022-11-18 14:46:00 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:46:00 [notice] Start processing section 'xdmod.jobs-cloud-extract-openstack' 2022-11-18 14:46:01 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackEventMapInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782761.1398, end_time: 1668782761.6484, elapsed_time: 0.50853, records_examined: 60, records_loaded: 60) 2022-11-18 14:46:02 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782762.4829, end_time: 1668782762.7723, elapsed_time: 0.28944, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:03 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudHostIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782762.9417, end_time: 1668782763.4033, elapsed_time: 0.46166, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:04 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudImageIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782764.3788, end_time: 1668782764.5974, elapsed_time: 0.21862, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:06 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudInstanceTypeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782766.1332, end_time: 1668782766.6809, elapsed_time: 0.54771, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:11 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782768.7398, end_time: 1668782771.0541, elapsed_time: 2.31428, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:19 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackInstanceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782776.4548, end_time: 1668782778.6297, elapsed_time: 2.17494, records_examined: 2, records_loaded: 2) 2022-11-18 14:46:20 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_domain.sql' using delimiter '//' containing 1 statements 2022-11-18 14:46:20 [notice] Finished Processing 1 SQL statements 2022-11-18 14:46:22 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackUnknownDomainIngestor (ETL\Maintenance\ExecuteSql), start_time: 1668782780.7018, end_time: 1668782782.2001, elapsed_time: 1.49825) 2022-11-18 14:46:24 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782783.5377, end_time: 1668782784.2818, elapsed_time: 0.74406, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:30 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainToSubmissionVenueMappingStagingIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782786.8042, end_time: 1668782790.2549, elapsed_time: 3.45072, records_examined: 1, records_loaded: 1) 2022-11-18 14:46:34 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainToSubmissionVenueMappingIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782791.6701, end_time: 1668782793.9287, elapsed_time: 2.25852, records_examined: 1, records_loaded: 1) 2022-11-18 14:46:42 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudStagingEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782795.5189, end_time: 1668782802.0739, elapsed_time: 6.55503, records_examined: 1458, records_loaded: 1458) 2022-11-18 14:46:43 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782802.6819, end_time: 1668782803.5674, elapsed_time: 0.8855, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:44 [notice] Altering table modw_cloud.event 2022-11-18 14:46:46 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782804.35, end_time: 1668782805.9177, elapsed_time: 1.56765, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:48 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackInstanceDataIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782806.9959, end_time: 1668782808.7122, elapsed_time: 1.71634, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:50 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventAssetRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782809.4848, end_time: 1668782810.4975, elapsed_time: 1.0127, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:51 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782811.31, end_time: 1668782811.9006, elapsed_time: 0.59065, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:54 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/post_ingest_update.sql' using delimiter '//' containing 8 statements 2022-11-18 14:47:03 [notice] Finished Processing 8 SQL statements 2022-11-18 14:47:03 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudPostIngestUpdates (ETL\Maintenance\ExecuteSql), start_time: 1668782814.258, end_time: 1668782823.971, elapsed_time: 9.71297) 2022-11-18 14:47:04 [notice] Finished processing section 'xdmod.jobs-cloud-extract-openstack' 2022-11-18 14:47:04 [notice] Start processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 14:47:05 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782824.7077, end_time: 1668782825.4568, elapsed_time: 0.74908, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:07 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782826.7502, end_time: 1668782827.0927, elapsed_time: 0.34248, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:09 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782828.421, end_time: 1668782828.6278, elapsed_time: 0.20681, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:17 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi-resource--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782829.9549, end_time: 1668782834.2461, elapsed_time: 4.29118, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:31 [notice] (action: xdmod.jobs-cloud-ingest-pi.user-pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782840.0998, end_time: 1668782847.0002, elapsed_time: 6.90039, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:37 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_common/pi_to_account.sql' using delimiter '//' containing 1 statements 2022-11-18 14:47:41 [notice] Finished Processing 1 SQL statements 2022-11-18 14:47:41 [notice] (action: xdmod.jobs-cloud-ingest-pi.UpdateCloudProjectToPI (ETL\Maintenance\ExecuteSql), start_time: 1668782857.1494, end_time: 1668782861.9461, elapsed_time: 4.79666) 2022-11-18 14:47:49 [notice] Finished processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 14:47:51 [notice] Ingesting cloud resource specs 2022-11-18 14:48:17 [notice] Start processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 14:48:19 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 14:48:30 [notice] (action: xdmod.ingest-cloud-resource-specs.CloudResourceSpecsReconstructor (ETL\Ingestor\CloudResourceSpecsStateTransformIngestor), start_time: 1668782899.1484, end_time: 1668782908.5845, elapsed_time: 9.43609, records_examined: 0, records_loaded: 0) 2022-11-18 14:48:33 [notice] Finished processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 14:51:55 [notice] Start processing section 'xdmod.jobs-xdw-aggregate' 2022-11-18 14:51:56 [notice] (action: xdmod.jobs-xdw-aggregate.table-create (ETL\Maintenance\ManageTables), start_time: 1668783116.8095, end_time: 1668783116.8342, elapsed_time: 0.02472) 2022-11-18 14:52:08 [notice] Altering table modw_aggregates.jobfact_by_day 2022-11-18 14:52:10 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), unit: day, start_date: none, end_date: none) 2022-11-18 14:52:19 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), unit: day, periods: 0, start_date: none, end_date: none, start_time: 1668783130.5347, end_time: 1668783139.9689, elapsed_time: 9.43419) 2022-11-18 14:52:25 [notice] end (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), start_time: 1668783126.6243, end_time: 1668783145.8778, elapsed_time: 0.321m) 2022-11-18 14:52:32 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: month, start_date: none, end_date: none) 2022-11-18 14:52:40 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: month, periods: 0, start_date: none, end_date: none, start_time: 1668783152.665, end_time: 1668783160.4556, elapsed_time: 7.79054) 2022-11-18 14:52:43 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: quarter, start_date: none, end_date: none) 2022-11-18 14:52:53 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: quarter, periods: 0, start_date: none, end_date: none, start_time: 1668783163.4512, end_time: 1668783173.4146, elapsed_time: 9.9634) 2022-11-18 14:52:56 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: year, start_date: none, end_date: none) 2022-11-18 14:52:58 [notice] Duplicate column after substitution: ("${aggregation_unit}: ${:PERIOD_VALUE}") '${aggregation_unit}' -> 'year' 2022-11-18 14:53:03 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: year, periods: 0, start_date: none, end_date: none, start_time: 1668783176.5725, end_time: 1668783183.9137, elapsed_time: 7.34126) 2022-11-18 14:53:06 [notice] end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), start_time: 1668783151.9401, end_time: 1668783186.2613, elapsed_time: 0.572m) 2022-11-18 14:53:08 [notice] (action: xdmod.jobs-xdw-aggregate.aggregation-joblist (ETL\Ingestor\ExplodeTransformIngestor), start_time: 1668783186.5698, end_time: 1668783188.1944, elapsed_time: 1.62461, records_examined: 0, records_loaded: 0) 2022-11-18 14:53:08 [notice] Finished processing section 'xdmod.jobs-xdw-aggregate' 2022-11-18 14:53:09 [notice] start (action: Jobs.build-filter-list) 2022-11-18 14:53:09 [notice] start (action: Jobs.build-filter-list.provider) 2022-11-18 14:53:14 [notice] end (action: Jobs.build-filter-list.provider, start_time: 1668783189.8211, end_time: 1668783194.8386) 2022-11-18 14:53:14 [notice] start (action: Jobs.build-filter-list.pi) 2022-11-18 14:53:18 [notice] end (action: Jobs.build-filter-list.pi, start_time: 1668783194.9543, end_time: 1668783198.1492) 2022-11-18 14:53:18 [notice] start (action: Jobs.build-filter-list.person) 2022-11-18 14:53:19 [notice] end (action: Jobs.build-filter-list.person, start_time: 1668783198.223, end_time: 1668783199.1474) 2022-11-18 14:53:19 [notice] start (action: Jobs.build-filter-list.qos) 2022-11-18 14:53:27 [notice] end (action: Jobs.build-filter-list.qos, start_time: 1668783199.4481, end_time: 1668783207.5474) 2022-11-18 14:53:28 [notice] start (action: Jobs.build-filter-list.queue) 2022-11-18 14:53:34 [notice] end (action: Jobs.build-filter-list.queue, start_time: 1668783208.2488, end_time: 1668783214.4599) 2022-11-18 14:53:35 [notice] start (action: Jobs.build-filter-list.username) 2022-11-18 14:53:46 [notice] end (action: Jobs.build-filter-list.username, start_time: 1668783215.641, end_time: 1668783226.5012) 2022-11-18 14:53:46 [notice] start (action: Jobs.build-filter-list.resource_type) 2022-11-18 14:53:55 [notice] end (action: Jobs.build-filter-list.resource_type, start_time: 1668783226.5918, end_time: 1668783235.4679) 2022-11-18 14:53:55 [notice] start (action: Jobs.build-filter-list.resource) 2022-11-18 14:54:11 [notice] end (action: Jobs.build-filter-list.resource, start_time: 1668783235.6697, end_time: 1668783251.2337) 2022-11-18 14:54:12 [notice] start (action: Jobs.build-filter-list.nsfdirectorate) 2022-11-18 14:54:28 [notice] end (action: Jobs.build-filter-list.nsfdirectorate, start_time: 1668783252.3413, end_time: 1668783268.5996) 2022-11-18 14:54:28 [notice] start (action: Jobs.build-filter-list.parentscience) 2022-11-18 14:54:43 [notice] end (action: Jobs.build-filter-list.parentscience, start_time: 1668783268.9803, end_time: 1668783283.7803) 2022-11-18 14:54:44 [notice] start (action: Jobs.build-filter-list.jobwaittime) 2022-11-18 14:55:00 [notice] end (action: Jobs.build-filter-list.jobwaittime, start_time: 1668783284.3073, end_time: 1668783300.3243) 2022-11-18 14:55:00 [notice] start (action: Jobs.build-filter-list.gpucount) 2022-11-18 14:55:06 [notice] end (action: Jobs.build-filter-list.gpucount, start_time: 1668783300.6279, end_time: 1668783306.538) 2022-11-18 14:55:06 [notice] start (action: Jobs.build-filter-list.jobsize) 2022-11-18 14:55:12 [notice] end (action: Jobs.build-filter-list.jobsize, start_time: 1668783306.5649, end_time: 1668783312.5036) 2022-11-18 14:55:12 [notice] start (action: Jobs.build-filter-list.fieldofscience) 2022-11-18 14:55:22 [notice] end (action: Jobs.build-filter-list.fieldofscience, start_time: 1668783312.9124, end_time: 1668783322.989) 2022-11-18 14:55:23 [notice] start (action: Jobs.build-filter-list.jobwalltime) 2022-11-18 14:55:35 [notice] end (action: Jobs.build-filter-list.jobwalltime, start_time: 1668783323.8016, end_time: 1668783335.0804) 2022-11-18 14:55:35 [notice] start (action: Jobs.build-filter-list.nodecount) 2022-11-18 14:55:42 [notice] end (action: Jobs.build-filter-list.nodecount, start_time: 1668783335.3411, end_time: 1668783342.2567) 2022-11-18 14:55:43 [notice] end (action: Jobs.build-filter-list, start_time: 1668783189.5101, end_time: 1668783343.9514) 2022-11-18 14:55:44 [notice] Aggregating Cloud data 2022-11-18 14:56:08 [error] {"message":"(ETL\LockFile): Cannot obtain lock. Process '15866' already running and executing overlapping actions (xdmod.cloud-state-pipeline.cloud-state-action, xdmod.cloud-state-pipeline.delete-session-records, xdmod.cloud-state-pipeline.cloud-session-records, xdmod.cloud-state-pipeline.CloudEventAggregatorByDay, xdmod.cloud-state-pipeline.CloudEventAggregator, xdmod.cloud-state-pipeline.CloudAggregatorSessionlist)"} 2022-11-18 14:56:08 [critical] Aggregation failed: (ETL\LockFile): Cannot obtain lock. Process '15866' already running and executing overlapping actions (xdmod.cloud-state-pipeline.cloud-state-action, xdmod.cloud-state-pipeline.delete-session-records, xdmod.cloud-state-pipeline.cloud-session-records, xdmod.cloud-state-pipeline.CloudEventAggregatorByDay, xdmod.cloud-state-pipeline.CloudEventAggregator, xdmod.cloud-state-pipeline.CloudAggregatorSessionlist) (stacktrace: #0 /usr/share/xdmod/classes/ETL/LockFile.php(181): CCR\Loggable->logAndThrowException('Cannot obtain l...')

1 /usr/share/xdmod/classes/ETL/EtlOverseer.php(420): ETL\LockFile->lock(Array)

2 /usr/share/xdmod/classes/ETL/Utilities.php(281): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))

3 /usr/share/xdmod/classes/OpenXdmod/DataWarehouseInitializer.php(362): ETL\Utilities::runEtlPipeline(Array, Object(CCR\Logger), Array)

4 /usr/bin/xdmod-ingestor(310): OpenXdmod\DataWarehouseInitializer->aggregateCloudData('2018-01-01 12:3...')

5 /usr/bin/xdmod-ingestor(21): main()

6 {main})

Robert Bartlett Baron , said 9 months ago Here is the ingestion command and log:

sh-4.2$ xdmod-ingestor --last-modified-start-date "2018-01-01 12:30:00" 2022-11-18 14:37:26 [notice] xdmod-ingestor start (process_start_time: 2022-11-18 14:37:26) 2022-11-18 14:37:32 [notice] Start processing section 'xdmod.staging-ingest-common' 2022-11-18 14:37:33 [notice] (action: xdmod.staging-ingest-common.organization (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782253.0898, end_time: 1668782253.8878, elapsed_time: 0.79795, records_examined: 1, records_loaded: 1) 2022-11-18 14:37:36 [notice] (action: xdmod.staging-ingest-common.resource-types (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782254.7207, end_time: 1668782256.8016, elapsed_time: 2.08097, records_examined: 12, records_loaded: 12) 2022-11-18 14:37:37 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-11-18 14:37:37 [notice] Finished Processing 2 SQL statements 2022-11-18 14:37:37 [notice] (action: xdmod.staging-ingest-common.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1668782257.4346, end_time: 1668782257.751, elapsed_time: 0.3164) 2022-11-18 14:37:39 [notice] (action: xdmod.staging-ingest-common.resource-config (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782258.2356, end_time: 1668782259.6188, elapsed_time: 1.38321, records_examined: 4, records_loaded: 4) 2022-11-18 14:37:40 [notice] (action: xdmod.staging-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782260.5663, end_time: 1668782260.6984, elapsed_time: 0.13208, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:42 [notice] (action: xdmod.staging-ingest-common.resource-specs (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782261.2415, end_time: 1668782262.6654, elapsed_time: 1.42389, records_examined: 4, records_loaded: 4) 2022-11-18 14:37:42 [notice] Finished processing section 'xdmod.staging-ingest-common' 2022-11-18 14:37:42 [notice] Start processing section 'xdmod.staging-ingest-jobs' 2022-11-18 14:37:43 [notice] (action: xdmod.staging-ingest-jobs.pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782262.8887, end_time: 1668782263.0787, elapsed_time: 0.18999, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:43 [notice] (action: xdmod.staging-ingest-jobs.pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782263.2071, end_time: 1668782263.7534, elapsed_time: 0.5463, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:44 [notice] (action: xdmod.staging-ingest-jobs.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782263.9407, end_time: 1668782264.0781, elapsed_time: 0.13747, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:47 [notice] (action: xdmod.staging-ingest-jobs.union-user-pi--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782265.2306, end_time: 1668782267.8562, elapsed_time: 2.62563, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:49 [notice] (action: xdmod.staging-ingest-jobs.union-user-pi--user (ETL\Ingestor\DatabaseIngestor), start_time: 1668782268.4381, end_time: 1668782269.5069, elapsed_time: 1.06889, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:51 [notice] (action: xdmod.staging-ingest-jobs.union-user-pi-resource--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782270.6111, end_time: 1668782271.0482, elapsed_time: 0.43713, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:52 [notice] (action: xdmod.staging-ingest-jobs.union-user-pi-resource--user (ETL\Ingestor\DatabaseIngestor), start_time: 1668782271.4732, end_time: 1668782272.5758, elapsed_time: 1.10258, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:53 [notice] (action: xdmod.staging-ingest-jobs.user-pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782272.9127, end_time: 1668782273.5679, elapsed_time: 0.65521, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:54 [notice] (action: xdmod.staging-ingest-jobs.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782274.1879, end_time: 1668782274.4982, elapsed_time: 0.31028, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:55 [notice] (action: xdmod.staging-ingest-jobs.job (ETL\Ingestor\DatabaseIngestor), start_time: 1668782275.2116, end_time: 1668782275.7292, elapsed_time: 0.51756, records_examined: 0, records_loaded: 0) 2022-11-18 14:37:55 [notice] Finished processing section 'xdmod.staging-ingest-jobs' 2022-11-18 14:38:08 [notice] Start processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:38:09 [notice] (action: xdmod.hpcdb-ingest-common.resource-types (ETL\Ingestor\DatabaseIngestor), start_time: 1668782289.1751, end_time: 1668782289.6658, elapsed_time: 0.49071, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:11 [notice] (action: xdmod.hpcdb-ingest-common.organizations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782290.2902, end_time: 1668782290.9747, elapsed_time: 0.68451, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:13 [notice] (action: xdmod.hpcdb-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782291.8006, end_time: 1668782292.5565, elapsed_time: 0.75587, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:16 [notice] (action: xdmod.hpcdb-ingest-common.resource-specs (ETL\Ingestor\DatabaseIngestor), start_time: 1668782295.1641, end_time: 1668782296.0684, elapsed_time: 0.90435, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:17 [notice] (action: xdmod.hpcdb-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782296.3747, end_time: 1668782297.1076, elapsed_time: 0.73288, records_examined: 4, records_loaded: 4) 2022-11-18 14:38:20 [notice] (action: xdmod.hpcdb-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782298.8535, end_time: 1668782299.6109, elapsed_time: 0.75733, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:21 [notice] (action: xdmod.hpcdb-ingest-common.accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782300.4844, end_time: 1668782301.0317, elapsed_time: 0.54733, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:22 [notice] (action: xdmod.hpcdb-ingest-common.allocations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782301.6411, end_time: 1668782302.2351, elapsed_time: 0.59405, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:24 [notice] (action: xdmod.hpcdb-ingest-common.people (ETL\Ingestor\DatabaseIngestor), start_time: 1668782304.0609, end_time: 1668782304.6507, elapsed_time: 0.58984, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:25 [notice] (action: xdmod.hpcdb-ingest-common.requests (ETL\Ingestor\DatabaseIngestor), start_time: 1668782305.4205, end_time: 1668782305.5571, elapsed_time: 0.1366, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:26 [notice] (action: xdmod.hpcdb-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782306.1478, end_time: 1668782306.8313, elapsed_time: 0.68345, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:29 [notice] (action: xdmod.hpcdb-ingest-common.allocations-on-resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782307.4337, end_time: 1668782309.0491, elapsed_time: 1.61536, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:31 [notice] (action: xdmod.hpcdb-ingest-common.email-addresses (ETL\Ingestor\DatabaseIngestor), start_time: 1668782309.686, end_time: 1668782310.923, elapsed_time: 1.23702, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:33 [notice] (action: xdmod.hpcdb-ingest-common.principal-investigators (ETL\Ingestor\DatabaseIngestor), start_time: 1668782312.1788, end_time: 1668782313.0736, elapsed_time: 0.89478, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:35 [notice] (action: xdmod.hpcdb-ingest-common.system-accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782313.857, end_time: 1668782315.1514, elapsed_time: 1.29437, records_examined: 100, records_loaded: 100) 2022-11-18 14:38:35 [notice] Finished processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:38:35 [notice] Start processing section 'xdmod.hpcdb-ingest-jobs' 2022-11-18 14:38:36 [notice] (action: xdmod.hpcdb-ingest-jobs.jobs (ETL\Ingestor\DatabaseIngestor), start_time: 1668782315.3995, end_time: 1668782316.0341, elapsed_time: 0.63455, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:37 [notice] Finished processing section 'xdmod.hpcdb-ingest-jobs' 2022-11-18 14:38:39 [notice] Start processing section 'xdmod.hpcdb-prep-xdw-job-ingest-by-new-jobs' 2022-11-18 14:38:46 [notice] (action: xdmod.hpcdb-prep-xdw-job-ingest-by-new-jobs.jobs (ETL\Ingestor\DatabaseIngestor), start_time: 1668782320.326, end_time: 1668782326.1468, elapsed_time: 5.82084, records_examined: 0, records_loaded: 0) 2022-11-18 14:38:46 [notice] Finished processing section 'xdmod.hpcdb-prep-xdw-job-ingest-by-new-jobs' 2022-11-18 14:39:04 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:39:05 [notice] (action: xdmod.hpcdb-xdw-ingest-common.account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782344.8463, end_time: 1668782345.0555, elapsed_time: 0.2092, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:09 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782347.2283, end_time: 1668782348.8155, elapsed_time: 1.58723, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:11 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation (ETL\Ingestor\DatabaseIngestor), start_time: 1668782350.3354, end_time: 1668782351.4383, elapsed_time: 1.10293, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:14 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782353.0139, end_time: 1668782353.9586, elapsed_time: 0.94473, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:18 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-on-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782354.1978, end_time: 1668782356.7613, elapsed_time: 2.5635, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:22 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science (ETL\Ingestor\DatabaseIngestor), start_time: 1668782360.4581, end_time: 1668782362.0642, elapsed_time: 1.60611, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:31 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science-hierarchy (ETL\Ingestor\DatabaseIngestor), start_time: 1668782363.7311, end_time: 1668782371.7921, elapsed_time: 8.06099, records_examined: 1, records_loaded: 1) 2022-11-18 14:39:42 [notice] (action: xdmod.hpcdb-xdw-ingest-common.organization (ETL\Ingestor\DatabaseIngestor), start_time: 1668782372.5013, end_time: 1668782382.8002, elapsed_time: 10.29893, records_examined: 1, records_loaded: 1) 2022-11-18 14:39:46 [notice] (action: xdmod.hpcdb-xdw-ingest-common.pi-person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782383.6165, end_time: 1668782385.8353, elapsed_time: 2.21883, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:52 [notice] (action: xdmod.hpcdb-xdw-ingest-common.person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782386.1076, end_time: 1668782392.4814, elapsed_time: 6.3738, records_examined: 50, records_loaded: 50) 2022-11-18 14:39:52 [notice] (action: xdmod.hpcdb-xdw-ingest-common.people-under-pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782392.7004, end_time: 1668782392.8812, elapsed_time: 0.18079, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:53 [notice] (action: xdmod.hpcdb-xdw-ingest-common.principal-investigator (ETL\Ingestor\DatabaseIngestor), start_time: 1668782393.0374, end_time: 1668782393.1634, elapsed_time: 0.12608, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:53 [notice] (action: xdmod.hpcdb-xdw-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782393.2881, end_time: 1668782393.4051, elapsed_time: 0.11699, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:53 [notice] (action: xdmod.hpcdb-xdw-ingest-common.request (ETL\Ingestor\DatabaseIngestor), start_time: 1668782393.5463, end_time: 1668782393.6904, elapsed_time: 0.14413, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:54 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-type (ETL\Ingestor\DatabaseIngestor), start_time: 1668782393.8937, end_time: 1668782394.1615, elapsed_time: 0.26771, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:54 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-spec (ETL\Ingestor\DatabaseIngestor), start_time: 1668782394.3101, end_time: 1668782394.4601, elapsed_time: 0.14999, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:54 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782394.5626, end_time: 1668782394.6981, elapsed_time: 0.13555, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:55 [notice] (action: xdmod.hpcdb-xdw-ingest-common.service-provider (ETL\Ingestor\DatabaseIngestor), start_time: 1668782394.821, end_time: 1668782394.9545, elapsed_time: 0.13346, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:55 [notice] (action: xdmod.hpcdb-xdw-ingest-common.system-account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782395.1444, end_time: 1668782395.419, elapsed_time: 0.27453, records_examined: 100, records_loaded: 100) 2022-11-18 14:39:55 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:39:55 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-jobs' 2022-11-18 14:39:55 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.node-count (ETL\Ingestor\DatabaseIngestor), start_time: 1668782395.6046, end_time: 1668782395.7537, elapsed_time: 0.14908, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:56 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.queue (ETL\Ingestor\DatabaseIngestor), start_time: 1668782395.9228, end_time: 1668782396.1844, elapsed_time: 0.26164, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:56 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.hosts (ETL\Ingestor\HpcdbHostsIngestor), start_time: 1668782396.3429, end_time: 1668782396.5851, elapsed_time: 0.24219, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:56 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.job-hosts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782396.7356, end_time: 1668782396.8647, elapsed_time: 0.12909, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:56 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/jobs/jobhosts-post-ingest-truncate.sql' using delimiter '//' containing 1 statements 2022-11-18 14:39:57 [notice] Finished Processing 1 SQL statements

rob-baron commented 10 months ago

2022-11-18 14:39:57 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.jobhosts-post-truncate (ETL\Maintenance\ExecuteSql), start_time: 1668782396.992, end_time: 1668782397.5382, elapsed_time: 0.54619) 2022-11-18 14:39:57 [notice] Altering table modw.job_records 2022-11-18 14:39:57 [notice] Altering table modw.job_tasks 2022-11-18 14:39:58 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.job-records-job-tasks (ETL\Ingestor\DatabaseIngestor), start_time: 1668782397.666, end_time: 1668782398.1073, elapsed_time: 0.44133, records_examined: 0, records_loaded: 0) 2022-11-18 14:39:58 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/jobs/job-record-hpc-post-ingest.sql' using delimiter '//' containing 2 statements 2022-11-18 14:39:58 [notice] Finished Processing 2 SQL statements 2022-11-18 14:39:58 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.HpcdbPostIngestJobUpdates (ETL\Maintenance\ExecuteSql), start_time: 1668782398.259, end_time: 1668782398.4705, elapsed_time: 0.21149) 2022-11-18 14:39:58 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-jobs' 2022-11-18 14:39:58 [notice] Ingesting generic cloud log files 2022-11-18 14:40:42 [notice] Start processing section 'xdmod.staging-ingest-common' 2022-11-18 14:40:43 [notice] (action: xdmod.staging-ingest-common.organization (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782442.5576, end_time: 1668782443.4387, elapsed_time: 0.88109, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:45 [notice] (action: xdmod.staging-ingest-common.resource-types (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782443.6108, end_time: 1668782445.1275, elapsed_time: 1.51672, records_examined: 12, records_loaded: 12) 2022-11-18 14:40:45 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-11-18 14:40:45 [notice] Finished Processing 2 SQL statements 2022-11-18 14:40:45 [notice] (action: xdmod.staging-ingest-common.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1668782445.297, end_time: 1668782445.5244, elapsed_time: 0.22739) 2022-11-18 14:40:46 [notice] (action: xdmod.staging-ingest-common.resource-config (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782445.6516, end_time: 1668782446.7171, elapsed_time: 1.06543, records_examined: 4, records_loaded: 4) 2022-11-18 14:40:47 [notice] (action: xdmod.staging-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782446.8999, end_time: 1668782447.0334, elapsed_time: 0.13351, records_examined: 0, records_loaded: 0) 2022-11-18 14:40:47 [notice] (action: xdmod.staging-ingest-common.resource-specs (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782447.1552, end_time: 1668782447.8429, elapsed_time: 0.68779, records_examined: 4, records_loaded: 4) 2022-11-18 14:40:47 [notice] Finished processing section 'xdmod.staging-ingest-common' 2022-11-18 14:40:48 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:40:48 [notice] Altering table modw_cloud.event 2022-11-18 14:40:48 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 14:40:48 [notice] Altering table modw_cloud.session_records 2022-11-18 14:40:48 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1668782448.0819, end_time: 1668782448.5274, elapsed_time: 0.44552) 2022-11-18 14:40:48 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782448.811, end_time: 1668782448.9707, elapsed_time: 0.15977, records_examined: 5, records_loaded: 5) 2022-11-18 14:40:49 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782449.1124, end_time: 1668782449.3668, elapsed_time: 0.25442, records_examined: 65, records_loaded: 65) 2022-11-18 14:40:50 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782449.5527, end_time: 1668782450.6202, elapsed_time: 1.06752, records_examined: 4, records_loaded: 4) 2022-11-18 14:40:51 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782450.872, end_time: 1668782450.987, elapsed_time: 0.11499, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:51 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782451.1728, end_time: 1668782451.3106, elapsed_time: 0.13779, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:51 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782451.4301, end_time: 1668782451.5324, elapsed_time: 0.10227, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:51 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782451.663, end_time: 1668782451.844, elapsed_time: 0.18104, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:52 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782451.9628, end_time: 1668782452.1627, elapsed_time: 0.19993, records_examined: 1, records_loaded: 1) 2022-11-18 14:40:52 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782452.2759, end_time: 1668782452.3984, elapsed_time: 0.12254, records_examined: 8, records_loaded: 8) 2022-11-18 14:40:52 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782452.7365, end_time: 1668782452.8947, elapsed_time: 0.15821, records_examined: 7, records_loaded: 7) 2022-11-18 14:40:53 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:40:53 [notice] Start processing section 'xdmod.jobs-cloud-import-users-generic' 2022-11-18 14:40:53 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudStagingUserIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782453.2051, end_time: 1668782453.3236, elapsed_time: 0.1184, records_examined: 0, records_loaded: 0) 2022-11-18 14:40:53 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudStagingUserResourceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782453.4501, end_time: 1668782453.6152, elapsed_time: 0.16515, records_examined: 0, records_loaded: 0) 2022-11-18 14:40:54 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingHpcdbPeopleIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782453.7356, end_time: 1668782454.3419, elapsed_time: 0.6063, records_examined: 0, records_loaded: 0) 2022-11-18 14:40:57 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudloudStagingHpcdbSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782455.2347, end_time: 1668782457.0196, elapsed_time: 1.78485, records_examined: 100, records_loaded: 100) 2022-11-18 14:40:58 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782458.2435, end_time: 1668782458.9109, elapsed_time: 0.66746, records_examined: 100, records_loaded: 100) 2022-11-18 14:41:04 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingPersonIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782459.7932, end_time: 1668782464.3235, elapsed_time: 4.53034, records_examined: 50, records_loaded: 50) 2022-11-18 14:41:04 [notice] Finished processing section 'xdmod.jobs-cloud-import-users-generic' 2022-11-18 14:41:04 [notice] Start processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:41:06 [notice] (action: xdmod.hpcdb-ingest-common.resource-types (ETL\Ingestor\DatabaseIngestor), start_time: 1668782465.2803, end_time: 1668782466.71, elapsed_time: 1.42976, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:08 [notice] (action: xdmod.hpcdb-ingest-common.organizations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782467.4046, end_time: 1668782468.1126, elapsed_time: 0.708, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:10 [notice] (action: xdmod.hpcdb-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782469.3205, end_time: 1668782470.1014, elapsed_time: 0.78088, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:11 [notice] (action: xdmod.hpcdb-ingest-common.resource-specs (ETL\Ingestor\DatabaseIngestor), start_time: 1668782470.6531, end_time: 1668782471.3775, elapsed_time: 0.72441, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:13 [notice] (action: xdmod.hpcdb-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782472.0661, end_time: 1668782473.213, elapsed_time: 1.14696, records_examined: 4, records_loaded: 4) 2022-11-18 14:41:15 [notice] (action: xdmod.hpcdb-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782474.4145, end_time: 1668782474.8578, elapsed_time: 0.44332, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:16 [notice] (action: xdmod.hpcdb-ingest-common.accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782476.3728, end_time: 1668782476.6071, elapsed_time: 0.23432, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:18 [notice] (action: xdmod.hpcdb-ingest-common.allocations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782477.4697, end_time: 1668782478.0964, elapsed_time: 0.62668, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:20 [notice] (action: xdmod.hpcdb-ingest-common.people (ETL\Ingestor\DatabaseIngestor), start_time: 1668782478.7544, end_time: 1668782479.852, elapsed_time: 1.09758, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:22 [notice] (action: xdmod.hpcdb-ingest-common.requests (ETL\Ingestor\DatabaseIngestor), start_time: 1668782481.749, end_time: 1668782482.6878, elapsed_time: 0.93879, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:24 [notice] (action: xdmod.hpcdb-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782483.6428, end_time: 1668782484.2517, elapsed_time: 0.60881, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:27 [notice] (action: xdmod.hpcdb-ingest-common.allocations-on-resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782485.4036, end_time: 1668782487.5777, elapsed_time: 2.17416, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:29 [notice] (action: xdmod.hpcdb-ingest-common.email-addresses (ETL\Ingestor\DatabaseIngestor), start_time: 1668782487.7596, end_time: 1668782488.9533, elapsed_time: 1.19364, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:30 [notice] (action: xdmod.hpcdb-ingest-common.principal-investigators (ETL\Ingestor\DatabaseIngestor), start_time: 1668782489.7231, end_time: 1668782490.6485, elapsed_time: 0.92543, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:35 [notice] (action: xdmod.hpcdb-ingest-common.system-accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782492.6003, end_time: 1668782495.019, elapsed_time: 2.41876, records_examined: 100, records_loaded: 100) 2022-11-18 14:41:36 [notice] Finished processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:41:36 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:41:38 [notice] (action: xdmod.hpcdb-xdw-ingest-common.account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782496.6293, end_time: 1668782498.2751, elapsed_time: 1.64585, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:41 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782500.8061, end_time: 1668782501.8378, elapsed_time: 1.03172, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:44 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation (ETL\Ingestor\DatabaseIngestor), start_time: 1668782502.8287, end_time: 1668782503.0462, elapsed_time: 0.21753, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:51 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782507.3606, end_time: 1668782510.0521, elapsed_time: 2.69147, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:54 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-on-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782513.1627, end_time: 1668782514.2513, elapsed_time: 1.08864, records_examined: 0, records_loaded: 0) 2022-11-18 14:41:55 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science (ETL\Ingestor\DatabaseIngestor), start_time: 1668782514.7511, end_time: 1668782515.2682, elapsed_time: 0.51712, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:03 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science-hierarchy (ETL\Ingestor\DatabaseIngestor), start_time: 1668782516.427, end_time: 1668782523.3488, elapsed_time: 6.92174, records_examined: 1, records_loaded: 1) 2022-11-18 14:42:10 [notice] (action: xdmod.hpcdb-xdw-ingest-common.organization (ETL\Ingestor\DatabaseIngestor), start_time: 1668782524.2033, end_time: 1668782530.0566, elapsed_time: 5.85333, records_examined: 1, records_loaded: 1) 2022-11-18 14:42:14 [notice] (action: xdmod.hpcdb-xdw-ingest-common.pi-person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782531.2807, end_time: 1668782533.475, elapsed_time: 2.19429, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:19 [notice] (action: xdmod.hpcdb-xdw-ingest-common.person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782536.3265, end_time: 1668782539.4257, elapsed_time: 3.09918, records_examined: 50, records_loaded: 50) 2022-11-18 14:42:21 [notice] (action: xdmod.hpcdb-xdw-ingest-common.people-under-pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782540.6028, end_time: 1668782540.9104, elapsed_time: 0.30758, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:25 [notice] (action: xdmod.hpcdb-xdw-ingest-common.principal-investigator (ETL\Ingestor\DatabaseIngestor), start_time: 1668782543.7748, end_time: 1668782545.22, elapsed_time: 1.44525, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:31 [notice] (action: xdmod.hpcdb-xdw-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782547.5542, end_time: 1668782551.1251, elapsed_time: 3.57094, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:34 [notice] (action: xdmod.hpcdb-xdw-ingest-common.request (ETL\Ingestor\DatabaseIngestor), start_time: 1668782551.8925, end_time: 1668782554.4559, elapsed_time: 2.56346, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:38 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-type (ETL\Ingestor\DatabaseIngestor), start_time: 1668782555.7907, end_time: 1668782558.2351, elapsed_time: 2.44436, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:41 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-spec (ETL\Ingestor\DatabaseIngestor), start_time: 1668782559.7698, end_time: 1668782561.2622, elapsed_time: 1.49238, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:44 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782563.1296, end_time: 1668782564.1205, elapsed_time: 0.9909, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:46 [notice] (action: xdmod.hpcdb-xdw-ingest-common.service-provider (ETL\Ingestor\DatabaseIngestor), start_time: 1668782564.947, end_time: 1668782566.4064, elapsed_time: 1.45937, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:49 [notice] (action: xdmod.hpcdb-xdw-ingest-common.system-account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782567.6409, end_time: 1668782569.4468, elapsed_time: 1.80593, records_examined: 100, records_loaded: 100) 2022-11-18 14:42:49 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:42:50 [notice] Start processing section 'xdmod.jobs-cloud-extract-generic' 2022-11-18 14:42:52 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782572.2487, end_time: 1668782572.8575, elapsed_time: 0.60877, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:53 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudHostIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782573.3821, end_time: 1668782573.485, elapsed_time: 0.10297, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:55 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudImageIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782574.9145, end_time: 1668782575.0533, elapsed_time: 0.1388, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:56 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceTypeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782575.4014, end_time: 1668782575.5932, elapsed_time: 0.19184, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:57 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782576.6259, end_time: 1668782576.8593, elapsed_time: 0.2333, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:58 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudVolumeAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782577.433, end_time: 1668782578.0495, elapsed_time: 0.61649, records_examined: 0, records_loaded: 0) 2022-11-18 14:42:58 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_generic/post_ingest_updates.sql' using delimiter '//' containing 4 statements 2022-11-18 14:43:00 [notice] Finished Processing 4 SQL statements 2022-11-18 14:43:00 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudPostIngestUpdates (ETL\Maintenance\ExecuteSql), start_time: 1668782578.6532, end_time: 1668782580.5485, elapsed_time: 1.89526) 2022-11-18 14:43:03 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudStagingEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782581.0771, end_time: 1668782583.1575, elapsed_time: 2.08042, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:04 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudRootVolumeAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782583.6902, end_time: 1668782583.8285, elapsed_time: 0.13829, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:04 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudIpAddressAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782584.3776, end_time: 1668782584.9215, elapsed_time: 0.54398, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:05 [notice] Altering table modw_cloud.event 2022-11-18 14:43:06 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782585.3975, end_time: 1668782586.5012, elapsed_time: 1.10374, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:10 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceDataIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782587.5314, end_time: 1668782590.0665, elapsed_time: 2.53516, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:12 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventAssetRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782590.2058, end_time: 1668782590.7573, elapsed_time: 0.55142, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:13 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782592.812, end_time: 1668782592.9826, elapsed_time: 0.17058, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:13 [notice] Finished processing section 'xdmod.jobs-cloud-extract-generic' 2022-11-18 14:43:14 [notice] Start processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 14:43:15 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782595.1651, end_time: 1668782595.617, elapsed_time: 0.45187, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:16 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782595.8594, end_time: 1668782596.121, elapsed_time: 0.26158, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:17 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782596.638, end_time: 1668782596.8652, elapsed_time: 0.22719, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:17 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi-resource--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782597.3687, end_time: 1668782597.9494, elapsed_time: 0.58064, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:19 [notice] (action: xdmod.jobs-cloud-ingest-pi.user-pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782598.5894, end_time: 1668782598.9271, elapsed_time: 0.33766, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:19 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_common/pi_to_account.sql' using delimiter '//' containing 1 statements 2022-11-18 14:43:21 [notice] Finished Processing 1 SQL statements 2022-11-18 14:43:21 [notice] (action: xdmod.jobs-cloud-ingest-pi.UpdateCloudProjectToPI (ETL\Maintenance\ExecuteSql), start_time: 1668782599.9832, end_time: 1668782601.8728, elapsed_time: 1.88961) 2022-11-18 14:43:22 [notice] Finished processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 14:43:23 [notice] Ingesting cloud resource specs 2022-11-18 14:43:24 [notice] Start processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 14:43:26 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 14:43:27 [notice] (action: xdmod.ingest-cloud-resource-specs.CloudResourceSpecsReconstructor (ETL\Ingestor\CloudResourceSpecsStateTransformIngestor), start_time: 1668782606.2335, end_time: 1668782607.0264, elapsed_time: 0.79289, records_examined: 0, records_loaded: 0) 2022-11-18 14:43:27 [notice] Finished processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 14:43:27 [notice] Ingesting OpenStack event log data 2022-11-18 14:44:33 [notice] Start processing section 'xdmod.staging-ingest-common' 2022-11-18 14:44:37 [notice] (action: xdmod.staging-ingest-common.organization (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782673.9396, end_time: 1668782675.83, elapsed_time: 1.89038, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:42 [notice] (action: xdmod.staging-ingest-common.resource-types (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782678.8296, end_time: 1668782681.8446, elapsed_time: 3.01499, records_examined: 12, records_loaded: 12) 2022-11-18 14:44:43 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-11-18 14:44:44 [notice] Finished Processing 2 SQL statements 2022-11-18 14:44:44 [notice] (action: xdmod.staging-ingest-common.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1668782683.362, end_time: 1668782684.4577, elapsed_time: 1.09578) 2022-11-18 14:44:49 [notice] (action: xdmod.staging-ingest-common.resource-config (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782686.0769, end_time: 1668782689.377, elapsed_time: 3.30007, records_examined: 4, records_loaded: 4) 2022-11-18 14:44:50 [notice] (action: xdmod.staging-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782689.5291, end_time: 1668782690.0714, elapsed_time: 0.54228, records_examined: 0, records_loaded: 0) 2022-11-18 14:44:51 [notice] (action: xdmod.staging-ingest-common.resource-specs (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782690.3026, end_time: 1668782691.7531, elapsed_time: 1.45047, records_examined: 4, records_loaded: 4) 2022-11-18 14:44:51 [notice] Finished processing section 'xdmod.staging-ingest-common' 2022-11-18 14:44:51 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:44:52 [notice] Altering table modw_cloud.event 2022-11-18 14:44:52 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 14:44:52 [notice] Altering table modw_cloud.session_records 2022-11-18 14:44:52 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1668782692.2298, end_time: 1668782692.7671, elapsed_time: 0.53733) 2022-11-18 14:44:53 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782693.1563, end_time: 1668782693.3195, elapsed_time: 0.16317, records_examined: 5, records_loaded: 5) 2022-11-18 14:44:53 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782693.4744, end_time: 1668782693.679, elapsed_time: 0.20459, records_examined: 65, records_loaded: 65) 2022-11-18 14:44:54 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782693.7979, end_time: 1668782694.1205, elapsed_time: 0.32265, records_examined: 4, records_loaded: 4) 2022-11-18 14:44:54 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782694.3415, end_time: 1668782694.4854, elapsed_time: 0.14392, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:55 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782694.6187, end_time: 1668782694.9508, elapsed_time: 0.33214, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:55 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782695.1663, end_time: 1668782695.2688, elapsed_time: 0.10253, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:55 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782695.5323, end_time: 1668782695.9617, elapsed_time: 0.42932, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:56 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782696.2516, end_time: 1668782696.3805, elapsed_time: 0.12893, records_examined: 1, records_loaded: 1) 2022-11-18 14:44:56 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782696.5123, end_time: 1668782696.8918, elapsed_time: 0.37943, records_examined: 8, records_loaded: 8) 2022-11-18 14:44:57 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782697.1418, end_time: 1668782697.3181, elapsed_time: 0.17621, records_examined: 7, records_loaded: 7) 2022-11-18 14:44:57 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-11-18 14:44:57 [notice] Start processing section 'xdmod.jobs-cloud-import-users-openstack' 2022-11-18 14:44:58 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackStagingUserIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782698.0389, end_time: 1668782698.3302, elapsed_time: 0.29125, records_examined: 0, records_loaded: 0) 2022-11-18 14:44:58 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingUserResourceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782698.4677, end_time: 1668782698.6172, elapsed_time: 0.14952, records_examined: 0, records_loaded: 0) 2022-11-18 14:44:59 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingHpcdbPeopleIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782699.0507, end_time: 1668782699.1934, elapsed_time: 0.14267, records_examined: 0, records_loaded: 0) 2022-11-18 14:44:59 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingHpcdbSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782699.3924, end_time: 1668782699.6192, elapsed_time: 0.22678, records_examined: 100, records_loaded: 100) 2022-11-18 14:45:00 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782699.7739, end_time: 1668782700.264, elapsed_time: 0.49006, records_examined: 100, records_loaded: 100) 2022-11-18 14:45:04 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingPersonIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782700.8139, end_time: 1668782704.6441, elapsed_time: 3.83022, records_examined: 50, records_loaded: 50) 2022-11-18 14:45:04 [notice] Finished processing section 'xdmod.jobs-cloud-import-users-openstack' 2022-11-18 14:45:05 [notice] Start processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:45:06 [notice] (action: xdmod.hpcdb-ingest-common.resource-types (ETL\Ingestor\DatabaseIngestor), start_time: 1668782705.2362, end_time: 1668782705.9958, elapsed_time: 0.75967, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:06 [notice] (action: xdmod.hpcdb-ingest-common.organizations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782706.1973, end_time: 1668782706.8457, elapsed_time: 0.64839, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:07 [notice] (action: xdmod.hpcdb-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782707.2484, end_time: 1668782707.6471, elapsed_time: 0.39872, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:10 [notice] (action: xdmod.hpcdb-ingest-common.resource-specs (ETL\Ingestor\DatabaseIngestor), start_time: 1668782707.9986, end_time: 1668782710.0964, elapsed_time: 2.09778, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:11 [notice] (action: xdmod.hpcdb-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782710.7515, end_time: 1668782711.329, elapsed_time: 0.57747, records_examined: 4, records_loaded: 4) 2022-11-18 14:45:12 [notice] (action: xdmod.hpcdb-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782711.8736, end_time: 1668782712.4016, elapsed_time: 0.52794, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:13 [notice] (action: xdmod.hpcdb-ingest-common.accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782712.8052, end_time: 1668782713.379, elapsed_time: 0.57381, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:14 [notice] (action: xdmod.hpcdb-ingest-common.allocations (ETL\Ingestor\DatabaseIngestor), start_time: 1668782713.9358, end_time: 1668782714.6091, elapsed_time: 0.6733, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:17 [notice] (action: xdmod.hpcdb-ingest-common.people (ETL\Ingestor\DatabaseIngestor), start_time: 1668782716.0495, end_time: 1668782716.9634, elapsed_time: 0.91392, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:18 [notice] (action: xdmod.hpcdb-ingest-common.requests (ETL\Ingestor\DatabaseIngestor), start_time: 1668782717.1298, end_time: 1668782718.1101, elapsed_time: 0.98031, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:20 [notice] (action: xdmod.hpcdb-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782719.2262, end_time: 1668782720.4221, elapsed_time: 1.19594, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:22 [notice] (action: xdmod.hpcdb-ingest-common.allocations-on-resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668782721.534, end_time: 1668782722.0449, elapsed_time: 0.51085, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:24 [notice] (action: xdmod.hpcdb-ingest-common.email-addresses (ETL\Ingestor\DatabaseIngestor), start_time: 1668782722.2805, end_time: 1668782723.5511, elapsed_time: 1.27054, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:25 [notice] (action: xdmod.hpcdb-ingest-common.principal-investigators (ETL\Ingestor\DatabaseIngestor), start_time: 1668782724.8294, end_time: 1668782725.5623, elapsed_time: 0.73289, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:26 [notice] (action: xdmod.hpcdb-ingest-common.system-accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668782726.0158, end_time: 1668782726.3895, elapsed_time: 0.37369, records_examined: 100, records_loaded: 100) 2022-11-18 14:45:27 [notice] Finished processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 14:45:27 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:45:28 [notice] (action: xdmod.hpcdb-xdw-ingest-common.account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782727.5059, end_time: 1668782728.1756, elapsed_time: 0.66976, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:29 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668782728.7051, end_time: 1668782729.1014, elapsed_time: 0.39634, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:30 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation (ETL\Ingestor\DatabaseIngestor), start_time: 1668782729.6778, end_time: 1668782730.8554, elapsed_time: 1.17767, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:31 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782730.9811, end_time: 1668782731.4124, elapsed_time: 0.43129, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:32 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-on-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782731.9401, end_time: 1668782732.5155, elapsed_time: 0.57539, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:33 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science (ETL\Ingestor\DatabaseIngestor), start_time: 1668782732.7102, end_time: 1668782733.4382, elapsed_time: 0.72802, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:36 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science-hierarchy (ETL\Ingestor\DatabaseIngestor), start_time: 1668782733.7564, end_time: 1668782736.414, elapsed_time: 2.65761, records_examined: 1, records_loaded: 1) 2022-11-18 14:45:42 [notice] (action: xdmod.hpcdb-xdw-ingest-common.organization (ETL\Ingestor\DatabaseIngestor), start_time: 1668782739.3389, end_time: 1668782742.2525, elapsed_time: 2.9136, records_examined: 1, records_loaded: 1) 2022-11-18 14:45:43 [notice] (action: xdmod.hpcdb-xdw-ingest-common.pi-person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782743.3588, end_time: 1668782743.655, elapsed_time: 0.29617, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:46 [notice] (action: xdmod.hpcdb-xdw-ingest-common.person (ETL\Ingestor\DatabaseIngestor), start_time: 1668782743.8195, end_time: 1668782746.0366, elapsed_time: 2.21704, records_examined: 50, records_loaded: 50) 2022-11-18 14:45:50 [notice] (action: xdmod.hpcdb-xdw-ingest-common.people-under-pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782748.5951, end_time: 1668782749.5405, elapsed_time: 0.94539, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:51 [notice] (action: xdmod.hpcdb-xdw-ingest-common.principal-investigator (ETL\Ingestor\DatabaseIngestor), start_time: 1668782750.2334, end_time: 1668782751.2954, elapsed_time: 1.06197, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:52 [notice] (action: xdmod.hpcdb-xdw-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668782751.974, end_time: 1668782752.1218, elapsed_time: 0.14788, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:53 [notice] (action: xdmod.hpcdb-xdw-ingest-common.request (ETL\Ingestor\DatabaseIngestor), start_time: 1668782752.2676, end_time: 1668782753.2369, elapsed_time: 0.96931, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:54 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-type (ETL\Ingestor\DatabaseIngestor), start_time: 1668782753.511, end_time: 1668782754.3598, elapsed_time: 0.84879, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:56 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-spec (ETL\Ingestor\DatabaseIngestor), start_time: 1668782755.3038, end_time: 1668782755.7395, elapsed_time: 0.43567, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:57 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668782756.8636, end_time: 1668782757.1858, elapsed_time: 0.3222, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:58 [notice] (action: xdmod.hpcdb-xdw-ingest-common.service-provider (ETL\Ingestor\DatabaseIngestor), start_time: 1668782757.5362, end_time: 1668782758.3492, elapsed_time: 0.81301, records_examined: 0, records_loaded: 0) 2022-11-18 14:45:59 [notice] (action: xdmod.hpcdb-xdw-ingest-common.system-account (ETL\Ingestor\DatabaseIngestor), start_time: 1668782758.5511, end_time: 1668782759.6849, elapsed_time: 1.13377, records_examined: 100, records_loaded: 100) 2022-11-18 14:46:00 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 14:46:00 [notice] Start processing section 'xdmod.jobs-cloud-extract-openstack' 2022-11-18 14:46:01 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackEventMapInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782761.1398, end_time: 1668782761.6484, elapsed_time: 0.50853, records_examined: 60, records_loaded: 60) 2022-11-18 14:46:02 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782762.4829, end_time: 1668782762.7723, elapsed_time: 0.28944, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:03 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudHostIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782762.9417, end_time: 1668782763.4033, elapsed_time: 0.46166, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:04 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudImageIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782764.3788, end_time: 1668782764.5974, elapsed_time: 0.21862, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:06 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudInstanceTypeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782766.1332, end_time: 1668782766.6809, elapsed_time: 0.54771, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:11 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782768.7398, end_time: 1668782771.0541, elapsed_time: 2.31428, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:19 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackInstanceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782776.4548, end_time: 1668782778.6297, elapsed_time: 2.17494, records_examined: 2, records_loaded: 2) 2022-11-18 14:46:20 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_domain.sql' using delimiter '//' containing 1 statements 2022-11-18 14:46:20 [notice] Finished Processing 1 SQL statements 2022-11-18 14:46:22 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackUnknownDomainIngestor (ETL\Maintenance\ExecuteSql), start_time: 1668782780.7018, end_time: 1668782782.2001, elapsed_time: 1.49825) 2022-11-18 14:46:24 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782783.5377, end_time: 1668782784.2818, elapsed_time: 0.74406, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:30 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainToSubmissionVenueMappingStagingIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1668782786.8042, end_time: 1668782790.2549, elapsed_time: 3.45072, records_examined: 1, records_loaded: 1) 2022-11-18 14:46:34 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainToSubmissionVenueMappingIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782791.6701, end_time: 1668782793.9287, elapsed_time: 2.25852, records_examined: 1, records_loaded: 1) 2022-11-18 14:46:42 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudStagingEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782795.5189, end_time: 1668782802.0739, elapsed_time: 6.55503, records_examined: 1458, records_loaded: 1458) 2022-11-18 14:46:43 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782802.6819, end_time: 1668782803.5674, elapsed_time: 0.8855, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:44 [notice] Altering table modw_cloud.event 2022-11-18 14:46:46 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782804.35, end_time: 1668782805.9177, elapsed_time: 1.56765, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:48 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackInstanceDataIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782806.9959, end_time: 1668782808.7122, elapsed_time: 1.71634, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:50 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventAssetRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782809.4848, end_time: 1668782810.4975, elapsed_time: 1.0127, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:51 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668782811.31, end_time: 1668782811.9006, elapsed_time: 0.59065, records_examined: 0, records_loaded: 0) 2022-11-18 14:46:54 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/post_ingest_update.sql' using delimiter '//' containing 8 statements 2022-11-18 14:47:03 [notice] Finished Processing 8 SQL statements 2022-11-18 14:47:03 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudPostIngestUpdates (ETL\Maintenance\ExecuteSql), start_time: 1668782814.258, end_time: 1668782823.971, elapsed_time: 9.71297) 2022-11-18 14:47:04 [notice] Finished processing section 'xdmod.jobs-cloud-extract-openstack' 2022-11-18 14:47:04 [notice] Start processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 14:47:05 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782824.7077, end_time: 1668782825.4568, elapsed_time: 0.74908, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:07 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782826.7502, end_time: 1668782827.0927, elapsed_time: 0.34248, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:09 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782828.421, end_time: 1668782828.6278, elapsed_time: 0.20681, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:17 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi-resource--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668782829.9549, end_time: 1668782834.2461, elapsed_time: 4.29118, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:31 [notice] (action: xdmod.jobs-cloud-ingest-pi.user-pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668782840.0998, end_time: 1668782847.0002, elapsed_time: 6.90039, records_examined: 0, records_loaded: 0) 2022-11-18 14:47:37 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_common/pi_to_account.sql' using delimiter '//' containing 1 statements 2022-11-18 14:47:41 [notice] Finished Processing 1 SQL statements 2022-11-18 14:47:41 [notice] (action: xdmod.jobs-cloud-ingest-pi.UpdateCloudProjectToPI (ETL\Maintenance\ExecuteSql), start_time: 1668782857.1494, end_time: 1668782861.9461, elapsed_time: 4.79666) 2022-11-18 14:47:49 [notice] Finished processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 14:47:51 [notice] Ingesting cloud resource specs 2022-11-18 14:48:17 [notice] Start processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 14:48:19 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 14:48:30 [notice] (action: xdmod.ingest-cloud-resource-specs.CloudResourceSpecsReconstructor (ETL\Ingestor\CloudResourceSpecsStateTransformIngestor), start_time: 1668782899.1484, end_time: 1668782908.5845, elapsed_time: 9.43609, records_examined: 0, records_loaded: 0) 2022-11-18 14:48:33 [notice] Finished processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 14:51:55 [notice] Start processing section 'xdmod.jobs-xdw-aggregate' 2022-11-18 14:51:56 [notice] (action: xdmod.jobs-xdw-aggregate.table-create (ETL\Maintenance\ManageTables), start_time: 1668783116.8095, end_time: 1668783116.8342, elapsed_time: 0.02472) 2022-11-18 14:52:08 [notice] Altering table modw_aggregates.jobfact_by_day 2022-11-18 14:52:10 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), unit: day, start_date: none, end_date: none) 2022-11-18 14:52:19 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), unit: day, periods: 0, start_date: none, end_date: none, start_time: 1668783130.5347, end_time: 1668783139.9689, elapsed_time: 9.43419) 2022-11-18 14:52:25 [notice] end (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), start_time: 1668783126.6243, end_time: 1668783145.8778, elapsed_time: 0.321m) 2022-11-18 14:52:32 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: month, start_date: none, end_date: none) 2022-11-18 14:52:40 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: month, periods: 0, start_date: none, end_date: none, start_time: 1668783152.665, end_time: 1668783160.4556, elapsed_time: 7.79054) 2022-11-18 14:52:43 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: quarter, start_date: none, end_date: none) 2022-11-18 14:52:53 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: quarter, periods: 0, start_date: none, end_date: none, start_time: 1668783163.4512, end_time: 1668783173.4146, elapsed_time: 9.9634) 2022-11-18 14:52:56 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: year, start_date: none, end_date: none) 2022-11-18 14:52:58 [notice] Duplicate column after substitution: ("${aggregation_unit}: ${:PERIOD_VALUE}") '${aggregation_unit}' -> 'year' 2022-11-18 14:53:03 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: year, periods: 0, start_date: none, end_date: none, start_time: 1668783176.5725, end_time: 1668783183.9137, elapsed_time: 7.34126) 2022-11-18 14:53:06 [notice] end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), start_time: 1668783151.9401, end_time: 1668783186.2613, elapsed_time: 0.572m) 2022-11-18 14:53:08 [notice] (action: xdmod.jobs-xdw-aggregate.aggregation-joblist (ETL\Ingestor\ExplodeTransformIngestor), start_time: 1668783186.5698, end_time: 1668783188.1944, elapsed_time: 1.62461, records_examined: 0, records_loaded: 0) 2022-11-18 14:53:08 [notice] Finished processing section 'xdmod.jobs-xdw-aggregate' 2022-11-18 14:53:09 [notice] start (action: Jobs.build-filter-list) 2022-11-18 14:53:09 [notice] start (action: Jobs.build-filter-list.provider) 2022-11-18 14:53:14 [notice] end (action: Jobs.build-filter-list.provider, start_time: 1668783189.8211, end_time: 1668783194.8386) 2022-11-18 14:53:14 [notice] start (action: Jobs.build-filter-list.pi) 2022-11-18 14:53:18 [notice] end (action: Jobs.build-filter-list.pi, start_time: 1668783194.9543, end_time: 1668783198.1492) 2022-11-18 14:53:18 [notice] start (action: Jobs.build-filter-list.person) 2022-11-18 14:53:19 [notice] end (action: Jobs.build-filter-list.person, start_time: 1668783198.223, end_time: 1668783199.1474) 2022-11-18 14:53:19 [notice] start (action: Jobs.build-filter-list.qos) 2022-11-18 14:53:27 [notice] end (action: Jobs.build-filter-list.qos, start_time: 1668783199.4481, end_time: 1668783207.5474) 2022-11-18 14:53:28 [notice] start (action: Jobs.build-filter-list.queue) 2022-11-18 14:53:34 [notice] end (action: Jobs.build-filter-list.queue, start_time: 1668783208.2488, end_time: 1668783214.4599) 2022-11-18 14:53:35 [notice] start (action: Jobs.build-filter-list.username) 2022-11-18 14:53:46 [notice] end (action: Jobs.build-filter-list.username, start_time: 1668783215.641, end_time: 1668783226.5012) 2022-11-18 14:53:46 [notice] start (action: Jobs.build-filter-list.resource_type) 2022-11-18 14:53:55 [notice] end (action: Jobs.build-filter-list.resource_type, start_time: 1668783226.5918, end_time: 1668783235.4679) 2022-11-18 14:53:55 [notice] start (action: Jobs.build-filter-list.resource) 2022-11-18 14:54:11 [notice] end (action: Jobs.build-filter-list.resource, start_time: 1668783235.6697, end_time: 1668783251.2337) 2022-11-18 14:54:12 [notice] start (action: Jobs.build-filter-list.nsfdirectorate) 2022-11-18 14:54:28 [notice] end (action: Jobs.build-filter-list.nsfdirectorate, start_time: 1668783252.3413, end_time: 1668783268.5996) 2022-11-18 14:54:28 [notice] start (action: Jobs.build-filter-list.parentscience) 2022-11-18 14:54:43 [notice] end (action: Jobs.build-filter-list.parentscience, start_time: 1668783268.9803, end_time: 1668783283.7803) 2022-11-18 14:54:44 [notice] start (action: Jobs.build-filter-list.jobwaittime) 2022-11-18 14:55:00 [notice] end (action: Jobs.build-filter-list.jobwaittime, start_time: 1668783284.3073, end_time: 1668783300.3243) 2022-11-18 14:55:00 [notice] start (action: Jobs.build-filter-list.gpucount) 2022-11-18 14:55:06 [notice] end (action: Jobs.build-filter-list.gpucount, start_time: 1668783300.6279, end_time: 1668783306.538) 2022-11-18 14:55:06 [notice] start (action: Jobs.build-filter-list.jobsize) 2022-11-18 14:55:12 [notice] end (action: Jobs.build-filter-list.jobsize, start_time: 1668783306.5649, end_time: 1668783312.5036) 2022-11-18 14:55:12 [notice] start (action: Jobs.build-filter-list.fieldofscience) 2022-11-18 14:55:22 [notice] end (action: Jobs.build-filter-list.fieldofscience, start_time: 1668783312.9124, end_time: 1668783322.989) 2022-11-18 14:55:23 [notice] start (action: Jobs.build-filter-list.jobwalltime) 2022-11-18 14:55:35 [notice] end (action: Jobs.build-filter-list.jobwalltime, start_time: 1668783323.8016, end_time: 1668783335.0804) 2022-11-18 14:55:35 [notice] start (action: Jobs.build-filter-list.nodecount) 2022-11-18 14:55:42 [notice] end (action: Jobs.build-filter-list.nodecount, start_time: 1668783335.3411, end_time: 1668783342.2567) 2022-11-18 14:55:43 [notice] end (action: Jobs.build-filter-list, start_time: 1668783189.5101, end_time: 1668783343.9514) 2022-11-18 14:55:44 [notice] Aggregating Cloud data 2022-11-18 14:56:08 [error] {"message":"(ETL\LockFile): Cannot obtain lock. Process '15866' already running and executing overlapping actions (xdmod.cloud-state-pipeline.cloud-state-action, xdmod.cloud-state-pipeline.delete-session-records, xdmod.cloud-state-pipeline.cloud-session-records, xdmod.cloud-state-pipeline.CloudEventAggregatorByDay, xdmod.cloud-state-pipeline.CloudEventAggregator, xdmod.cloud-state-pipeline.CloudAggregatorSessionlist)"} 2022-11-18 14:56:08 [critical] Aggregation failed: (ETL\LockFile): Cannot obtain lock. Process '15866' already running and executing overlapping actions (xdmod.cloud-state-pipeline.cloud-state-action, xdmod.cloud-state-pipeline.delete-session-records, xdmod.cloud-state-pipeline.cloud-session-records, xdmod.cloud-state-pipeline.CloudEventAggregatorByDay, xdmod.cloud-state-pipeline.CloudEventAggregator, xdmod.cloud-state-pipeline.CloudAggregatorSessionlist) (stacktrace: #0 /usr/share/xdmod/classes/ETL/LockFile.php(181): CCR\Loggable->logAndThrowException('Cannot obtain l...')

1 /usr/share/xdmod/classes/ETL/EtlOverseer.php(420): ETL\LockFile->lock(Array)

2 /usr/share/xdmod/classes/ETL/Utilities.php(281): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))

3 /usr/share/xdmod/classes/OpenXdmod/DataWarehouseInitializer.php(362): ETL\Utilities::runEtlPipeline(Array, Object(CCR\Logger), Array)

4 /usr/bin/xdmod-ingestor(310): OpenXdmod\DataWarehouseInitializer->aggregateCloudData('2018-01-01 12:3...')

5 /usr/bin/xdmod-ingestor(21): main()

6 {main})

Robert Bartlett Baron , said 9 months ago I found the other processes, killed them and reran the ingestor, the following is the output:

xdmod-ingestor --last-modified-start-date "2018-01-01 12:30:00" 2022-11-18 15:14:41 [notice] xdmod-ingestor start (process_start_time: 2022-11-18 15:14:41) 2022-11-18 15:15:39 [warning] Process '15866' is not running, releasing lock file. 2022-11-18 15:15:42 [notice] Start processing section 'xdmod.staging-ingest-common' 2022-11-18 15:15:48 [notice] (action: xdmod.staging-ingest-common.organization (ETL\Ingestor\StructuredFileIngestor), start_time: 1668784543.8461, end_time: 1668784548.1768, elapsed_time: 4.33069, records_examined: 1, records_loaded: 1) 2022-11-18 15:15:59 [notice] (action: xdmod.staging-ingest-common.resource-types (ETL\Ingestor\StructuredFileIngestor), start_time: 1668784549.2526, end_time: 1668784559.0376, elapsed_time: 9.78502, records_examined: 12, records_loaded: 12) 2022-11-18 15:16:01 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-11-18 15:16:06 [notice] Finished Processing 2 SQL statements 2022-11-18 15:16:07 [notice] (action: xdmod.staging-ingest-common.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1668784561.4327, end_time: 1668784567.2086, elapsed_time: 5.77592) 2022-11-18 15:16:17 [notice] (action: xdmod.staging-ingest-common.resource-config (ETL\Ingestor\StructuredFileIngestor), start_time: 1668784570.5771, end_time: 1668784576.4971, elapsed_time: 5.92, records_examined: 4, records_loaded: 4) 2022-11-18 15:16:20 [notice] (action: xdmod.staging-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668784579.311, end_time: 1668784580.2526, elapsed_time: 0.94157, records_examined: 0, records_loaded: 0) 2022-11-18 15:16:26 [notice] (action: xdmod.staging-ingest-common.resource-specs (ETL\Ingestor\StructuredFileIngestor), start_time: 1668784581.5486, end_time: 1668784585.4155, elapsed_time: 3.86694, records_examined: 4, records_loaded: 4) 2022-11-18 15:16:27 [notice] Finished processing section 'xdmod.staging-ingest-common' 2022-11-18 15:16:28 [notice] Start processing section 'xdmod.staging-ingest-jobs' 2022-11-18 15:16:31 [notice] (action: xdmod.staging-ingest-jobs.pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668784590.6035, end_time: 1668784591.762, elapsed_time: 1.15844, records_examined: 0, records_loaded: 0) 2022-11-18 15:16:35 [notice] (action: xdmod.staging-ingest-jobs.pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668784593.4404, end_time: 1668784594.8105, elapsed_time: 1.37003, records_examined: 0, records_loaded: 0) 2022-11-18 15:16:42 [notice] (action: xdmod.staging-ingest-jobs.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668784600.2998, end_time: 1668784601.1082, elapsed_time: 0.80836, records_examined: 0, records_loaded: 0) 2022-11-18 15:16:44 [notice] (action: xdmod.staging-ingest-jobs.union-user-pi--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668784604.3868, end_time: 1668784604.823, elapsed_time: 0.43621, records_examined: 0, records_loaded: 0) 2022-11-18 15:16:50 [notice] (action: xdmod.staging-ingest-jobs.union-user-pi--user (ETL\Ingestor\DatabaseIngestor), start_time: 1668784606.7596, end_time: 1668784609.7057, elapsed_time: 2.94608, records_examined: 0, records_loaded: 0) 2022-11-18 15:16:51 [notice] (action: xdmod.staging-ingest-jobs.union-user-pi-resource--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668784611.0508, end_time: 1668784611.1843, elapsed_time: 0.13348, records_examined: 0, records_loaded: 0) 2022-11-18 15:16:54 [notice] (action: xdmod.staging-ingest-jobs.union-user-pi-resource--user (ETL\Ingestor\DatabaseIngestor), start_time: 1668784612.3854, end_time: 1668784614.0458, elapsed_time: 1.66036, records_examined: 0, records_loaded: 0) 2022-11-18 15:16:57 [notice] (action: xdmod.staging-ingest-jobs.user-pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668784614.9515, end_time: 1668784616.8448, elapsed_time: 1.89332, records_examined: 0, records_loaded: 0) 2022-11-18 15:17:01 [notice] (action: xdmod.staging-ingest-jobs.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668784618.3334, end_time: 1668784619.695, elapsed_time: 1.36163, records_examined: 0, records_loaded: 0) 2022-11-18 15:17:03 [notice] (action: xdmod.staging-ingest-jobs.job (ETL\Ingestor\DatabaseIngestor), start_time: 1668784623.3395, end_time: 1668784623.7859, elapsed_time: 0.44644, records_examined: 0, records_loaded: 0) 2022-11-18 15:17:04 [notice] Finished processing section 'xdmod.staging-ingest-jobs' 2022-11-18 15:18:08 [notice] Start processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 15:18:13 [notice] (action: xdmod.hpcdb-ingest-common.resource-types (ETL\Ingestor\DatabaseIngestor), start_time: 1668784690.234, end_time: 1668784691.9436, elapsed_time: 1.70954, records_examined: 0, records_loaded: 0) 2022-11-18 15:18:20 [notice] (action: xdmod.hpcdb-ingest-common.organizations (ETL\Ingestor\DatabaseIngestor), start_time: 1668784697.3268, end_time: 1668784700.0304, elapsed_time: 2.70363, records_examined: 0, records_loaded: 0) 2022-11-18 15:18:27 [notice] (action: xdmod.hpcdb-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668784701.8161, end_time: 1668784705.5064, elapsed_time: 3.69028, records_examined: 0, records_loaded: 0) 2022-11-18 15:18:39 [notice] (action: xdmod.hpcdb-ingest-common.resource-specs (ETL\Ingestor\DatabaseIngestor), start_time: 1668784712.7789, end_time: 1668784718.2103, elapsed_time: 5.43141, records_examined: 0, records_loaded: 0) 2022-11-18 15:18:41 [notice] (action: xdmod.hpcdb-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668784719.7348, end_time: 1668784721.3563, elapsed_time: 1.62155, records_examined: 4, records_loaded: 4) 2022-11-18 15:18:48 [notice] (action: xdmod.hpcdb-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668784724.9387, end_time: 1668784726.3175, elapsed_time: 1.37878, records_examined: 0, records_loaded: 0) 2022-11-18 15:18:53 [notice] (action: xdmod.hpcdb-ingest-common.accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668784729.1467, end_time: 1668784732.7066, elapsed_time: 3.55989, records_examined: 0, records_loaded: 0) 2022-11-18 15:18:58 [notice] (action: xdmod.hpcdb-ingest-common.allocations (ETL\Ingestor\DatabaseIngestor), start_time: 1668784734.3706, end_time: 1668784737.1795, elapsed_time: 2.80893, records_examined: 0, records_loaded: 0) 2022-11-18 15:19:09 [notice] (action: xdmod.hpcdb-ingest-common.people (ETL\Ingestor\DatabaseIngestor), start_time: 1668784743.8084, end_time: 1668784749.2755, elapsed_time: 5.46714, records_examined: 0, records_loaded: 0) 2022-11-18 15:19:11 [notice] (action: xdmod.hpcdb-ingest-common.requests (ETL\Ingestor\DatabaseIngestor), start_time: 1668784749.903, end_time: 1668784750.2826, elapsed_time: 0.37961, records_examined: 0, records_loaded: 0) 2022-11-18 15:19:20 [notice] (action: xdmod.hpcdb-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668784751.8174, end_time: 1668784757.9268, elapsed_time: 6.10939, records_examined: 0, records_loaded: 0) 2022-11-18 15:19:39 [notice] (action: xdmod.hpcdb-ingest-common.allocations-on-resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668784771.7391, end_time: 1668784777.6411, elapsed_time: 5.90202, records_examined: 0, records_loaded: 0) 2022-11-18 15:19:48 [notice] (action: xdmod.hpcdb-ingest-common.email-addresses (ETL\Ingestor\DatabaseIngestor), start_time: 1668784781.4544, end_time: 1668784788.7725, elapsed_time: 7.31809, records_examined: 0, records_loaded: 0) 2022-11-18 15:19:54 [notice] (action: xdmod.hpcdb-ingest-common.principal-investigators (ETL\Ingestor\DatabaseIngestor), start_time: 1668784792.1679, end_time: 1668784794.1991, elapsed_time: 2.03121, records_examined: 0, records_loaded: 0) 2022-11-18 15:20:04 [notice] (action: xdmod.hpcdb-ingest-common.system-accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668784797.6092, end_time: 1668784804.3075, elapsed_time: 6.69827, records_examined: 100, records_loaded: 100) 2022-11-18 15:20:08 [notice] Finished processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 15:20:08 [notice] Start processing section 'xdmod.hpcdb-ingest-jobs' 2022-11-18 15:20:08 [notice] (action: xdmod.hpcdb-ingest-jobs.jobs (ETL\Ingestor\DatabaseIngestor), start_time: 1668784808.3429, end_time: 1668784808.4538, elapsed_time: 0.11093, records_examined: 0, records_loaded: 0) 2022-11-18 15:20:09 [notice] Finished processing section 'xdmod.hpcdb-ingest-jobs' 2022-11-18 15:20:28 [notice] Start processing section 'xdmod.hpcdb-prep-xdw-job-ingest-by-new-jobs' 2022-11-18 15:20:36 [notice] (action: xdmod.hpcdb-prep-xdw-job-ingest-by-new-jobs.jobs (ETL\Ingestor\DatabaseIngestor), start_time: 1668784829.8167, end_time: 1668784836.2547, elapsed_time: 6.43802, records_examined: 0, records_loaded: 0) 2022-11-18 15:20:36 [notice] Finished processing section 'xdmod.hpcdb-prep-xdw-job-ingest-by-new-jobs' 2022-11-18 15:22:00 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 15:22:07 [notice] (action: xdmod.hpcdb-xdw-ingest-common.account (ETL\Ingestor\DatabaseIngestor), start_time: 1668784921.5212, end_time: 1668784925.2007, elapsed_time: 3.67947, records_examined: 0, records_loaded: 0) 2022-11-18 15:22:16 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668784931.1911, end_time: 1668784934.9637, elapsed_time: 3.77259, records_examined: 0, records_loaded: 0) 2022-11-18 15:22:21 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation (ETL\Ingestor\DatabaseIngestor), start_time: 1668784939.9195, end_time: 1668784941.4618, elapsed_time: 1.54227, records_examined: 0, records_loaded: 0) 2022-11-18 15:22:28 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668784941.6708, end_time: 1668784946.8963, elapsed_time: 5.22546, records_examined: 0, records_loaded: 0) 2022-11-18 15:22:36 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-on-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668784953.1282, end_time: 1668784955.5831, elapsed_time: 2.45489, records_examined: 0, records_loaded: 0) 2022-11-18 15:22:49 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science (ETL\Ingestor\DatabaseIngestor), start_time: 1668784961.6196, end_time: 1668784968.4122, elapsed_time: 6.79257, records_examined: 0, records_loaded: 0) 2022-11-18 15:23:15 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science-hierarchy (ETL\Ingestor\DatabaseIngestor), start_time: 1668784974.3063, end_time: 1668784994.6917, elapsed_time: 20.38541, records_examined: 1, records_loaded: 1) 2022-11-18 15:23:20 [notice] (action: xdmod.hpcdb-xdw-ingest-common.organization (ETL\Ingestor\DatabaseIngestor), start_time: 1668784996.7588, end_time: 1668785000.5331, elapsed_time: 3.77423, records_examined: 1, records_loaded: 1)

rob-baron commented 10 months ago

022-11-18 15:23:27 [notice] (action: xdmod.hpcdb-xdw-ingest-common.pi-person (ETL\Ingestor\DatabaseIngestor), start_time: 1668785003.3238, end_time: 1668785007.3511, elapsed_time: 4.02729, records_examined: 0, records_loaded: 0) 2022-11-18 15:23:36 [notice] (action: xdmod.hpcdb-xdw-ingest-common.person (ETL\Ingestor\DatabaseIngestor), start_time: 1668785008.8337, end_time: 1668785015.4884, elapsed_time: 6.65465, records_examined: 50, records_loaded: 50) 2022-11-18 15:23:40 [notice] (action: xdmod.hpcdb-xdw-ingest-common.people-under-pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668785017.6497, end_time: 1668785020.5988, elapsed_time: 2.94914, records_examined: 0, records_loaded: 0) 2022-11-18 15:23:47 [notice] (action: xdmod.hpcdb-xdw-ingest-common.principal-investigator (ETL\Ingestor\DatabaseIngestor), start_time: 1668785023.1016, end_time: 1668785027.1134, elapsed_time: 4.01177, records_examined: 0, records_loaded: 0) 2022-11-18 15:23:52 [notice] (action: xdmod.hpcdb-xdw-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668785029.7408, end_time: 1668785032.2371, elapsed_time: 2.49633, records_examined: 0, records_loaded: 0) 2022-11-18 15:23:56 [notice] (action: xdmod.hpcdb-xdw-ingest-common.request (ETL\Ingestor\DatabaseIngestor), start_time: 1668785035.0669, end_time: 1668785036.0499, elapsed_time: 0.983, records_examined: 0, records_loaded: 0) 2022-11-18 15:24:01 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-type (ETL\Ingestor\DatabaseIngestor), start_time: 1668785036.3329, end_time: 1668785040.6806, elapsed_time: 4.34767, records_examined: 0, records_loaded: 0) 2022-11-18 15:24:06 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-spec (ETL\Ingestor\DatabaseIngestor), start_time: 1668785044.0567, end_time: 1668785045.9007, elapsed_time: 1.84399, records_examined: 0, records_loaded: 0) 2022-11-18 15:24:13 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668785049.115, end_time: 1668785052.0253, elapsed_time: 2.91031, records_examined: 0, records_loaded: 0) 2022-11-18 15:24:19 [notice] (action: xdmod.hpcdb-xdw-ingest-common.service-provider (ETL\Ingestor\DatabaseIngestor), start_time: 1668785054.7688, end_time: 1668785057.5691, elapsed_time: 2.8003, records_examined: 0, records_loaded: 0) 2022-11-18 15:24:32 [notice] (action: xdmod.hpcdb-xdw-ingest-common.system-account (ETL\Ingestor\DatabaseIngestor), start_time: 1668785066.1015, end_time: 1668785071.6608, elapsed_time: 5.55923, records_examined: 100, records_loaded: 100) 2022-11-18 15:24:34 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 15:24:36 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-jobs' 2022-11-18 15:24:42 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.node-count (ETL\Ingestor\DatabaseIngestor), start_time: 1668785078.7233, end_time: 1668785081.8562, elapsed_time: 3.13293, records_examined: 0, records_loaded: 0) 2022-11-18 15:24:47 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.queue (ETL\Ingestor\DatabaseIngestor), start_time: 1668785082.8273, end_time: 1668785086.8332, elapsed_time: 4.00588, records_examined: 0, records_loaded: 0) 2022-11-18 15:24:59 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.hosts (ETL\Ingestor\HpcdbHostsIngestor), start_time: 1668785093.977, end_time: 1668785099.0514, elapsed_time: 5.07443, records_examined: 0, records_loaded: 0) 2022-11-18 15:25:04 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.job-hosts (ETL\Ingestor\DatabaseIngestor), start_time: 1668785101.2075, end_time: 1668785103.6912, elapsed_time: 2.48375, records_examined: 0, records_loaded: 0) 2022-11-18 15:25:05 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/jobs/jobhosts-post-ingest-truncate.sql' using delimiter '//' containing 1 statements 2022-11-18 15:25:10 [notice] Finished Processing 1 SQL statements 2022-11-18 15:25:11 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.jobhosts-post-truncate (ETL\Maintenance\ExecuteSql), start_time: 1668785105.9512, end_time: 1668785111.208, elapsed_time: 5.2568) 2022-11-18 15:25:12 [notice] Altering table modw.job_records 2022-11-18 15:25:13 [notice] Altering table modw.job_tasks 2022-11-18 15:25:22 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.job-records-job-tasks (ETL\Ingestor\DatabaseIngestor), start_time: 1668785112.8485, end_time: 1668785121.4894, elapsed_time: 8.64084, records_examined: 0, records_loaded: 0) 2022-11-18 15:25:26 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/jobs/job-record-hpc-post-ingest.sql' using delimiter '//' containing 2 statements 2022-11-18 15:25:32 [notice] Finished Processing 2 SQL statements 2022-11-18 15:25:33 [notice] (action: xdmod.hpcdb-xdw-ingest-jobs.HpcdbPostIngestJobUpdates (ETL\Maintenance\ExecuteSql), start_time: 1668785126.7318, end_time: 1668785133.2198, elapsed_time: 6.48792) 2022-11-18 15:25:34 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-jobs' 2022-11-18 15:25:36 [notice] Ingesting generic cloud log files 2022-11-18 15:29:41 [notice] Start processing section 'xdmod.staging-ingest-common' 2022-11-18 15:29:45 [notice] (action: xdmod.staging-ingest-common.organization (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785381.9173, end_time: 1668785384.8968, elapsed_time: 2.97949, records_examined: 1, records_loaded: 1) 2022-11-18 15:30:00 [notice] (action: xdmod.staging-ingest-common.resource-types (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785389.913, end_time: 1668785400.015, elapsed_time: 10.10196, records_examined: 12, records_loaded: 12) 2022-11-18 15:30:03 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-11-18 15:30:07 [notice] Finished Processing 2 SQL statements 2022-11-18 15:30:08 [notice] (action: xdmod.staging-ingest-common.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1668785403.2925, end_time: 1668785408.2379, elapsed_time: 4.94547) 2022-11-18 15:30:24 [notice] (action: xdmod.staging-ingest-common.resource-config (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785411.0117, end_time: 1668785424.4737, elapsed_time: 13.46202, records_examined: 4, records_loaded: 4) 2022-11-18 15:30:36 [notice] (action: xdmod.staging-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668785428.6098, end_time: 1668785434.0443, elapsed_time: 5.43457, records_examined: 0, records_loaded: 0) 2022-11-18 15:30:55 [notice] (action: xdmod.staging-ingest-common.resource-specs (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785443.5053, end_time: 1668785454.7586, elapsed_time: 11.25327, records_examined: 4, records_loaded: 4) 2022-11-18 15:30:58 [notice] Finished processing section 'xdmod.staging-ingest-common' 2022-11-18 15:30:59 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-11-18 15:31:01 [notice] Altering table modw_cloud.event 2022-11-18 15:31:03 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 15:31:06 [notice] Altering table modw_cloud.session_records 2022-11-18 15:31:10 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1668785460.959, end_time: 1668785470.2929, elapsed_time: 9.33397) 2022-11-18 15:31:17 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785475.5737, end_time: 1668785477.8627, elapsed_time: 2.28902, records_examined: 5, records_loaded: 5) 2022-11-18 15:31:26 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785478.8295, end_time: 1668785484.0813, elapsed_time: 5.25176, records_examined: 65, records_loaded: 65) 2022-11-18 15:31:36 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785489.2681, end_time: 1668785494.0372, elapsed_time: 4.76907, records_examined: 4, records_loaded: 4) 2022-11-18 15:31:47 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785499.6617, end_time: 1668785505.9797, elapsed_time: 6.31803, records_examined: 1, records_loaded: 1) 2022-11-18 15:31:53 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785512.5794, end_time: 1668785513.3668, elapsed_time: 0.7874, records_examined: 1, records_loaded: 1) 2022-11-18 15:31:58 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785513.9859, end_time: 1668785517.3737, elapsed_time: 3.38786, records_examined: 1, records_loaded: 1) 2022-11-18 15:32:03 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785519.576, end_time: 1668785523.0599, elapsed_time: 3.48395, records_examined: 1, records_loaded: 1) 2022-11-18 15:32:12 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785526.7866, end_time: 1668785531.231, elapsed_time: 4.44435, records_examined: 1, records_loaded: 1) 2022-11-18 15:32:15 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785533.5797, end_time: 1668785534.929, elapsed_time: 1.34937, records_examined: 8, records_loaded: 8) 2022-11-18 15:32:17 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668785536.8794, end_time: 1668785537.5652, elapsed_time: 0.68576, records_examined: 7, records_loaded: 7) 2022-11-18 15:32:19 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-11-18 15:32:21 [notice] Start processing section 'xdmod.jobs-cloud-import-users-generic' 2022-11-18 15:32:25 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudStagingUserIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668785543.2196, end_time: 1668785545.1245, elapsed_time: 1.90498, records_examined: 0, records_loaded: 0) 2022-11-18 15:32:39 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudStagingUserResourceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668785551.4228, end_time: 1668785557.1382, elapsed_time: 5.7154, records_examined: 0, records_loaded: 0) 2022-11-18 15:32:58 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingHpcdbPeopleIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668785567.787, end_time: 1668785574.1425, elapsed_time: 6.35545, records_examined: 0, records_loaded: 0) 2022-11-18 15:33:18 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudloudStagingHpcdbSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668785583.511, end_time: 1668785597.1943, elapsed_time: 13.68334, records_examined: 100, records_loaded: 100) 2022-11-18 15:33:37 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668785601.8288, end_time: 1668785611.7874, elapsed_time: 9.95852, records_examined: 100, records_loaded: 100) 2022-11-18 15:34:01 [notice] (action: xdmod.jobs-cloud-import-users-generic.GenericCloudCloudStagingPersonIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668785621.7021, end_time: 1668785640.0259, elapsed_time: 18.32373, records_examined: 50, records_loaded: 50) 2022-11-18 15:34:03 [notice] Finished processing section 'xdmod.jobs-cloud-import-users-generic' 2022-11-18 15:34:03 [notice] Start processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 15:34:13 [notice] (action: xdmod.hpcdb-ingest-common.resource-types (ETL\Ingestor\DatabaseIngestor), start_time: 1668785645.7489, end_time: 1668785652.0104, elapsed_time: 6.26147, records_examined: 0, records_loaded: 0) 2022-11-18 15:34:27 [notice] (action: xdmod.hpcdb-ingest-common.organizations (ETL\Ingestor\DatabaseIngestor), start_time: 1668785659.0448, end_time: 1668785666.1576, elapsed_time: 7.11285, records_examined: 0, records_loaded: 0) 2022-11-18 15:34:42 [notice] (action: xdmod.hpcdb-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668785672.7121, end_time: 1668785680.1311, elapsed_time: 7.419, records_examined: 0, records_loaded: 0) 2022-11-18 15:34:55 [notice] (action: xdmod.hpcdb-ingest-common.resource-specs (ETL\Ingestor\DatabaseIngestor), start_time: 1668785685.8025, end_time: 1668785692.8713, elapsed_time: 7.06877, records_examined: 0, records_loaded: 0) 2022-11-18 15:35:11 [notice] (action: xdmod.hpcdb-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668785701.205, end_time: 1668785709.4952, elapsed_time: 8.29019, records_examined: 4, records_loaded: 4) 2022-11-18 15:35:17 [notice] (action: xdmod.hpcdb-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668785715.794, end_time: 1668785717.8061, elapsed_time: 2.01209, records_examined: 0, records_loaded: 0) 2022-11-18 15:35:32 [notice] (action: xdmod.hpcdb-ingest-common.accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668785726.4037, end_time: 1668785732.0859, elapsed_time: 5.68219, records_examined: 0, records_loaded: 0) 2022-11-18 15:35:49 [notice] (action: xdmod.hpcdb-ingest-common.allocations (ETL\Ingestor\DatabaseIngestor), start_time: 1668785740.1236, end_time: 1668785747.4673, elapsed_time: 7.34369, records_examined: 0, records_loaded: 0) 2022-11-18 15:35:59 [notice] (action: xdmod.hpcdb-ingest-common.people (ETL\Ingestor\DatabaseIngestor), start_time: 1668785752.7028, end_time: 1668785757.6935, elapsed_time: 4.99073, records_examined: 0, records_loaded: 0) 2022-11-18 15:36:13 [notice] (action: xdmod.hpcdb-ingest-common.requests (ETL\Ingestor\DatabaseIngestor), start_time: 1668785765.5252, end_time: 1668785771.5152, elapsed_time: 5.98998, records_examined: 0, records_loaded: 0) 2022-11-18 15:36:29 [notice] (action: xdmod.hpcdb-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668785782.0486, end_time: 1668785788.3145, elapsed_time: 6.26592, records_examined: 0, records_loaded: 0) 2022-11-18 15:36:40 [notice] (action: xdmod.hpcdb-ingest-common.allocations-on-resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668785795.3366, end_time: 1668785799.5126, elapsed_time: 4.17599, records_examined: 0, records_loaded: 0) 2022-11-18 15:36:50 [notice] (action: xdmod.hpcdb-ingest-common.email-addresses (ETL\Ingestor\DatabaseIngestor), start_time: 1668785804.7025, end_time: 1668785808.2794, elapsed_time: 3.57692, records_examined: 0, records_loaded: 0) 2022-11-18 15:36:59 [notice] (action: xdmod.hpcdb-ingest-common.principal-investigators (ETL\Ingestor\DatabaseIngestor), start_time: 1668785814.5478, end_time: 1668785819.5032, elapsed_time: 4.95545, records_examined: 0, records_loaded: 0) 2022-11-18 15:37:20 [notice] (action: xdmod.hpcdb-ingest-common.system-accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668785825.7141, end_time: 1668785835.0916, elapsed_time: 9.37754, records_examined: 100, records_loaded: 100) 2022-11-18 15:37:22 [notice] Finished processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 15:37:23 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 15:37:30 [notice] (action: xdmod.hpcdb-xdw-ingest-common.account (ETL\Ingestor\DatabaseIngestor), start_time: 1668785843.922, end_time: 1668785849.286, elapsed_time: 5.36399, records_examined: 0, records_loaded: 0) 2022-11-18 15:37:39 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668785854.6044, end_time: 1668785856.6997, elapsed_time: 2.09525, records_examined: 0, records_loaded: 0) 2022-11-18 15:37:52 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation (ETL\Ingestor\DatabaseIngestor), start_time: 1668785860.5856, end_time: 1668785870.7739, elapsed_time: 10.18825, records_examined: 0, records_loaded: 0) 2022-11-18 15:38:11 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668785877.5465, end_time: 1668785890.0136, elapsed_time: 12.46701, records_examined: 0, records_loaded: 0) 2022-11-18 15:38:29 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-on-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668785899.6472, end_time: 1668785907.7101, elapsed_time: 8.06292, records_examined: 0, records_loaded: 0) 2022-11-18 15:38:44 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science (ETL\Ingestor\DatabaseIngestor), start_time: 1668785914.6843, end_time: 1668785924.0804, elapsed_time: 9.39607, records_examined: 0, records_loaded: 0) 2022-11-18 15:39:25 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science-hierarchy (ETL\Ingestor\DatabaseIngestor), start_time: 1668785935.3505, end_time: 1668785963.5371, elapsed_time: 28.18656, records_examined: 1, records_loaded: 1) 2022-11-18 15:40:09 [notice] (action: xdmod.hpcdb-xdw-ingest-common.organization (ETL\Ingestor\DatabaseIngestor), start_time: 1668785977.9494, end_time: 1668786009.4507, elapsed_time: 31.50131, records_examined: 1, records_loaded: 1) 2022-11-18 15:40:12 [notice] (action: xdmod.hpcdb-xdw-ingest-common.pi-person (ETL\Ingestor\DatabaseIngestor), start_time: 1668786009.7356, end_time: 1668786011.7365, elapsed_time: 2.00081, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:15 [notice] (action: xdmod.hpcdb-xdw-ingest-common.person (ETL\Ingestor\DatabaseIngestor), start_time: 1668786012.6905, end_time: 1668786015.4383, elapsed_time: 2.74783, records_examined: 50, records_loaded: 50) 2022-11-18 15:40:17 [notice] (action: xdmod.hpcdb-xdw-ingest-common.people-under-pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668786016.7104, end_time: 1668786017.7468, elapsed_time: 1.03641, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:22 [notice] (action: xdmod.hpcdb-xdw-ingest-common.principal-investigator (ETL\Ingestor\DatabaseIngestor), start_time: 1668786019.8412, end_time: 1668786020.8248, elapsed_time: 0.98357, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:26 [notice] (action: xdmod.hpcdb-xdw-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668786023.8029, end_time: 1668786026.5912, elapsed_time: 2.78835, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:29 [notice] (action: xdmod.hpcdb-xdw-ingest-common.request (ETL\Ingestor\DatabaseIngestor), start_time: 1668786027.512, end_time: 1668786028.7244, elapsed_time: 1.21237, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:31 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-type (ETL\Ingestor\DatabaseIngestor), start_time: 1668786030.7415, end_time: 1668786031.5974, elapsed_time: 0.85593, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:32 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-spec (ETL\Ingestor\DatabaseIngestor), start_time: 1668786032.0122, end_time: 1668786032.6637, elapsed_time: 0.65143, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:35 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668786033.8581, end_time: 1668786035.0971, elapsed_time: 1.23898, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:41 [notice] (action: xdmod.hpcdb-xdw-ingest-common.service-provider (ETL\Ingestor\DatabaseIngestor), start_time: 1668786035.7186, end_time: 1668786040.1931, elapsed_time: 4.4745, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:45 [notice] (action: xdmod.hpcdb-xdw-ingest-common.system-account (ETL\Ingestor\DatabaseIngestor), start_time: 1668786044.3656, end_time: 1668786045.5147, elapsed_time: 1.14911, records_examined: 100, records_loaded: 100) 2022-11-18 15:40:46 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 15:40:46 [notice] Start processing section 'xdmod.jobs-cloud-extract-generic' 2022-11-18 15:40:49 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786047.789, end_time: 1668786048.9938, elapsed_time: 1.2048, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:51 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudHostIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786050.4455, end_time: 1668786051.1946, elapsed_time: 0.7491, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:55 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudImageIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786053.3473, end_time: 1668786054.9568, elapsed_time: 1.60947, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:56 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceTypeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786056.0812, end_time: 1668786056.4545, elapsed_time: 0.37334, records_examined: 0, records_loaded: 0) 2022-11-18 15:40:59 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786057.5881, end_time: 1668786059.46, elapsed_time: 1.87192, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:00 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudVolumeAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786060.1791, end_time: 1668786060.3533, elapsed_time: 0.17426, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:00 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_generic/post_ingest_updates.sql' using delimiter '//' containing 4 statements 2022-11-18 15:41:02 [notice] Finished Processing 4 SQL statements 2022-11-18 15:41:02 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudPostIngestUpdates (ETL\Maintenance\ExecuteSql), start_time: 1668786060.5145, end_time: 1668786062.1108, elapsed_time: 1.59634) 2022-11-18 15:41:03 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudStagingEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786062.5423, end_time: 1668786063.7749, elapsed_time: 1.2326, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:04 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudRootVolumeAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786064.1477, end_time: 1668786064.4247, elapsed_time: 0.27701, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:05 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudIpAddressAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786064.6175, end_time: 1668786064.8856, elapsed_time: 0.26808, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:05 [notice] Altering table modw_cloud.event 2022-11-18 15:41:06 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786065.2772, end_time: 1668786066.0469, elapsed_time: 0.76968, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:07 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudInstanceDataIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786066.6347, end_time: 1668786067.4012, elapsed_time: 0.76653, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:09 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventAssetRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786068.2914, end_time: 1668786069.341, elapsed_time: 1.0496, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:10 [notice] (action: xdmod.jobs-cloud-extract-generic.GenericCloudEventAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786070.1756, end_time: 1668786070.7249, elapsed_time: 0.54939, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:10 [notice] Finished processing section 'xdmod.jobs-cloud-extract-generic' 2022-11-18 15:41:11 [notice] Start processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 15:41:12 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668786071.6797, end_time: 1668786072.2657, elapsed_time: 0.58601, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:14 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668786073.4067, end_time: 1668786074.2347, elapsed_time: 0.82799, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:16 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668786075.5558, end_time: 1668786076.3725, elapsed_time: 0.81667, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:17 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi-resource--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668786076.9583, end_time: 1668786077.5305, elapsed_time: 0.57224, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:19 [notice] (action: xdmod.jobs-cloud-ingest-pi.user-pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668786078.7634, end_time: 1668786079.2739, elapsed_time: 0.51054, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:19 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_common/pi_to_account.sql' using delimiter '//' containing 1 statements 2022-11-18 15:41:20 [notice] Finished Processing 1 SQL statements 2022-11-18 15:41:20 [notice] (action: xdmod.jobs-cloud-ingest-pi.UpdateCloudProjectToPI (ETL\Maintenance\ExecuteSql), start_time: 1668786079.6596, end_time: 1668786080.2552, elapsed_time: 0.59556) 2022-11-18 15:41:20 [notice] Finished processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 15:41:20 [notice] Ingesting cloud resource specs 2022-11-18 15:41:24 [notice] Start processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 15:41:25 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 15:41:30 [notice] (action: xdmod.ingest-cloud-resource-specs.CloudResourceSpecsReconstructor (ETL\Ingestor\CloudResourceSpecsStateTransformIngestor), start_time: 1668786085.5054, end_time: 1668786090.1142, elapsed_time: 4.60872, records_examined: 0, records_loaded: 0) 2022-11-18 15:41:31 [notice] Finished processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 15:41:31 [notice] Ingesting OpenStack event log data 2022-11-18 15:47:10 [notice] Start processing section 'xdmod.staging-ingest-common' 2022-11-18 15:47:25 [notice] (action: xdmod.staging-ingest-common.organization (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786431.8977, end_time: 1668786445.394, elapsed_time: 13.49628, records_examined: 1, records_loaded: 1) 2022-11-18 15:47:42 [notice] (action: xdmod.staging-ingest-common.resource-types (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786452.3062, end_time: 1668786461.9047, elapsed_time: 9.59855, records_examined: 12, records_loaded: 12) 2022-11-18 15:47:46 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-11-18 15:47:50 [notice] Finished Processing 2 SQL statements 2022-11-18 15:47:52 [notice] (action: xdmod.staging-ingest-common.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1668786466.3279, end_time: 1668786472.1428, elapsed_time: 5.81489) 2022-11-18 15:48:11 [notice] (action: xdmod.staging-ingest-common.resource-config (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786478.4873, end_time: 1668786490.7869, elapsed_time: 12.29951, records_examined: 4, records_loaded: 4) 2022-11-18 15:48:18 [notice] (action: xdmod.staging-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668786493.6249, end_time: 1668786496.185, elapsed_time: 2.56012, records_examined: 0, records_loaded: 0) 2022-11-18 15:48:27 [notice] (action: xdmod.staging-ingest-common.resource-specs (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786500.9609, end_time: 1668786507.143, elapsed_time: 6.18218, records_examined: 4, records_loaded: 4) 2022-11-18 15:48:28 [notice] Finished processing section 'xdmod.staging-ingest-common' 2022-11-18 15:48:28 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-11-18 15:48:30 [notice] Altering table modw_cloud.event 2022-11-18 15:48:31 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 15:48:32 [notice] Altering table modw_cloud.session_records 2022-11-18 15:48:32 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1668786510.5571, end_time: 1668786512.9091, elapsed_time: 2.35207) 2022-11-18 15:48:34 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786513.7597, end_time: 1668786514.1754, elapsed_time: 0.41575, records_examined: 5, records_loaded: 5) 2022-11-18 15:48:34 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786514.481, end_time: 1668786514.782, elapsed_time: 0.301, records_examined: 65, records_loaded: 65) 2022-11-18 15:48:36 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786515.4055, end_time: 1668786515.8163, elapsed_time: 0.41077, records_examined: 4, records_loaded: 4) 2022-11-18 15:48:36 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786516.2983, end_time: 1668786516.6357, elapsed_time: 0.33738, records_examined: 1, records_loaded: 1) 2022-11-18 15:48:37 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786517.141, end_time: 1668786517.419, elapsed_time: 0.27805, records_examined: 1, records_loaded: 1) 2022-11-18 15:48:38 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786517.621, end_time: 1668786518.1835, elapsed_time: 0.56252, records_examined: 1, records_loaded: 1) 2022-11-18 15:48:39 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786518.5208, end_time: 1668786519.0898, elapsed_time: 0.56899, records_examined: 1, records_loaded: 1) 2022-11-18 15:48:40 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786519.5105, end_time: 1668786520.0581, elapsed_time: 0.5476, records_examined: 1, records_loaded: 1) 2022-11-18 15:48:41 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786520.8343, end_time: 1668786520.9881, elapsed_time: 0.15374, records_examined: 8, records_loaded: 8) 2022-11-18 15:48:42 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786521.8399, end_time: 1668786522.0232, elapsed_time: 0.18327, records_examined: 7, records_loaded: 7) 2022-11-18 15:48:42 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-11-18 15:48:42 [notice] Start processing section 'xdmod.jobs-cloud-import-users-openstack' 2022-11-18 15:48:43 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackStagingUserIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786522.9335, end_time: 1668786523.68, elapsed_time: 0.74651, records_examined: 0, records_loaded: 0) 2022-11-18 15:48:45 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingUserResourceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786524.3078, end_time: 1668786525.076, elapsed_time: 0.76814, records_examined: 0, records_loaded: 0) 2022-11-18 15:48:46 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingHpcdbPeopleIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786526.1385, end_time: 1668786526.7293, elapsed_time: 0.59079, records_examined: 0, records_loaded: 0) 2022-11-18 15:48:49 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingHpcdbSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786527.225, end_time: 1668786528.9824, elapsed_time: 1.7574, records_examined: 100, records_loaded: 100) 2022-11-18 15:48:54 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingSystemAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786530.1381, end_time: 1668786534.4472, elapsed_time: 4.30911, records_examined: 100, records_loaded: 100) 2022-11-18 15:49:04 [notice] (action: xdmod.jobs-cloud-import-users-openstack.OpenStackCloudStagingPersonIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786536.5447, end_time: 1668786542.7512, elapsed_time: 6.20647, records_examined: 50, records_loaded: 50) 2022-11-18 15:49:05 [notice] Finished processing section 'xdmod.jobs-cloud-import-users-openstack' 2022-11-18 15:49:06 [notice] Start processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 15:49:18 [notice] (action: xdmod.hpcdb-ingest-common.resource-types (ETL\Ingestor\DatabaseIngestor), start_time: 1668786554.6855, end_time: 1668786557.5738, elapsed_time: 2.88836, records_examined: 0, records_loaded: 0) 2022-11-18 15:49:26 [notice] (action: xdmod.hpcdb-ingest-common.organizations (ETL\Ingestor\DatabaseIngestor), start_time: 1668786563.4645, end_time: 1668786565.6422, elapsed_time: 2.17767, records_examined: 0, records_loaded: 0) 2022-11-18 15:49:35 [notice] (action: xdmod.hpcdb-ingest-common.resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668786570.949, end_time: 1668786572.5724, elapsed_time: 1.62339, records_examined: 0, records_loaded: 0) 2022-11-18 15:49:47 [notice] (action: xdmod.hpcdb-ingest-common.resource-specs (ETL\Ingestor\DatabaseIngestor), start_time: 1668786579.3049, end_time: 1668786586.3822, elapsed_time: 7.07734, records_examined: 0, records_loaded: 0) 2022-11-18 15:50:01 [notice] (action: xdmod.hpcdb-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668786591.0982, end_time: 1668786599.4541, elapsed_time: 8.35589, records_examined: 4, records_loaded: 4) 2022-11-18 15:50:09 [notice] (action: xdmod.hpcdb-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668786605.1414, end_time: 1668786608.3584, elapsed_time: 3.217, records_examined: 0, records_loaded: 0) 2022-11-18 15:50:17 [notice] (action: xdmod.hpcdb-ingest-common.accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668786611.5582, end_time: 1668786615.9675, elapsed_time: 4.40926, records_examined: 0, records_loaded: 0) 2022-11-18 15:50:28 [notice] (action: xdmod.hpcdb-ingest-common.allocations (ETL\Ingestor\DatabaseIngestor), start_time: 1668786621.7378, end_time: 1668786626.3575, elapsed_time: 4.61962, records_examined: 0, records_loaded: 0) 2022-11-18 15:50:37 [notice] (action: xdmod.hpcdb-ingest-common.people (ETL\Ingestor\DatabaseIngestor), start_time: 1668786631.9655, end_time: 1668786635.4753, elapsed_time: 3.50979, records_examined: 0, records_loaded: 0) 2022-11-18 15:50:49 [notice] (action: xdmod.hpcdb-ingest-common.requests (ETL\Ingestor\DatabaseIngestor), start_time: 1668786639.6392, end_time: 1668786646.6672, elapsed_time: 7.02796, records_examined: 0, records_loaded: 0) 2022-11-18 15:50:59 [notice] (action: xdmod.hpcdb-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668786653.7306, end_time: 1668786658.4599, elapsed_time: 4.72931, records_examined: 0, records_loaded: 0) 2022-11-18 15:51:07 [notice] (action: xdmod.hpcdb-ingest-common.allocations-on-resources (ETL\Ingestor\DatabaseIngestor), start_time: 1668786663.1497, end_time: 1668786666.8372, elapsed_time: 3.68754, records_examined: 0, records_loaded: 0) 2022-11-18 15:51:12 [notice] (action: xdmod.hpcdb-ingest-common.email-addresses (ETL\Ingestor\DatabaseIngestor), start_time: 1668786668.8045, end_time: 1668786672.1152, elapsed_time: 3.31071, records_examined: 0, records_loaded: 0) 2022-11-18 15:51:16 [notice] (action: xdmod.hpcdb-ingest-common.principal-investigators (ETL\Ingestor\DatabaseIngestor), start_time: 1668786673.6563, end_time: 1668786675.7705, elapsed_time: 2.11416, records_examined: 0, records_loaded: 0) 2022-11-18 15:51:24 [notice] (action: xdmod.hpcdb-ingest-common.system-accounts (ETL\Ingestor\DatabaseIngestor), start_time: 1668786679.3717, end_time: 1668786684.3441, elapsed_time: 4.97242, records_examined: 100, records_loaded: 100) 2022-11-18 15:51:27 [notice] Finished processing section 'xdmod.hpcdb-ingest-common' 2022-11-18 15:51:27 [notice] Start processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 15:51:34 [notice] (action: xdmod.hpcdb-xdw-ingest-common.account (ETL\Ingestor\DatabaseIngestor), start_time: 1668786689.1485, end_time: 1668786693.2725, elapsed_time: 4.12394, records_examined: 0, records_loaded: 0) 2022-11-18 15:51:41 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-breakdown (ETL\Ingestor\DatabaseIngestor), start_time: 1668786696.6185, end_time: 1668786700.6241, elapsed_time: 4.00565, records_examined: 0, records_loaded: 0) 2022-11-18 15:51:50 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation (ETL\Ingestor\DatabaseIngestor), start_time: 1668786702.0213, end_time: 1668786708.5946, elapsed_time: 6.57329, records_examined: 0, records_loaded: 0) 2022-11-18 15:51:58 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668786712.2219, end_time: 1668786715.9823, elapsed_time: 3.76045, records_examined: 0, records_loaded: 0) 2022-11-18 15:52:05 [notice] (action: xdmod.hpcdb-xdw-ingest-common.allocation-on-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668786721.5929, end_time: 1668786724.2905, elapsed_time: 2.69762, records_examined: 0, records_loaded: 0) 2022-11-18 15:52:14 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science (ETL\Ingestor\DatabaseIngestor), start_time: 1668786729.1379, end_time: 1668786732.5574, elapsed_time: 3.41953, records_examined: 0, records_loaded: 0) 2022-11-18 15:52:31 [notice] (action: xdmod.hpcdb-xdw-ingest-common.field-of-science-hierarchy (ETL\Ingestor\DatabaseIngestor), start_time: 1668786738.2687, end_time: 1668786750.8182, elapsed_time: 12.54946, records_examined: 1, records_loaded: 1) 2022-11-18 15:52:48 [notice] (action: xdmod.hpcdb-xdw-ingest-common.organization (ETL\Ingestor\DatabaseIngestor), start_time: 1668786755.9316, end_time: 1668786767.8711, elapsed_time: 11.93947, records_examined: 1, records_loaded: 1) 2022-11-18 15:52:59 [notice] (action: xdmod.hpcdb-xdw-ingest-common.pi-person (ETL\Ingestor\DatabaseIngestor), start_time: 1668786771.1661, end_time: 1668786777.9824, elapsed_time: 6.81635, records_examined: 0, records_loaded: 0) 2022-11-18 15:53:15 [notice] (action: xdmod.hpcdb-xdw-ingest-common.person (ETL\Ingestor\DatabaseIngestor), start_time: 1668786782.9748, end_time: 1668786794.6871, elapsed_time: 11.7123, records_examined: 50, records_loaded: 50) 2022-11-18 15:53:19 [notice] (action: xdmod.hpcdb-xdw-ingest-common.people-under-pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668786796.1165, end_time: 1668786799.0601, elapsed_time: 2.94354, records_examined: 0, records_loaded: 0) 2022-11-18 15:53:27 [notice] (action: xdmod.hpcdb-xdw-ingest-common.principal-investigator (ETL\Ingestor\DatabaseIngestor), start_time: 1668786801.7559, end_time: 1668786805.896, elapsed_time: 4.14017, records_examined: 0, records_loaded: 0) 2022-11-18 15:53:32 [notice] (action: xdmod.hpcdb-xdw-ingest-common.qos (ETL\Ingestor\DatabaseIngestor), start_time: 1668786810.7368, end_time: 1668786812.5928, elapsed_time: 1.85602, records_examined: 0, records_loaded: 0) 2022-11-18 15:53:34 [notice] (action: xdmod.hpcdb-xdw-ingest-common.request (ETL\Ingestor\DatabaseIngestor), start_time: 1668786814.6318, end_time: 1668786814.8323, elapsed_time: 0.20043, records_examined: 0, records_loaded: 0) 2022-11-18 15:53:42 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-type (ETL\Ingestor\DatabaseIngestor), start_time: 1668786818.8747, end_time: 1668786822.3165, elapsed_time: 3.44176, records_examined: 0, records_loaded: 0) 2022-11-18 15:53:52 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-spec (ETL\Ingestor\DatabaseIngestor), start_time: 1668786828.2974, end_time: 1668786831.8595, elapsed_time: 3.56206, records_examined: 0, records_loaded: 0) 2022-11-18 15:53:56 [notice] (action: xdmod.hpcdb-xdw-ingest-common.resource-allocated (ETL\Ingestor\DatabaseIngestor), start_time: 1668786835.9595, end_time: 1668786836.809, elapsed_time: 0.84945, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:03 [notice] (action: xdmod.hpcdb-xdw-ingest-common.service-provider (ETL\Ingestor\DatabaseIngestor), start_time: 1668786839.0096, end_time: 1668786841.9285, elapsed_time: 2.91892, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:08 [notice] (action: xdmod.hpcdb-xdw-ingest-common.system-account (ETL\Ingestor\DatabaseIngestor), start_time: 1668786844.3484, end_time: 1668786848.7334, elapsed_time: 4.38502, records_examined: 100, records_loaded: 100) 2022-11-18 15:54:08 [notice] Finished processing section 'xdmod.hpcdb-xdw-ingest-common' 2022-11-18 15:54:09 [notice] Start processing section 'xdmod.jobs-cloud-extract-openstack' 2022-11-18 15:54:11 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackEventMapInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786850.2743, end_time: 1668786851.6809, elapsed_time: 1.40655, records_examined: 60, records_loaded: 60) 2022-11-18 15:54:13 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudAccountIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786852.9455, end_time: 1668786853.1006, elapsed_time: 0.15509, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:14 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudHostIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786854.148, end_time: 1668786854.418, elapsed_time: 0.26997, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:21 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudImageIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786856.5231, end_time: 1668786860.5466, elapsed_time: 4.02359, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:26 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudInstanceTypeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786863.0118, end_time: 1668786865.9515, elapsed_time: 2.93968, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:31 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786869.3719, end_time: 1668786870.9856, elapsed_time: 1.61373, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:33 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackInstanceIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786872.5834, end_time: 1668786873.6972, elapsed_time: 1.11386, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:35 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_domain.sql' using delimiter '//' containing 1 statements 2022-11-18 15:54:36 [notice] Finished Processing 1 SQL statements 2022-11-18 15:54:36 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackUnknownDomainIngestor (ETL\Maintenance\ExecuteSql), start_time: 1668786875.8407, end_time: 1668786876.8629, elapsed_time: 1.02217) 2022-11-18 15:54:40 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786877.9507, end_time: 1668786879.2669, elapsed_time: 1.31619, records_examined: 0, records_loaded: 0) 2022-11-18 15:54:44 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainToSubmissionVenueMappingStagingIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1668786881.9098, end_time: 1668786884.4419, elapsed_time: 2.53205, records_examined: 1, records_loaded: 1) 2022-11-18 15:54:53 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackDomainToSubmissionVenueMappingIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786887.0002, end_time: 1668786892.6863, elapsed_time: 5.68606, records_examined: 1, records_loaded: 1) 2022-11-18 15:54:57 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudStagingEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786894.6175, end_time: 1668786897.5075, elapsed_time: 2.88991, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:04 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786899.2163, end_time: 1668786901.8572, elapsed_time: 2.64084, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:05 [notice] Altering table modw_cloud.event 2022-11-18 15:55:07 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786905.6761, end_time: 1668786907.6218, elapsed_time: 1.94569, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:09 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackInstanceDataIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786908.4857, end_time: 1668786908.7287, elapsed_time: 0.24302, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:11 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventAssetRootVolumeIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786909.4899, end_time: 1668786911.1937, elapsed_time: 1.70375, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:12 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudEventAssetIngestor (ETL\Ingestor\DatabaseIngestor), start_time: 1668786911.3881, end_time: 1668786911.6371, elapsed_time: 0.249, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:13 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/post_ingest_update.sql' using delimiter '//' containing 8 statements 2022-11-18 15:55:20 [notice] Finished Processing 8 SQL statements 2022-11-18 15:55:20 [notice] (action: xdmod.jobs-cloud-extract-openstack.OpenStackCloudPostIngestUpdates (ETL\Maintenance\ExecuteSql), start_time: 1668786913.3968, end_time: 1668786920.7416, elapsed_time: 7.34489) 2022-11-18 15:55:20 [notice] Finished processing section 'xdmod.jobs-cloud-extract-openstack' 2022-11-18 15:55:20 [notice] Start processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 15:55:25 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668786922.4458, end_time: 1668786924.4148, elapsed_time: 1.96905, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:27 [notice] (action: xdmod.jobs-cloud-ingest-pi.pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668786926.7081, end_time: 1668786926.9874, elapsed_time: 0.2793, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:31 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668786929.6903, end_time: 1668786931.7865, elapsed_time: 2.09627, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:33 [notice] (action: xdmod.jobs-cloud-ingest-pi.union-user-pi-resource--pi (ETL\Ingestor\DatabaseIngestor), start_time: 1668786931.9306, end_time: 1668786933.1503, elapsed_time: 1.21971, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:36 [notice] (action: xdmod.jobs-cloud-ingest-pi.user-pi-resource (ETL\Ingestor\DatabaseIngestor), start_time: 1668786935.1446, end_time: 1668786936.0473, elapsed_time: 0.90267, records_examined: 0, records_loaded: 0) 2022-11-18 15:55:37 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_common/pi_to_account.sql' using delimiter '//' containing 1 statements 2022-11-18 15:55:39 [notice] Finished Processing 1 SQL statements 2022-11-18 15:55:39 [notice] (action: xdmod.jobs-cloud-ingest-pi.UpdateCloudProjectToPI (ETL\Maintenance\ExecuteSql), start_time: 1668786937.4267, end_time: 1668786939.7187, elapsed_time: 2.29201) 2022-11-18 15:55:43 [notice] Finished processing section 'xdmod.jobs-cloud-ingest-pi' 2022-11-18 15:55:46 [notice] Ingesting cloud resource specs 2022-11-18 15:55:58 [notice] Start processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 15:56:01 [notice] Altering table modw_cloud.cloud_resource_specs 2022-11-18 15:56:09 [notice] (action: xdmod.ingest-cloud-resource-specs.CloudResourceSpecsReconstructor (ETL\Ingestor\CloudResourceSpecsStateTransformIngestor), start_time: 1668786961.8143, end_time: 1668786968.2795, elapsed_time: 6.46521, records_examined: 0, records_loaded: 0) 2022-11-18 15:56:10 [notice] Finished processing section 'xdmod.ingest-cloud-resource-specs' 2022-11-18 15:59:16 [notice] Start processing section 'xdmod.jobs-xdw-aggregate' 2022-11-18 15:59:16 [notice] (action: xdmod.jobs-xdw-aggregate.table-create (ETL\Maintenance\ManageTables), start_time: 1668787156.1829, end_time: 1668787156.2056, elapsed_time: 0.02264) 2022-11-18 15:59:16 [notice] Altering table modw_aggregates.jobfact_by_day 2022-11-18 15:59:17 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), unit: day, start_date: none, end_date: none) 2022-11-18 15:59:18 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), unit: day, periods: 0, start_date: none, end_date: none, start_time: 1668787157.3148, end_time: 1668787158.5586, elapsed_time: 1.24378) 2022-11-18 15:59:19 [notice] end (action: xdmod.jobs-xdw-aggregate.aggregate-days (ETL\Aggregator\JobListAggregator), start_time: 1668787156.4688, end_time: 1668787159.3993, elapsed_time: 0.049m) 2022-11-18 15:59:22 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: month, start_date: none, end_date: none) 2022-11-18 15:59:24 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: month, periods: 0, start_date: none, end_date: none, start_time: 1668787162.1865, end_time: 1668787164.3445, elapsed_time: 2.15799) 2022-11-18 15:59:24 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: quarter, start_date: none, end_date: none) 2022-11-18 15:59:25 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: quarter, periods: 0, start_date: none, end_date: none, start_time: 1668787164.4854, end_time: 1668787165.5088, elapsed_time: 1.0234) 2022-11-18 15:59:25 [notice] aggregate start (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: year, start_date: none, end_date: none) 2022-11-18 15:59:25 [notice] Duplicate column after substitution: ("${aggregation_unit}: ${:PERIOD_VALUE}") '${aggregation_unit}' -> 'year' 2022-11-18 15:59:26 [notice] aggregate end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), unit: year, periods: 0, start_date: none, end_date: none, start_time: 1668787165.6477, end_time: 1668787166.0445, elapsed_time: 0.39686) 2022-11-18 15:59:27 [notice] end (action: xdmod.jobs-xdw-aggregate.aggregate (ETL\Aggregator\SimpleAggregator), start_time: 1668787162.0421, end_time: 1668787167.5638, elapsed_time: 0.092m) 2022-11-18 15:59:30 [notice] (action: xdmod.jobs-xdw-aggregate.aggregation-joblist (ETL\Ingestor\ExplodeTransformIngestor), start_time: 1668787168.5406, end_time: 1668787170.4715, elapsed_time: 1.93089, records_examined: 0, records_loaded: 0) 2022-11-18 15:59:31 [notice] Finished processing section 'xdmod.jobs-xdw-aggregate' 2022-11-18 15:59:32 [notice] start (action: Jobs.build-filter-list) 2022-11-18 15:59:32 [notice] start (action: Jobs.build-filter-list.provider) 2022-11-18 15:59:38 [notice] end (action: Jobs.build-filter-list.provider, start_time: 1668787172.9053, end_time: 1668787178.3662) 2022-11-18 15:59:40 [notice] start (action: Jobs.build-filter-list.pi) 2022-11-18 15:59:54 [notice] end (action: Jobs.build-filter-list.pi, start_time: 1668787180.6379, end_time: 1668787194.9686) 2022-11-18 15:59:56 [notice] start (action: Jobs.build-filter-list.person) 2022-11-18 16:00:05 [notice] end (action: Jobs.build-filter-list.person, start_time: 1668787196.2245, end_time: 1668787205.887) 2022-11-18 16:00:07 [notice] start (action: Jobs.build-filter-list.qos) 2022-11-18 16:00:30 [notice] end (action: Jobs.build-filter-list.qos, start_time: 1668787207.3373, end_time: 1668787230.6662) 2022-11-18 16:00:31 [notice] start (action: Jobs.build-filter-list.queue) 2022-11-18 16:00:51 [notice] end (action: Jobs.build-filter-list.queue, start_time: 1668787231.7975, end_time: 1668787251.498) 2022-11-18 16:00:52 [notice] start (action: Jobs.build-filter-list.username) 2022-11-18 16:00:59 [notice] end (action: Jobs.build-filter-list.username, start_time: 1668787252.287, end_time: 1668787259.0777) 2022-11-18 16:00:59 [notice] start (action: Jobs.build-filter-list.resource_type) 2022-11-18 16:01:06 [notice] end (action: Jobs.build-filter-list.resource_type, start_time: 1668787259.2722, end_time: 1668787266.6477) 2022-11-18 16:01:06 [notice] start (action: Jobs.build-filter-list.resource) 2022-11-18 16:01:12 [notice] end (action: Jobs.build-filter-list.resource, start_time: 1668787266.8471, end_time: 1668787272.3465) 2022-11-18 16:01:12 [notice] start (action: Jobs.build-filter-list.nsfdirectorate) 2022-11-18 16:01:15 [notice] end (action: Jobs.build-filter-list.nsfdirectorate, start_time: 1668787272.3918, end_time: 1668787275.8547) 2022-11-18 16:01:16 [notice] start (action: Jobs.build-filter-list.parentscience) 2022-11-18 16:01:23 [notice] end (action: Jobs.build-filter-list.parentscience, start_time: 1668787276.0826, end_time: 1668787283.8281) 2022-11-18 16:01:24 [notice] start (action: Jobs.build-filter-list.jobwaittime) 2022-11-18 16:01:33 [notice] end (action: Jobs.build-filter-list.jobwaittime, start_time: 1668787284.163, end_time: 1668787293.0688) 2022-11-18 16:01:33 [notice] start (action: Jobs.build-filter-list.gpucount) 2022-11-18 16:01:41 [notice] end (action: Jobs.build-filter-list.gpucount, start_time: 1668787293.1557, end_time: 1668787301.9806) 2022-11-18 16:01:42 [notice] start (action: Jobs.build-filter-list.jobsize) 2022-11-18 16:01:55 [notice] end (action: Jobs.build-filter-list.jobsize, start_time: 1668787302.9366, end_time: 1668787315.2966) 2022-11-18 16:01:55 [notice] start (action: Jobs.build-filter-list.fieldofscience) 2022-11-18 16:02:06 [notice] end (action: Jobs.build-filter-list.fieldofscience, start_time: 1668787315.453, end_time: 1668787326.6246) 2022-11-18 16:02:06 [notice] start (action: Jobs.build-filter-list.jobwalltime) 2022-11-18 16:02:12 [notice] end (action: Jobs.build-filter-list.jobwalltime, start_time: 1668787326.8634, end_time: 1668787332.474) 2022-11-18 16:02:12 [notice] start (action: Jobs.build-filter-list.nodecount) 2022-11-18 16:02:19 [notice] end (action: Jobs.build-filter-list.nodecount, start_time: 1668787332.5319, end_time: 1668787339.3167) 2022-11-18 16:02:19 [notice] end (action: Jobs.build-filter-list, start_time: 1668787172.48, end_time: 1668787339.3977) 2022-11-18 16:02:19 [notice] Aggregating Cloud data 2022-11-18 16:02:31 [notice] Start processing section 'xdmod.cloud-state-pipeline' 2022-11-18 16:02:37 [notice] (action: xdmod.cloud-state-pipeline.cloud-state-action (ETL\Ingestor\CloudStateReconstructorTransformIngestor), start_time: 1668787353.662, end_time: 1668787357.4395, elapsed_time: 3.77754, records_examined: 973, records_loaded: 973) 2022-11-18 16:02:38 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_common/delete_session_records.sql' using delimiter '//' containing 1 statements 2022-11-18 16:02:39 [notice] Finished Processing 1 SQL statements 2022-11-18 16:02:39 [notice] (action: xdmod.cloud-state-pipeline.delete-session-records (ETL\Maintenance\ExecuteSql), start_time: 1668787358.3441, end_time: 1668787359.4087, elapsed_time: 1.06458) 2022-11-18 16:02:40 [notice] Altering table modw_cloud.session_records 2022-11-18 16:02:42 [warning] SQL warnings on table 'modw_cloud.session_records' generated by action xdmod.cloud-state-pipeline.cloud-session-records 2022-11-18 16:02:42 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:42 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:43 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:43 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:44 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:44 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:44 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:44 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:45 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:46 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:48 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:48 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:48 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:49 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:50 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:50 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:51 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:55 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:55 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:56 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:02:58 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:02:58 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:00 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:00 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:02 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:02 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:02 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:03 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:03 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:03 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:03 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:04 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:04 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:04 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:06 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:08 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:08 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:08 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:08 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:10 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:13 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:14 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:15 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:15 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:16 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:16 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:18 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:19 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:20 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:20 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:21 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:22 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:22 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:25 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:28 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:30 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:31 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:32 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:32 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:32 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:33 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:33 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:34 [warning] Warning 1048 Column 'processorbucket_id' cannot be null 2022-11-18 16:03:34 [warning] Warning 1048 Column 'memorybucket_id' cannot be null 2022-11-18 16:03:38 [notice] (action: xdmod.cloud-state-pipeline.cloud-session-records (ETL\Ingestor\DatabaseIngestor), start_time: 1668787360.8646, end_time: 1668787417.5108, elapsed_time: 56.64625, records_examined: 973, records_loaded: 973) 2022-11-18 16:03:42 [notice] Altering table modw_cloud.cloudfact_by_day 2022-11-18 16:03:43 [notice] aggregate start (action: xdmod.cloud-state-pipeline.CloudEventAggregatorByDay (ETL\Aggregator\JobListAggregator), unit: day, start_date: none, end_date: none) 2022-11-18 16:03:46 [notice] aggregate end (action: xdmod.cloud-state-pipeline.CloudEventAggregatorByDay (ETL\Aggregator\JobListAggregator), unit: day, periods: 0, start_date: none, end_date: none, start_time: 1668787423.4463, end_time: 1668787426.1729, elapsed_time: 2.72664) 2022-11-18 16:03:50 [notice] end (action: xdmod.cloud-state-pipeline.CloudEventAggregatorByDay (ETL\Aggregator\JobListAggregator), start_time: 1668787420.6689, end_time: 1668787430.0355, elapsed_time: 0.156m) 2022-11-18 16:03:55 [notice] aggregate start (action: xdmod.cloud-state-pipeline.CloudEventAggregator (ETL\Aggregator\SimpleAggregator), unit: month, start_date: none, end_date: none) 2022-11-18 16:03:57 [notice] aggregate end (action: xdmod.cloud-state-pipeline.CloudEventAggregator (ETL\Aggregator\SimpleAggregator), unit: month, periods: 0, start_date: none, end_date: none, start_time: 1668787435.3619, end_time: 1668787437.7976, elapsed_time: 2.43564) 2022-11-18 16:03:57 [notice] aggregate start (action: xdmod.cloud-state-pipeline.CloudEventAggregator (ETL\Aggregator\SimpleAggregator), unit: quarter, start_date: none, end_date: none) 2022-11-18 16:04:05 [notice] aggregate end (action: xdmod.cloud-state-pipeline.CloudEventAggregator (ETL\Aggregator\SimpleAggregator), unit: quarter, periods: 0, start_date: none, end_date: none, start_time: 1668787437.9879, end_time: 1668787445.0393, elapsed_time: 7.05132) 2022-11-18 16:04:06 [notice] aggregate start (action: xdmod.cloud-state-pipeline.CloudEventAggregator (ETL\Aggregator\SimpleAggregator), unit: year, start_date: none, end_date: none) 2022-11-18 16:04:06 [notice] Duplicate column after substitution: ("${aggregation_unit}: ${:PERIOD_VALUE}") '${aggregation_unit}' -> 'year' 2022-11-18 16:04:10 [notice] aggregate end (action: xdmod.cloud-state-pipeline.CloudEventAggregator (ETL\Aggregator\SimpleAggregator), unit: year, periods: 0, start_date: none, end_date: none, start_time: 1668787446.0064, end_time: 1668787450.6547, elapsed_time: 4.6483)

rob-baron commented 10 months ago

2022-11-18 16:04:12 [notice] end (action: xdmod.cloud-state-pipeline.CloudEventAggregator (ETL\Aggregator\SimpleAggregator), start_time: 1668787435.2287, end_time: 1668787452.25, elapsed_time: 0.284m) 2022-11-18 16:04:15 [notice] (action: xdmod.cloud-state-pipeline.CloudAggregatorSessionlist (ETL\Ingestor\ExplodeTransformIngestor), start_time: 1668787454.6036, end_time: 1668787455.7311, elapsed_time: 1.12755, records_examined: 0, records_loaded: 0) 2022-11-18 16:04:18 [notice] Finished processing section 'xdmod.cloud-state-pipeline' 2022-11-18 16:04:19 [notice] Aggregating Cloud Resource Specification data 2022-11-18 16:04:27 [notice] Start processing section 'xdmod.aggregate-cloud-resource-specs' 2022-11-18 16:04:28 [notice] aggregate start (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: day, start_date: none, end_date: none) 2022-11-18 16:04:29 [notice] aggregate end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: day, periods: 0, start_date: none, end_date: none, start_time: 1668787468.8182, end_time: 1668787469.6359, elapsed_time: 0.81771) 2022-11-18 16:04:29 [notice] aggregate start (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: month, start_date: none, end_date: none) 2022-11-18 16:04:31 [notice] aggregate end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: month, periods: 0, start_date: none, end_date: none, start_time: 1668787469.7783, end_time: 1668787471.1054, elapsed_time: 1.32708) 2022-11-18 16:04:32 [notice] aggregate start (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: quarter, start_date: none, end_date: none) 2022-11-18 16:04:33 [notice] aggregate end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: quarter, periods: 0, start_date: none, end_date: none, start_time: 1668787472.3443, end_time: 1668787473.1389, elapsed_time: 0.79464) 2022-11-18 16:04:33 [notice] aggregate start (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: year, start_date: none, end_date: none) 2022-11-18 16:04:33 [notice] Duplicate column after substitution: ("${aggregation_unit}: ${:PERIOD_VALUE}") '${aggregation_unit}' -> 'year' 2022-11-18 16:04:34 [notice] aggregate end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: year, periods: 0, start_date: none, end_date: none, start_time: 1668787473.5372, end_time: 1668787474.9699, elapsed_time: 1.43276) 2022-11-18 16:04:36 [notice] end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), start_time: 1668787468.3844, end_time: 1668787476.6522, elapsed_time: 0.138m) 2022-11-18 16:04:37 [notice] Finished processing section 'xdmod.aggregate-cloud-resource-specs' 2022-11-18 16:04:37 [notice] start (action: Cloud.build-filter-list) 2022-11-18 16:04:38 [notice] start (action: Cloud.build-filter-list.vm_size_memory) 2022-11-18 16:04:46 [notice] end (action: Cloud.build-filter-list.vm_size_memory, start_time: 1668787478.1616, end_time: 1668787486.47) 2022-11-18 16:04:46 [notice] start (action: Cloud.build-filter-list.vm_size) 2022-11-18 16:04:51 [notice] end (action: Cloud.build-filter-list.vm_size, start_time: 1668787486.9611, end_time: 1668787491.2977) 2022-11-18 16:04:51 [notice] start (action: Cloud.build-filter-list.username) 2022-11-18 16:04:55 [notice] end (action: Cloud.build-filter-list.username, start_time: 1668787491.5247, end_time: 1668787495.5244) 2022-11-18 16:04:55 [notice] start (action: Cloud.build-filter-list.pi) 2022-11-18 16:05:00 [notice] end (action: Cloud.build-filter-list.pi, start_time: 1668787495.6949, end_time: 1668787500.126) 2022-11-18 16:05:02 [notice] start (action: Cloud.build-filter-list.fieldofscience) 2022-11-18 16:05:13 [notice] end (action: Cloud.build-filter-list.fieldofscience, start_time: 1668787502.2525, end_time: 1668787513.6675) 2022-11-18 16:05:14 [notice] start (action: Cloud.build-filter-list.parentscience) 2022-11-18 16:05:19 [notice] end (action: Cloud.build-filter-list.parentscience, start_time: 1668787514.0749, end_time: 1668787519.1727) 2022-11-18 16:05:19 [notice] start (action: Cloud.build-filter-list.nsfdirectorate) 2022-11-18 16:05:33 [notice] end (action: Cloud.build-filter-list.nsfdirectorate, start_time: 1668787519.2622, end_time: 1668787533.7678) 2022-11-18 16:05:33 [notice] start (action: Cloud.build-filter-list.submission_venue) 2022-11-18 16:05:41 [notice] end (action: Cloud.build-filter-list.submission_venue, start_time: 1668787533.8129, end_time: 1668787541.2554) 2022-11-18 16:05:41 [notice] start (action: Cloud.build-filter-list.provider) 2022-11-18 16:05:45 [notice] end (action: Cloud.build-filter-list.provider, start_time: 1668787541.9571, end_time: 1668787545.7098) 2022-11-18 16:05:46 [notice] start (action: Cloud.build-filter-list.domain) 2022-11-18 16:05:52 [notice] end (action: Cloud.build-filter-list.domain, start_time: 1668787546.3312, end_time: 1668787552.2485) 2022-11-18 16:05:52 [notice] start (action: Cloud.build-filter-list.project) 2022-11-18 16:05:55 [notice] end (action: Cloud.build-filter-list.project, start_time: 1668787552.4259, end_time: 1668787555.4826) 2022-11-18 16:05:55 [notice] start (action: Cloud.build-filter-list.person) 2022-11-18 16:05:57 [notice] end (action: Cloud.build-filter-list.person, start_time: 1668787555.5073, end_time: 1668787557.6217) 2022-11-18 16:05:57 [notice] start (action: Cloud.build-filter-list.resource) 2022-11-18 16:06:00 [notice] end (action: Cloud.build-filter-list.resource, start_time: 1668787557.7797, end_time: 1668787560.666) 2022-11-18 16:06:00 [notice] start (action: Cloud.build-filter-list.configuration) 2022-11-18 16:06:13 [notice] end (action: Cloud.build-filter-list.configuration, start_time: 1668787560.691, end_time: 1668787573.8202) 2022-11-18 16:06:14 [notice] end (action: Cloud.build-filter-list, start_time: 1668787477.8009, end_time: 1668787574.6387) 2022-11-18 16:06:16 [notice] Storage realm not enabled, not aggregating 2022-11-18 16:06:17 [notice] Aggregating Cloud Resource Specification data 2022-11-18 16:06:35 [notice] Start processing section 'xdmod.aggregate-cloud-resource-specs' 2022-11-18 16:06:42 [notice] aggregate start (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: day, start_date: none, end_date: none) 2022-11-18 16:06:48 [notice] aggregate end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: day, periods: 0, start_date: none, end_date: none, start_time: 1668787602.0891, end_time: 1668787608.15, elapsed_time: 6.06095) 2022-11-18 16:06:49 [notice] aggregate start (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: month, start_date: none, end_date: none) 2022-11-18 16:06:51 [notice] aggregate end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: month, periods: 0, start_date: none, end_date: none, start_time: 1668787609.1734, end_time: 1668787611.712, elapsed_time: 2.53858) 2022-11-18 16:06:53 [notice] aggregate start (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: quarter, start_date: none, end_date: none) 2022-11-18 16:06:58 [notice] aggregate end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: quarter, periods: 0, start_date: none, end_date: none, start_time: 1668787613.9553, end_time: 1668787618.4161, elapsed_time: 4.46081) 2022-11-18 16:07:00 [notice] aggregate start (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: year, start_date: none, end_date: none) 2022-11-18 16:07:01 [notice] Duplicate column after substitution: ("${aggregation_unit}: ${:PERIOD_VALUE}") '${aggregation_unit}' -> 'year' 2022-11-18 16:07:03 [notice] aggregate end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), unit: year, periods: 0, start_date: none, end_date: none, start_time: 1668787620.834, end_time: 1668787623.3351, elapsed_time: 2.50112) 2022-11-18 16:07:03 [notice] end (action: xdmod.aggregate-cloud-resource-specs.CloudResourceSpecsAggregator (ETL\Aggregator\SimpleAggregator), start_time: 1668787598.9948, end_time: 1668787623.8298, elapsed_time: 0.414m) 2022-11-18 16:07:05 [notice] Finished processing section 'xdmod.aggregate-cloud-resource-specs' 2022-11-18 16:07:08 [notice] xdmod-ingestor end (process_end_time: 2022-11-18 16:07:08) Robert Bartlett Baron , said 9 months ago No viewable data in the interface.

Any suggestions?

Robert Baron Gregary Dean , said 9 months ago Cc: msd@bu.edu, joachimw@bu.edu Ticket: https://help.xdmod.org/support/tickets/31579

Hi Robert,

Thanks for the information. Can you send me the logs you are ingesting? It looks like the data is being ingested fine but then is not being aggregated. This is why it is not showing up in the interface. I believe this is similar to the issue you had in your last ticket and it was an issue with the data in the log files so let's start there.

Thanks, Greg Robert Bartlett Baron , said 9 months ago Greg,

Sure. Attached are the log files.

Keep in mind, these are the same log files that were loaded on to a different instance of xdmod that then displayed the data in the log files.

Robert TGZ stack_events... (74.1 KB) Robert Bartlett Baron , said 9 months ago Greg,

Any ideas or suggestions?

Robert Gregary Dean , said 9 months ago Cc: msd@bu.edu, joachimw@bu.edu Ticket: https://help.xdmod.org/support/tickets/31579

Hi Robert,

Thanks for sending the logs. I shredded and ingested the logs you sent and was able to see data in my developement instance of XDMoD. Let's try a couple of other commands to see if we can get to the bottom of this.

xdmod-ingestor --aggregate=cloud --last-modified-start-date "2018-01-01 12:30:00" --debug

The above command will probably produce a lot of output so it's probably better just to pipe the output to a file and send that to me.

After you run that command run the following SQL query.

SELECT FROM_UNIXTIME(MIN(start_time_ts)), FROM_UNIXTIME(MAX(end_time_ts)) FROM modw_cloud.event_reconstructed;

Thanks, greg Robert Bartlett Baron , said 9 months ago Greg,

Have you had a chance to look over the data that I am trying to ingest?

I am happy to get on zoom or other service to share my screen so that we can work though what is happening on the production side.

We are expecting to deploy this by the end of the month, and I suspect it is something that may have failed in the xdmod-setup. The same log files were shredded and ingested in a staging environment so, I suspect they are fine.

Rob, Gregary Dean , said 9 months ago Cc: msd@bu.edu, joachimw@bu.edu Hi Rob,

Yes, the log files looked fine. I was able to ingest the in a test xdmod instance I have and view the data in the metric explorer. The log files do not seem to be the issue here. Did you see the message I sent earlier this morning that had a couple of suggestions? You mention something going wrong during xdmod-setup, where there any errors or warnings during setup? Why do you suspect something failed during xdmod-setup? There should be an exceptions log file in /var/log/xdmod. Is there any data in that fiile or any of the other files in /var/log/xdmod? Also, running the acl-config command could help in making sure the the data in the xdmod config files are been put into the database. This is done in the xdmod-setup script but if something went wrong with xdmod-setup the configs may not have been loaded in an appropriate manner. Again, if you can run the commands from the message I sent earlier today that would be helpful.

-greg Robert Bartlett Baron , said 9 months ago Greg,

I have that suspicion since I had to run the acl-config command which fixed some issues that I was having in the interface. I didn't notice any exceptions in the exceptions log, or any other errors in /var/log/xdmod.

Will be running those commands shortly.

Rob. Robert Bartlett Baron , said 9 months ago Greg,

Attached is the log of the run and here is the results of the sql:

MariaDB [(none)]> SELECT FROM_UNIXTIME(MIN(start_time_ts)), FROM_UNIXTIME(MAX(end_time_ts)) FROM modw_cloud.event_reconstructed; +-----------------------------------+---------------------------------+ | FROM_UNIXTIME(MIN(start_time_ts)) | FROM_UNIXTIME(MAX(end_time_ts)) | +-----------------------------------+---------------------------------+ | 2022-02-08 23:11:13.000000 | 2022-11-09 20:47:07.269951 | +-----------------------------------+---------------------------------+ 1 row in set (0.01 sec)

Rob. LOG xdmod-ingest... (134 KB) Gregary Dean , said 9 months ago Ticket: https://help.xdmod.org/support/tickets/31579 Hi Rob,

Thanks for sending that information. It does seem that the data is not getting into the appropriate aggregate tables but I'm unsure why. I did take another look at the logs and noticed something that may cause some issues but I don't believe it's the root cause of the issue you are experiencing. It looks like quite a fiew of the events in the log file have the value 0 for the vcpu, memory_mb and disk_gb. The vcpu field is the biggest issue since that is the field used to calculate any core hour statistics. It looks like this affects most of the files before July 1. Like I said, I don't think this is the root cause of the current issue but it might cause some confusion when looking at data in that date range.

It seems that having a zoom call is probably the way to go now. I really don't have any free time until next week. Does Tuesday morning work for you? Say 9:30AM? In the meantime, one other thing we could do is for you to send me a copy of your XDMoD database. That way if I do find some free time our meeting I look at it and see if I can why this issue is happening. The following command should get all the necessary data.

mysqldump --databases modw modw_cloud modw_aggregates > /path/to/file.sql

-greg

Robert Bartlett Baron , said 9 months ago Greg,

Sure next Tuesday morning at 9:30 sounds good. I'll get the database dumps to you.

Rob. Gregary Dean , said 8 months ago Ticket: https://help.xdmod.org/support/tickets/31579

Hi Rob,

Great!. Here's the link for the zoom meeting for tomorrow.

https://buffalo.zoom.us/j/92712398752?pwd=QWdTZHJLME90ajJDcnUwR014YUFYZz09

-greg

Robert Bartlett Baron , said 8 months ago Greg,

Got your invite, will see you tomorrow at 9:30 on zoom.

attached is the dump you requested.

Rob. SQL mysql_dump.sql (1.15 MB) This ticket has been Closed | 8 months ago Reply

CCI-MOC / xdmod-cntr

xdmod ticket #31579 Why isn't data appearing in the interface #220

1 /usr/share/xdmod/classes/ETL/EtlOverseer.php(420): ETL\LockFile->lock(Array)

2 /usr/share/xdmod/classes/ETL/Utilities.php(281): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))

3 /usr/share/xdmod/classes/OpenXdmod/DataWarehouseInitializer.php(362): ETL\Utilities::runEtlPipeline(Array, Object(CCR\Logger), Array)

4 /usr/bin/xdmod-ingestor(310): OpenXdmod\DataWarehouseInitializer->aggregateCloudData('2018-01-01 12:3...')

5 /usr/bin/xdmod-ingestor(21): main()

6 {main})

1 /usr/share/xdmod/classes/ETL/EtlOverseer.php(420): ETL\LockFile->lock(Array)

2 /usr/share/xdmod/classes/ETL/Utilities.php(281): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))

3 /usr/share/xdmod/classes/OpenXdmod/DataWarehouseInitializer.php(362): ETL\Utilities::runEtlPipeline(Array, Object(CCR\Logger), Array)

4 /usr/bin/xdmod-ingestor(310): OpenXdmod\DataWarehouseInitializer->aggregateCloudData('2018-01-01 12:3...')

5 /usr/bin/xdmod-ingestor(21): main()

6 {main})