rob-baron
commented
10 months ago sh-4.2$ xdmod-shredder --debug -f openstack -d /root/xdmod_data/kaizen -r kaizen 2022-05-31 12:21:36 [info] Command: '/usr/bin/xdmod-shredder' '--debug' '-f' 'openstack' '-d' '/root/xdmod_data/kaizen' '-r' 'kaizen' 2022-05-31 12:21:36 [debug] Checking for required arguments 2022-05-31 12:21:36 [notice] xdmod-shredder start (process_start_time: 2022-05-31 12:21:36) 2022-05-31 12:21:36 [debug] Creating database connection 2022-05-31 12:21:37 [debug] Creating shredder for format 'openstack' 2022-05-31 12:21:37 [debug] Setting resource to 'kaizen' 2022-05-31 12:21:37 [debug] Loading configuration file /etc/xdmod/resources.json 2022-05-31 12:21:37 [debug] Parsing /etc/xdmod/resources.json 2022-05-31 12:21:37 [debug] Parsed 1 records 2022-05-31 12:21:37 [debug] Stored object Configuration\XdmodConfiguration (/etc/xdmod/resources.json) in APCu cache with key Configuration\XdmodConfiguration|/etc/xdmod/resources.json|40cd750bba9870f18aada2478b24840a in 0.001135s 2022-05-31 12:21:37 [debug] Running ETL pipeline "jobs-common, ingest-organizations, ingest-resource-types, ingest-resources, jobs-cloud-common" with parameters [] 2022-05-31 12:21:37 [debug] Loading configuration file /etc/xdmod/etl/etl.json 2022-05-31 12:21:37 [debug] Parsing /etc/xdmod/etl/etl.json 2022-05-31 12:21:37 [debug] Parsed 1 records 2022-05-31 12:21:37 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-import.json 2022-05-31 12:21:37 [debug] Parsing /etc/xdmod/etl/etl.d/acls-import.json 2022-05-31 12:21:37 [debug] Parsed 1 records 2022-05-31 12:21:37 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-05-31 12:21:37 [debug] Parsing /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-05-31 12:21:37 [debug] Parsed 1 records 2022-05-31 12:21:37 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/action_state_setup.json 2022-05-31 12:21:37 [debug] Parsing /etc/xdmod/etl/etl.d/action_state_setup.json 2022-05-31 12:21:37 [debug] Parsed 1 records 2022-05-31 12:21:37 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-05-31 12:21:37 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-05-31 12:21:37 [debug] Parsed 1 records 2022-05-31 12:21:37 [info] ETL\DataEndpoint\DirectoryScanner (name=Open Stack resource specifications, path=${CLOUD_RESOURCE_SPECS_DIRECTORY}): Relative path provided, absolute path recommended 2022-05-31 12:21:38 [debug] Qualifying relative path ${CLOUD_RESOURCE_SPECS_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/gateways.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/gateways.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/ingest_resources.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/ingest_resources.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/jobs.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-05-31 12:21:38 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [info] ETL\DataEndpoint\DirectoryScanner (name=Generic cloud event logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-05-31 12:21:38 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-05-31 12:21:38 [info] ETL\DataEndpoint\DirectoryScanner (name=Generic volume logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-05-31 12:21:38 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [info] ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-05-31 12:21:38 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_common.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_common.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/organizations.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/organizations.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/resource_types.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/resource_types.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/shredder.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/shredder.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/staging.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/staging.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [info] ETL\DataEndpoint\DirectoryScanner (name=usage-directory, path=${STORAGE_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-05-31 12:21:38 [debug] Qualifying relative path ${STORAGE_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/storage.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/storage.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/test_suite.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/test_suite.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/verify.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/verify.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdb.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/xdb.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-05-31 12:21:38 [debug] Parsing /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-05-31 12:21:38 [debug] Parsed 1 records 2022-05-31 12:21:38 [debug] Stored object ETL\Configuration\EtlConfiguration (/etc/xdmod/etl/etl.json) in APCu cache with key ETL\Configuration\EtlConfiguration|/etc/xdmod/etl/etl.json|cb328615c5b43bfbe1404ea2dbb0a7fc in 0.169982s 2022-05-31 12:21:38 [debug] Running ETL pipeline with script options {"default-module-name":"xdmod","process-sections":["jobs-common","ingest-organizations","ingest-resource-types","ingest-resources","jobs-cloud-common"]} 2022-05-31 12:21:38 [info] Verifying endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:41 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Countable types, path=/etc/xdmod/etl/etl_data.d/jobs/countable_type.json) 2022-05-31 12:21:41 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Job record types, path=/etc/xdmod/etl/etl_data.d/jobs/job_record_type.json) 2022-05-31 12:21:41 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Job task types, path=/etc/xdmod/etl/etl_data.d/jobs/job_task_type.json) 2022-05-31 12:21:41 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Units of measure, path=/etc/xdmod/etl/etl_data.d/jobs/unit.json) 2022-05-31 12:21:41 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Submission venues, path=/etc/xdmod/etl/etl_data.d/jobs/submission_venue.json) 2022-05-31 12:21:41 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Organization configuration, path=/etc/xdmod/etl/../organization.json, 1 filters) 2022-05-31 12:21:41 [info] Verifying endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:44 [info] Verifying endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:45 [info] Verifying endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:45 [info] Verifying endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:45 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Resource types, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-05-31 12:21:45 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Resources configuration, path=/etc/xdmod/etl/../resources.json) 2022-05-31 12:21:45 [info] Verifying endpoint: ETL\DataEndpoint\ConfigurationFileEndpoint (name=Resource Types configuration, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-05-31 12:21:45 [info] Verifying endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:45 [info] Verifying endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:46 [notice] Schema 'modw_cloud' does not exist for user 'xdmod', creating 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud asset types, path=/etc/xdmod/etl/etl_data.d/cloud_common/asset_type.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud event types, path=/etc/xdmod/etl/etl_data.d/cloud_common/event_type.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/record_type.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/host.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/account.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance_type.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/image.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=memory buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/memory_buckets.json) 2022-05-31 12:21:46 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=processor buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/processor_buckets.json) 2022-05-31 12:21:46 [info] Create action xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables) 2022-05-31 12:21:46 [info] Verifying action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables) 2022-05-31 12:21:46 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:46 [info] Source endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:46 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:46 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json' 2022-05-31 12:21:46 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json 2022-05-31 12:21:46 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json 2022-05-31 12:21:46 [debug] Parsed 1 records 2022-05-31 12:21:46 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json|40cd750bba9870f18aada2478b24840a in 0.002842s 2022-05-31 12:21:46 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json' 2022-05-31 12:21:46 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json 2022-05-31 12:21:46 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json 2022-05-31 12:21:46 [debug] Parsed 1 records 2022-05-31 12:21:46 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json|40cd750bba9870f18aada2478b24840a in 0.002342s 2022-05-31 12:21:46 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json' 2022-05-31 12:21:46 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/meta_job.json 2022-05-31 12:21:46 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/meta_job.json 2022-05-31 12:21:46 [debug] Parsed 1 records 2022-05-31 12:21:46 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json|40cd750bba9870f18aada2478b24840a in 0.001925s 2022-05-31 12:21:46 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json' 2022-05-31 12:21:46 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json 2022-05-31 12:21:46 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json 2022-05-31 12:21:46 [debug] Parsed 1 records 2022-05-31 12:21:46 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json|40cd750bba9870f18aada2478b24840a in 0.002075s 2022-05-31 12:21:46 [info] Create action xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:46 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json 2022-05-31 12:21:46 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/countable_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/countable_type.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003675s 2022-05-31 12:21:47 [info] Verifying action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Countable types, path=/etc/xdmod/etl/etl_data.d/jobs/countable_type.json) 2022-05-31 12:21:47 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [debug] Created ETL destination table object for table definition key 'countable_type' 2022-05-31 12:21:47 [info] Create action xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/job_record_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/job_record_type.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.004058s 2022-05-31 12:21:47 [info] Verifying action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Job record types, path=/etc/xdmod/etl/etl_data.d/jobs/job_record_type.json) 2022-05-31 12:21:47 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [debug] Created ETL destination table object for table definition key 'job_record_type' 2022-05-31 12:21:47 [info] Create action xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/job_task_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/job_task_type.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003743s 2022-05-31 12:21:47 [info] Verifying action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Job task types, path=/etc/xdmod/etl/etl_data.d/jobs/job_task_type.json) 2022-05-31 12:21:47 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [debug] Created ETL destination table object for table definition key 'job_task_type' 2022-05-31 12:21:47 [info] Create action xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/unit.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/unit.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/unit.json' to '/etc/xdmod/etl/etl_tables.d/jobs/unit.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/unit.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/unit.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.004094s 2022-05-31 12:21:47 [info] Verifying action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Units of measure, path=/etc/xdmod/etl/etl_data.d/jobs/unit.json) 2022-05-31 12:21:47 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [debug] Created ETL destination table object for table definition key 'unit' 2022-05-31 12:21:47 [info] Create action xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/submission_venue.json' to '/etc/xdmod/etl/etl_tables.d/jobs/submission_venue.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003820s 2022-05-31 12:21:47 [info] Verifying action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Submission venues, path=/etc/xdmod/etl/etl_data.d/jobs/submission_venue.json) 2022-05-31 12:21:47 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [debug] Created ETL destination table object for table definition key 'submission_venue' 2022-05-31 12:21:47 [info] Create action xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/organization.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/organization.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002896s 2022-05-31 12:21:47 [info] Verifying action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Organization configuration, path=/etc/xdmod/etl/../organization.json, 1 filters) 2022-05-31 12:21:47 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [debug] Created ETL destination table object for table definition key 'staging_organization' 2022-05-31 12:21:47 [info] Create action xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/organizations.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/organizations.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002904s 2022-05-31 12:21:47 [info] Verifying action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [debug] Created ETL destination table object for table definition key 'hpcdb_organizations' 2022-05-31 12:21:47 [debug] Create ETL source query object 2022-05-31 12:21:47 [debug] No destination_field_map specified 2022-05-31 12:21:47 [debug] Auto-generating destination_field_map using 3 source fields: organization_id, organization_name, organization_abbrev 2022-05-31 12:21:47 [debug] Available fields for table key 'hpcdb_organizations': organization_id, organization_abbrev, organization_name 2022-05-31 12:21:47 [debug] Generated destination_field_map: Table: hpcdb_organizations organization_id -> organization_id organization_abbrev -> organization_abbrev organization_name -> organization_name 2022-05-31 12:21:47 [info] Create action xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/organization.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/organization.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002653s 2022-05-31 12:21:47 [info] Verifying action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [debug] Created ETL destination table object for table definition key 'organization' 2022-05-31 12:21:47 [debug] Create ETL source query object 2022-05-31 12:21:47 [debug] No destination_field_map specified 2022-05-31 12:21:47 [debug] Auto-generating destination_field_map using 7 source fields: id, abbrev, name, short_name, long_name, order_id, organization_origin_id 2022-05-31 12:21:47 [debug] Available fields for table key 'organization': id, organizationtype_id, abbrev, name, url, phone, nsf_org_code, is_reconciled, amie_name, country_id, state_id, latitude, longitude, short_name, long_name, federation_instance_id, organization_origin_id, order_id 2022-05-31 12:21:47 [debug] Generated destination_field_map: Table: organization id -> id abbrev -> abbrev name -> name short_name -> short_name long_name -> long_name organization_origin_id -> organization_origin_id order_id -> order_id 2022-05-31 12:21:47 [info] Create action xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables) 2022-05-31 12:21:47 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json|40cd750bba9870f18aada2478b24840a in 0.001301s 2022-05-31 12:21:47 [info] Create action xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql) 2022-05-31 12:21:47 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:47 [info] Create action xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json 2022-05-31 12:21:47 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json 2022-05-31 12:21:47 [debug] Parsed 1 records 2022-05-31 12:21:47 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' 2022-05-31 12:21:47 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002733s 2022-05-31 12:21:47 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:47 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Resource types, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-05-31 12:21:48 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'staging_resource_type' 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-types.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-types.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002913s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'hpcdb_resource_types' 2022-05-31 12:21:48 [debug] Create ETL source query object 2022-05-31 12:21:48 [debug] No destination_field_map specified 2022-05-31 12:21:48 [debug] Auto-generating destination_field_map using 3 source fields: type_id, type_abbr, type_desc 2022-05-31 12:21:48 [debug] Available fields for table key 'hpcdb_resource_types': type_id, type_abbr, type_desc 2022-05-31 12:21:48 [debug] Generated destination_field_map: Table: hpcdb_resource_types type_id -> type_id type_abbr -> type_abbr type_desc -> type_desc 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-type.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002805s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'resourcetype' 2022-05-31 12:21:48 [debug] Create ETL source query object 2022-05-31 12:21:48 [debug] No destination_field_map specified 2022-05-31 12:21:48 [debug] Auto-generating destination_field_map using 3 source fields: id, description, abbrev 2022-05-31 12:21:48 [debug] Available fields for table key 'resourcetype': id, description, abbrev 2022-05-31 12:21:48 [debug] Generated destination_field_map: Table: resourcetype id -> id description -> description abbrev -> abbrev 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-config.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-config.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003015s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Resources configuration, path=/etc/xdmod/etl/../resources.json) 2022-05-31 12:21:48 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'staging_resource_config' 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002773s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'staging_resource' 2022-05-31 12:21:48 [debug] Create ETL source query object 2022-05-31 12:21:48 [debug] No destination_field_map specified 2022-05-31 12:21:48 [debug] Auto-generating destination_field_map using 1 source fields: resource_name 2022-05-31 12:21:48 [debug] Available fields for table key 'staging_resource': resource_id, resource_name 2022-05-31 12:21:48 [debug] Generated destination_field_map: Table: staging_resource resource_name -> resource_name 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resources.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resources.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003684s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'hpcdb_resources' 2022-05-31 12:21:48 [debug] Create ETL source query object 2022-05-31 12:21:48 [debug] No destination_field_map specified 2022-05-31 12:21:48 [debug] Auto-generating destination_field_map using 8 source fields: resource_id, resource_type_id, organization_id, resource_name, resource_code, resource_description, resource_shared_jobs, resource_timezone 2022-05-31 12:21:48 [debug] Available fields for table key 'hpcdb_resources': resource_id, resource_type_id, organization_id, resource_name, resource_code, resource_description, resource_shared_jobs, resource_timezone 2022-05-31 12:21:48 [debug] Generated destination_field_map: Table: hpcdb_resources resource_id -> resource_id resource_type_id -> resource_type_id organization_id -> organization_id resource_name -> resource_name resource_code -> resource_code resource_description -> resource_description resource_shared_jobs -> resource_shared_jobs resource_timezone -> resource_timezone 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-fact.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-fact.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003761s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'resourcefact' 2022-05-31 12:21:48 [debug] Create ETL source query object 2022-05-31 12:21:48 [debug] No destination_field_map specified 2022-05-31 12:21:48 [debug] Auto-generating destination_field_map using 9 source fields: id, resourcetype_id, organization_id, name, code, description, shared_jobs, timezone, resource_origin_id 2022-05-31 12:21:48 [debug] Available fields for table key 'resourcefact': id, resourcetype_id, organization_id, name, code, description, start_date, start_date_ts, end_date, end_date_ts, shared_jobs, timezone, resource_origin_id 2022-05-31 12:21:48 [debug] Generated destination_field_map: Table: resourcefact id -> id resourcetype_id -> resourcetype_id organization_id -> organization_id name -> name code -> code description -> description shared_jobs -> shared_jobs timezone -> timezone resource_origin_id -> resource_origin_id 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type-realms.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type-realms.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003315s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ETL\DataEndpoint\ConfigurationFileEndpoint (name=Resource Types configuration, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-05-31 12:21:48 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'staging_resource_type_realms' 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/acls/realms.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/acls/realms.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/acls/realms.json' to '/etc/xdmod/etl/etl_tables.d/acls/realms.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/acls/realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/acls/realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002954s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Destination endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'realms' 2022-05-31 12:21:48 [debug] Create ETL source query object 2022-05-31 12:21:48 [debug] No destination_field_map specified 2022-05-31 12:21:48 [debug] Auto-generating destination_field_map using 3 source fields: display, name, module_id 2022-05-31 12:21:48 [debug] Available fields for table key 'realms': realm_id, module_id, name, display 2022-05-31 12:21:48 [debug] Generated destination_field_map: Table: realms module_id -> module_id name -> name display -> display 2022-05-31 12:21:48 [info] Create action xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-type-realms.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-type-realms.json' 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003091s 2022-05-31 12:21:48 [info] Verifying action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Destination endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [debug] Created ETL destination table object for table definition key 'resource_type_realms' 2022-05-31 12:21:48 [debug] Create ETL source query object 2022-05-31 12:21:48 [debug] No destination_field_map specified 2022-05-31 12:21:48 [debug] Auto-generating destination_field_map using 2 source fields: resource_type_id, realm_id 2022-05-31 12:21:48 [debug] Available fields for table key 'resource_type_realms': resource_type_realm_id, resource_type_id, realm_id 2022-05-31 12:21:48 [debug] Generated destination_field_map: Table: resource_type_realms resource_type_id -> resource_type_id realm_id -> realm_id 2022-05-31 12:21:48 [info] Create action xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables) 2022-05-31 12:21:48 [info] Verifying action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables) 2022-05-31 12:21:48 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Source endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-05-31 12:21:48 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/account.json' 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/account.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/account.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/account.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/account.json|40cd750bba9870f18aada2478b24840a in 0.002772s 2022-05-31 12:21:48 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/event.json' 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/event.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/event.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/event.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/event.json|40cd750bba9870f18aada2478b24840a in 0.003039s 2022-05-31 12:21:48 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json' 2022-05-31 12:21:48 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/asset.json 2022-05-31 12:21:48 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/asset.json 2022-05-31 12:21:48 [debug] Parsed 1 records 2022-05-31 12:21:48 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json|40cd750bba9870f18aada2478b24840a in 0.002830s 2022-05-31 12:21:48 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/host.json' 2022-05-31 12:21:49 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/host.json 2022-05-31 12:21:49 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/host.json 2022-05-31 12:21:49 [debug] Parsed 1 records 2022-05-31 12:21:49 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/host.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/host.json|40cd750bba9870f18aada2478b24840a in 0.002284s 2022-05-31 12:21:49 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json' 2022-05-31 12:21:49 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json 2022-05-31 12:21:49 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json 2022-05-31 12:21:49 [debug] Parsed 1 records 2022-05-31 12:21:49 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json|40cd750bba9870f18aada2478b24840a in 0.002399s 2022-05-31 12:21:49 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json' 2022-05-31 12:21:49 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json 2022-05-31 12:21:49 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json 2022-05-31 12:21:49 [debug] Parsed 1 records 2022-05-31 12:21:49 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json|40cd750bba9870f18aada2478b24840a in 0.002438s 2022-05-31 12:21:49 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json' 2022-05-31 12:21:49 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json 2022-05-31 12:21:49 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json 2022-05-31 12:21:49 [debug] Parsed 1 records 2022-05-31 12:21:49 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json|40cd750bba9870f18aada2478b24840a in 0.002313s 2022-05-31 12:21:49 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json' 2022-05-31 12:21:49 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json 2022-05-31 12:21:49 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json 2022-05-31 12:21:49 [debug] Parsed 1 records 2022-05-31 12:21:49 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json|40cd750bba9870f18aada2478b24840a in 0.002268s 2022-05-31 12:21:49 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json' 2022-05-31 12:21:49 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json 2022-05-31 12:21:49 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json 2022-05-31 12:21:49 [debug] Parsed 1 records 2022-05-31 12:21:49 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json|40cd750bba9870f18aada2478b24840a in 0.002674s 2022-05-31 12:21:49 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json' 2022-05-31 12:21:49 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json 2022-05-31 12:21:49 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json 2022-05-31 12:21:49 [debug] Parsed 1 records 2022-05-31 12:21:49 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json|40cd750bba9870f18aada2478b24840a in 0.003672s 2022-05-31 12:21:49 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json' 2022-05-31 12:21:49 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json 2022-05-31 12:21:49 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json 2022-05-31 12:21:49 [debug] Parsed 1 records
Start with my questions:
1) I am trying to work with a minimal set of events - is this the right approach?
2) Any suggestions as to how to modify the data I am trying to shred and ingest in order to view it in the interface - or am I looking in the wrong spot in the interface?
3) My connection to mariadb is dropping occasionally (today more than normal), how large of an issue is this for shredding/ingesting this dataset?
4) This dataset is small, what should the times be for shredding/ingesting the data? Although my runtimes are all over the place, these take longer than I would expect.
Here is a snippet of data that I am collecting from openstack (file: /root/xdmod_data/kaizen/2022-05-23T00:00:00_2022-05-24T00:00:00.json) :
[ { "audit_period_beginning": "2022-05-23T14:11:34.000000", "audit_period_ending": "2022-05-23T17:40:27.055883", "disk_gb": "25", "domain": "default", "ephemeral_gb": "0", "event_type": "compute.instance.exists", "generated": "2022-05-23T17:40:27.055883", "host": "neu-5-36-stackcomp.kzn.moc", "instance_id": "7358a34e-f4c3-4861-805f-029f23123fe5", "instance_type": "m1.s2.xlarge", "instance_type_id": "9", "launched_at": "2022-05-23T14:11:34.000000", "memory_mb": "16384", "project_id": "f12f928576ae4d21bdb984da5dd1d3bf", "project_name": "ShiftStack", "raw": {}, "resource_id": "7358a34e-f4c3-4861-805f-029f23123fe5", "root_gb": "25", "service": "compute", "state": "ACTIVE", "tenant_id": "f12f928576ae4d21bdb984da5dd1d3bf", "user_id": "41c3b836577a411db0e49a3fb946ceff", "user_name": "unknown user", "vcpus": "8" }, { "audit_period_beginning": "2022-05-23T14:10:52.000000", "audit_period_ending": "2022-05-23T17:40:27.055883", "disk_gb": "25", "domain": "default", "ephemeral_gb": "0", "event_type": "compute.instance.exists", "generated": "2022-05-23T17:40:27.055883", "host": "neu-17-3-stackcomp.kzn.moc", "instance_id": "a717eeea-91ba-489b-ac1b-396e63542906", "instance_type": "m1.s2.xlarge", "instance_type_id": "9", "launched_at": "2022-05-23T14:10:52.000000", "memory_mb": "16384", "project_id": "f12f928576ae4d21bdb984da5dd1d3bf", "project_name": "ShiftStack", "raw": {}, "resource_id": "a717eeea-91ba-489b-ac1b-396e63542906", "root_gb": "25", "service": "compute", "state": "ACTIVE", "tenant_id": "f12f928576ae4d21bdb984da5dd1d3bf", "user_id": "41c3b836577a411db0e49a3fb946ceff", "user_name": "unknown user", "vcpus": "8" }, { "disk_gb": "25", "domain": "default", "ephemeral_gb": "0", "event_type": "compute.instance.create.start", "generated": "2022-05-23T14:11:07", "host": "neu-5-36-stackcomp.kzn.moc", "instance_id": "7358a34e-f4c3-4861-805f-029f23123fe5", "instance_type": "m1.s2.xlarge", "instance_type_id": "9", "launched_at": "2022-05-23T14:11:34.000000", "memory_mb": "16384", "project_id": "f12f928576ae4d21bdb984da5dd1d3bf", "project_name": "ShiftStack", "raw": {}, "resource_id": "7358a34e-f4c3-4861-805f-029f23123fe5", "root_gb": "25", "service": "compute", "state": "ACTIVE", "tenant_id": "f12f928576ae4d21bdb984da5dd1d3bf", "user_id": "41c3b836577a411db0e49a3fb946ceff", "user_name": "unknown user", "vcpus": "8" }, { "disk_gb": "25", "domain": "default", "ephemeral_gb": "0", "event_type": "compute.instance.create.end", "generated": "2022-05-23T14:11:17", "host": "neu-5-36-stackcomp.kzn.moc", "instance_id": "7358a34e-f4c3-4861-805f-029f23123fe5", "instance_type": "m1.s2.xlarge", "instance_type_id": "9", "launched_at": "2022-05-23T14:11:34.000000", "memory_mb": "16384", "project_id": "f12f928576ae4d21bdb984da5dd1d3bf", "project_name": "ShiftStack", "raw": {}, "resource_id": "7358a34e-f4c3-4861-805f-029f23123fe5", "root_gb": "25", "service": "compute", "state": "ACTIVE", "tenant_id": "f12f928576ae4d21bdb984da5dd1d3bf", "user_id": "41c3b836577a411db0e49a3fb946ceff", "user_name": "unknown user", "vcpus": "8" } ]
The resources.json file is as follows:
[ { "resource": "xdmodtest", "resource_type": "Cloud", "name": "XDMod Test Data" }, { "resource": "kaizen", "resource_type": "Cloud", "name": "MOC Kaizen" }, { "resource": "nerc-openstack", "resource_type": "Cloud", "name": "NERC OpenStack" } ]
Here are the shredding runs (first one stopped as the connection to the mariadb had been dropped - not sure why):