search

CCI-MOC / xdmod-cntr

A project to prototype the use of XDMOD with OpenStack and OpenShift on the MOC
1 stars 5 forks source link

xdmod ticket #29597 Data in the interface #207

Open rob-baron opened 1 year ago

rob-baron commented 1 year ago

Robert Bartlett Baron, reported over 1 year ago Ok, I have shredded and ingested from the reference xdmod-openstack file (https://github.com/ubccr/xdmod/tree/xdmod10.0/tests/artifacts/xdmod/referencedata). I brought up the xdmod interface, it is running without complaining that there is no data, however, it doesn't display any data for any time period.

which database values need to match with values in the file?

Gregary Dean , said over 1 year ago Ticket: https://help.xdmod.org/support/tickets/29597

Hi Robert,

There are a couple of things to note. The first is when running xdmod-shredder to make sure you specify the correct resource using the -r flag. It should be the resource you previously added. The second is that most of the reference data is from April 2018 so you have to look quite a ways back to see it. You said it didn't show for any time range so I'm assuming you did look back to at least April 2018 but just wanted to make sure.

As for more troubleshooting, can you check the following tables to see if they have data in them?

Do you still have the commands you ran when shredding and ingesting the data? Can you send those to me?

Thanks, Greg Robert Bartlett Baron , said over 1 year ago OK, shredding doesn't work:

xdmod-shredder -f openstack -d openstack -r Cloud 2022-04-04 15:28:33 [notice] xdmod-shredder start (process_start_time: 2022-04-04 15:28:33) 2022-04-04 15:28:37 [notice] Start processing section 'xdmod.jobs-common' 2022-04-04 15:28:37 [notice] (action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables), start_time: 1649086117.1748, end_time: 1649086117.2365, elapsed_time: 0.06171) 2022-04-04 15:28:37 [notice] (action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086117.3197, end_time: 1649086117.5459, elapsed_time: 0.22615, records_examined: 5, records_loaded: 5) 2022-04-04 15:28:37 [notice] (action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086117.6095, end_time: 1649086117.7845, elapsed_time: 0.17498, records_examined: 5, records_loaded: 5) 2022-04-04 15:28:38 [notice] (action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086117.8456, end_time: 1649086118.0374, elapsed_time: 0.19177, records_examined: 5, records_loaded: 5) 2022-04-04 15:28:38 [notice] (action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086118.1047, end_time: 1649086118.2985, elapsed_time: 0.1938, records_examined: 6, records_loaded: 6) 2022-04-04 15:28:38 [notice] (action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086118.357, end_time: 1649086118.5486, elapsed_time: 0.19155, records_examined: 4, records_loaded: 4) 2022-04-04 15:28:38 [notice] Finished processing section 'xdmod.jobs-common' 2022-04-04 15:28:38 [notice] Start processing section 'xdmod.ingest-organizations' 2022-04-04 15:28:38 [notice] (action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086118.6612, end_time: 1649086118.7848, elapsed_time: 0.12365, records_examined: 1, records_loaded: 1) 2022-04-04 15:28:38 [notice] (action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor), start_time: 1649086118.8448, end_time: 1649086118.9192, elapsed_time: 0.07448, records_examined: 0, records_loaded: 0) 2022-04-04 15:28:39 [notice] (action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_time: 1649086118.9853, end_time: 1649086119.2964, elapsed_time: 0.31109, records_examined: 1, records_loaded: 1) 2022-04-04 15:28:39 [notice] Finished processing section 'xdmod.ingest-organizations' 2022-04-04 15:28:39 [notice] Start processing section 'xdmod.ingest-resource-types' 2022-04-04 15:28:39 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables), start_time: 1649086119.4154, end_time: 1649086119.4365, elapsed_time: 0.02118) 2022-04-04 15:28:39 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-04-04 15:28:39 [notice] Finished Processing 2 SQL statements 2022-04-04 15:28:39 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1649086119.4833, end_time: 1649086119.5873, elapsed_time: 0.10402) 2022-04-04 15:28:39 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086119.642, end_time: 1649086119.7539, elapsed_time: 0.1119, records_examined: 12, records_loaded: 12) 2022-04-04 15:28:39 [notice] (action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor), start_time: 1649086119.8167, end_time: 1649086119.8894, elapsed_time: 0.07274, records_examined: 0, records_loaded: 0) 2022-04-04 15:28:40 [notice] (action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_time: 1649086119.9549, end_time: 1649086120.0291, elapsed_time: 0.07413, records_examined: 0, records_loaded: 0) 2022-04-04 15:28:40 [notice] Finished processing section 'xdmod.ingest-resource-types' 2022-04-04 15:28:40 [notice] Start processing section 'xdmod.ingest-resources' 2022-04-04 15:28:40 [notice] (action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086120.1264, end_time: 1649086120.2017, elapsed_time: 0.07532, records_examined: 1, records_loaded: 1) 2022-04-04 15:28:40 [notice] (action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_time: 1649086120.2743, end_time: 1649086120.3394, elapsed_time: 0.06509, records_examined: 0, records_loaded: 0) 2022-04-04 15:28:40 [notice] (action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor), start_time: 1649086120.4107, end_time: 1649086120.4879, elapsed_time: 0.07726, records_examined: 0, records_loaded: 0) 2022-04-04 15:28:40 [notice] (action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor), start_time: 1649086120.5515, end_time: 1649086120.6279, elapsed_time: 0.07639, records_examined: 0, records_loaded: 0) 2022-04-04 15:28:40 [notice] (action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086120.686, end_time: 1649086120.8814, elapsed_time: 0.19542, records_examined: 9, records_loaded: 9) 2022-04-04 15:28:41 [notice] (action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor), start_time: 1649086120.94, end_time: 1649086121.0115, elapsed_time: 0.07158, records_examined: 0, records_loaded: 0) 2022-04-04 15:28:41 [notice] (action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor), start_time: 1649086121.07, end_time: 1649086121.1917, elapsed_time: 0.12173, records_examined: 27, records_loaded: 27) 2022-04-04 15:28:41 [notice] Finished processing section 'xdmod.ingest-resources' 2022-04-04 15:28:41 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-04-04 15:28:41 [notice] Altering table modw_cloud.event 2022-04-04 15:28:41 [notice] Altering table modw_cloud.cloud_resource_specs 2022-04-04 15:28:41 [notice] Altering table modw_cloud.session_records 2022-04-04 15:28:41 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1649086121.2821, end_time: 1649086121.63, elapsed_time: 0.3479) 2022-04-04 15:28:41 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086121.6763, end_time: 1649086121.7445, elapsed_time: 0.06816, records_examined: 5, records_loaded: 5) 2022-04-04 15:28:41 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086121.814, end_time: 1649086121.9008, elapsed_time: 0.08671, records_examined: 65, records_loaded: 65) 2022-04-04 15:28:42 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086121.9631, end_time: 1649086122.0369, elapsed_time: 0.07377, records_examined: 4, records_loaded: 4) 2022-04-04 15:28:42 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086122.0919, end_time: 1649086122.161, elapsed_time: 0.06909, records_examined: 1, records_loaded: 1) 2022-04-04 15:28:42 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086122.2255, end_time: 1649086122.2948, elapsed_time: 0.06931, records_examined: 1, records_loaded: 1) 2022-04-04 15:28:42 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086122.3606, end_time: 1649086122.433, elapsed_time: 0.07241, records_examined: 1, records_loaded: 1) 2022-04-04 15:28:42 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086122.4981, end_time: 1649086122.5726, elapsed_time: 0.07456, records_examined: 1, records_loaded: 1) 2022-04-04 15:28:42 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086122.6424, end_time: 1649086122.7112, elapsed_time: 0.0688, records_examined: 1, records_loaded: 1) 2022-04-04 15:28:42 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086122.7708, end_time: 1649086122.8422, elapsed_time: 0.07134, records_examined: 8, records_loaded: 8) 2022-04-04 15:28:42 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1649086122.9023, end_time: 1649086122.9743, elapsed_time: 0.07199, records_examined: 7, records_loaded: 7) 2022-04-04 15:28:43 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-04-04 15:28:43 [error] {"message":"ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=\/etc\/xdmod\/etl\/etl_data.d\/openstack, handler=jsonfile): Path '\/etc\/xdmod\/etl\/etl_data.d\/openstack' is not a directory"} 2022-04-04 15:28:43 [critical] Failed to shred files: ETL\EtlOverseer: Error verifying data endpoints: ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=/etc/xdmod/etl/etl_data.d/openstack, handler=jsonfile): Path '/etc/xdmod/etl/etl_data.d/openstack' is not a directory (stacktrace: #0 /usr/share/xdmod/classes/ETL/EtlOverseer.php(379): ETL\EtlOverseer->verifyDataEndpoints(Object(ETL\Configuration\EtlConfiguration), true)

1 /usr/share/xdmod/classes/ETL/Utilities.php(281): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))

2 /usr/share/xdmod/classes/OpenXdmod/Shredder/aCloud.php(62): ETL\Utilities::runEtlPipeline(Array, Object(CCR\Logger), Array)

3 /usr/bin/xdmod-shredder(203): OpenXdmod\Shredder\aCloud->shredDirectory('openstack')

4 /usr/bin/xdmod-shredder(22): main()

5 {main})

Robert Bartlett Baron , said over 1 year ago changed the name of the directory from /root/openstack, to /root/cloud_openstack - here is the run:

sh-4.2$ xdmod-shredder -f openstack -d cloud_openstack -r Cloud 2022-04-04 17:13:23 [notice] xdmod-shredder start (process_start_time: 2022-04-04 17:13:23) 2022-04-04 17:13:27 [notice] Start processing section 'xdmod.jobs-common' 2022-04-04 17:13:27 [notice] (action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables), start_time: 1649092407.2439, end_time: 1649092407.307, elapsed_time: 0.06304) 2022-04-04 17:13:27 [notice] (action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092407.3536, end_time: 1649092407.6372, elapsed_time: 0.28358, records_examined: 5, records_loaded: 5) 2022-04-04 17:13:27 [notice] (action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092407.6965, end_time: 1649092407.8585, elapsed_time: 0.16199, records_examined: 5, records_loaded: 5) 2022-04-04 17:13:28 [notice] (action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092407.9115, end_time: 1649092408.0922, elapsed_time: 0.18066, records_examined: 5, records_loaded: 5) 2022-04-04 17:13:28 [notice] (action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092408.1544, end_time: 1649092408.3414, elapsed_time: 0.18699, records_examined: 6, records_loaded: 6) 2022-04-04 17:13:28 [notice] (action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092408.3883, end_time: 1649092408.5506, elapsed_time: 0.16234, records_examined: 4, records_loaded: 4) 2022-04-04 17:13:28 [notice] Finished processing section 'xdmod.jobs-common' 2022-04-04 17:13:28 [notice] Start processing section 'xdmod.ingest-organizations' 2022-04-04 17:13:28 [notice] (action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092408.6311, end_time: 1649092408.7429, elapsed_time: 0.11187, records_examined: 1, records_loaded: 1) 2022-04-04 17:13:28 [notice] (action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor), start_time: 1649092408.7994, end_time: 1649092408.8635, elapsed_time: 0.06416, records_examined: 0, records_loaded: 0) 2022-04-04 17:13:29 [notice] (action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_time: 1649092408.9238, end_time: 1649092409.1854, elapsed_time: 0.26152, records_examined: 1, records_loaded: 1) 2022-04-04 17:13:29 [notice] Finished processing section 'xdmod.ingest-organizations' 2022-04-04 17:13:29 [notice] Start processing section 'xdmod.ingest-resource-types' 2022-04-04 17:13:29 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables), start_time: 1649092409.2683, end_time: 1649092409.2879, elapsed_time: 0.01958) 2022-04-04 17:13:29 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-04-04 17:13:29 [notice] Finished Processing 2 SQL statements 2022-04-04 17:13:29 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1649092409.3319, end_time: 1649092409.4244, elapsed_time: 0.09256) 2022-04-04 17:13:29 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092409.4753, end_time: 1649092409.597, elapsed_time: 0.12163, records_examined: 12, records_loaded: 12) 2022-04-04 17:13:29 [notice] (action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor), start_time: 1649092409.6571, end_time: 1649092409.7285, elapsed_time: 0.0714, records_examined: 0, records_loaded: 0) 2022-04-04 17:13:29 [notice] (action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_time: 1649092409.7816, end_time: 1649092409.8637, elapsed_time: 0.08215, records_examined: 0, records_loaded: 0) 2022-04-04 17:13:29 [notice] Finished processing section 'xdmod.ingest-resource-types' 2022-04-04 17:13:29 [notice] Start processing section 'xdmod.ingest-resources' 2022-04-04 17:13:30 [notice] (action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092409.9488, end_time: 1649092410.0148, elapsed_time: 0.06599, records_examined: 1, records_loaded: 1) 2022-04-04 17:13:30 [notice] (action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_time: 1649092410.071, end_time: 1649092410.1378, elapsed_time: 0.06683, records_examined: 0, records_loaded: 0) 2022-04-04 17:13:30 [notice] (action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor), start_time: 1649092410.1983, end_time: 1649092410.2662, elapsed_time: 0.06792, records_examined: 0, records_loaded: 0) 2022-04-04 17:13:30 [notice] (action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor), start_time: 1649092410.3276, end_time: 1649092410.3966, elapsed_time: 0.06891, records_examined: 0, records_loaded: 0) 2022-04-04 17:13:30 [notice] (action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092410.4553, end_time: 1649092410.6518, elapsed_time: 0.1965, records_examined: 9, records_loaded: 9) 2022-04-04 17:13:30 [notice] (action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor), start_time: 1649092410.709, end_time: 1649092410.7802, elapsed_time: 0.07125, records_examined: 0, records_loaded: 0) 2022-04-04 17:13:30 [notice] (action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor), start_time: 1649092410.8375, end_time: 1649092410.9567, elapsed_time: 0.1192, records_examined: 36, records_loaded: 36) 2022-04-04 17:13:30 [notice] Finished processing section 'xdmod.ingest-resources' 2022-04-04 17:13:31 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-04-04 17:13:31 [notice] Altering table modw_cloud.event 2022-04-04 17:13:31 [notice] Altering table modw_cloud.cloud_resource_specs 2022-04-04 17:13:31 [notice] Altering table modw_cloud.session_records 2022-04-04 17:13:31 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1649092411.0442, end_time: 1649092411.3994, elapsed_time: 0.35515) 2022-04-04 17:13:31 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092411.4399, end_time: 1649092411.5055, elapsed_time: 0.0656, records_examined: 5, records_loaded: 5) 2022-04-04 17:13:31 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092411.5575, end_time: 1649092411.627, elapsed_time: 0.06954, records_examined: 65, records_loaded: 65) 2022-04-04 17:13:31 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092411.6811, end_time: 1649092411.7462, elapsed_time: 0.06506, records_examined: 4, records_loaded: 4) 2022-04-04 17:13:31 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092411.7999, end_time: 1649092411.8589, elapsed_time: 0.05905, records_examined: 1, records_loaded: 1) 2022-04-04 17:13:31 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092411.9089, end_time: 1649092411.9705, elapsed_time: 0.06162, records_examined: 1, records_loaded: 1) 2022-04-04 17:13:32 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092412.0246, end_time: 1649092412.0809, elapsed_time: 0.05632, records_examined: 1, records_loaded: 1) 2022-04-04 17:13:32 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092412.1312, end_time: 1649092412.1936, elapsed_time: 0.0624, records_examined: 1, records_loaded: 1) 2022-04-04 17:13:32 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092412.2441, end_time: 1649092412.3015, elapsed_time: 0.05738, records_examined: 1, records_loaded: 1) 2022-04-04 17:13:32 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092412.3538, end_time: 1649092412.413, elapsed_time: 0.05912, records_examined: 8, records_loaded: 8) 2022-04-04 17:13:32 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092412.4663, end_time: 1649092412.5268, elapsed_time: 0.06053, records_examined: 7, records_loaded: 7) 2022-04-04 17:13:32 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-04-04 17:13:32 [notice] Start processing section 'xdmod.jobs-cloud-ingest-openstack' 2022-04-04 17:13:33 [notice] (action: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649092412.9487, end_time: 1649092413.0117, elapsed_time: 0.06305, records_examined: 0, records_loaded: 0) 2022-04-04 17:13:33 [notice] Finished processing section 'xdmod.jobs-cloud-ingest-openstack' 2022-04-04 17:13:33 [notice] Done shredding files 2022-04-04 17:13:33 [notice] xdmod-shredder end (process_end_time: 2022-04-04 17:13:33)

Even though this completes, there still is no data in the tables you mentioned. Gregary Dean , said over 1 year ago Ticket: https://help.xdmod.org/support/tickets/29597

Hi Robert,

It would seem that the shredder is not finding any files to ingest data from. When using the -d flag you should specify the full path to where your openstack log files are. Instead of 'cloud_openstack', use '/root/cloudopenstack'. Using the --debug flag will also give some more output that might help track down issues. Also, the log file names should match the format %Y-%m-%dT%H:%i:%s%Y-%m-%dT%H:%i:%s.json. The file in this directory on github is an example of the filename format, https://github.com/ubccr/xdmod/tree/xdmod10.5/tests/artifacts/xdmod/referencedata/openstack.

-greg

rob-baron commented 1 year ago

So I ran the following:

$ xdmod-shredder --debug -f openstack -d /root/test/openstack -r Cloud 2022-04-05 15:07:55 [info] Command: '/usr/bin/xdmod-shredder' '--debug' '-f' 'openstack' '-d' '/root/test/openstack' '-r' 'Cloud' 2022-04-05 15:07:55 [debug] Checking for required arguments 2022-04-05 15:07:55 [notice] xdmod-shredder start (process_start_time: 2022-04-05 15:07:55) 2022-04-05 15:07:55 [debug] Creating database connection 2022-04-05 15:07:55 [debug] Creating shredder for format 'openstack' 2022-04-05 15:07:55 [debug] Setting resource to 'Cloud' 2022-04-05 15:07:55 [debug] Loading configuration file /etc/xdmod/resources.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/resources.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Stored object Configuration\XdmodConfiguration (/etc/xdmod/resources.json) in APCu cache with key Configuration\XdmodConfiguration|/etc/xdmod/resources.json|40cd750bba9870f18aada2478b24840a in 0.001026s 2022-04-05 15:07:55 [debug] Running ETL pipeline "jobs-common, ingest-organizations, ingest-resource-types, ingest-resources, jobs-cloud-common" with parameters [] 2022-04-05 15:07:55 [debug] Loading configuration file /etc/xdmod/etl/etl.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-import.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/acls-import.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/action_state_setup.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/action_state_setup.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [info] ETL\DataEndpoint\DirectoryScanner (name=Open Stack resource specifications, path=${CLOUD_RESOURCE_SPECS_DIRECTORY}): Relative path provided, absolute path recommended 2022-04-05 15:07:55 [debug] Qualifying relative path ${CLOUD_RESOURCE_SPECS_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/gateways.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/gateways.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/ingest_resources.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/ingest_resources.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/jobs.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-04-05 15:07:55 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [info] ETL\DataEndpoint\DirectoryScanner (name=Generic cloud event logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-04-05 15:07:55 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-04-05 15:07:55 [info] ETL\DataEndpoint\DirectoryScanner (name=Generic volume logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-04-05 15:07:55 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [info] ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=${CLOUD_EVENT_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-04-05 15:07:55 [debug] Qualifying relative path ${CLOUD_EVENT_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_common.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_common.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/organizations.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/organizations.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/resource_types.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/resource_types.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/shredder.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/shredder.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/staging.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/staging.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [info] ETL\DataEndpoint\DirectoryScanner (name=usage-directory, path=${STORAGE_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-04-05 15:07:55 [debug] Qualifying relative path ${STORAGE_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/storage.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/storage.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/test_suite.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/test_suite.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/verify.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/verify.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdb.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/xdb.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Stored object ETL\Configuration\EtlConfiguration (/etc/xdmod/etl/etl.json) in APCu cache with key ETL\Configuration\EtlConfiguration|/etc/xdmod/etl/etl.json|cb328615c5b43bfbe1404ea2dbb0a7fc in 0.192002s 2022-04-05 15:07:55 [debug] Running ETL pipeline with script options {"default-module-name":"xdmod","process-sections":["jobs-common","ingest-organizations","ingest-resource-types","ingest-resources","jobs-cloud-common"]} 2022-04-05 15:07:55 [info] Verifying endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Countable types, path=/etc/xdmod/etl/etl_data.d/jobs/countable_type.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Job record types, path=/etc/xdmod/etl/etl_data.d/jobs/job_record_type.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Job task types, path=/etc/xdmod/etl/etl_data.d/jobs/job_task_type.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Units of measure, path=/etc/xdmod/etl/etl_data.d/jobs/unit.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Submission venues, path=/etc/xdmod/etl/etl_data.d/jobs/submission_venue.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Organization configuration, path=/etc/xdmod/etl/../organization.json, 1 filters) 2022-04-05 15:07:55 [info] Verifying endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Verifying endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Verifying endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Verifying endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Resource types, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Resources configuration, path=/etc/xdmod/etl/../resources.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\ConfigurationFileEndpoint (name=Resource Types configuration, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-04-05 15:07:55 [info] Verifying endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Verifying endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud asset types, path=/etc/xdmod/etl/etl_data.d/cloud_common/asset_type.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud event types, path=/etc/xdmod/etl/etl_data.d/cloud_common/event_type.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/record_type.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/host.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/account.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance_type.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/image.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=memory buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/memory_buckets.json) 2022-04-05 15:07:55 [info] Verifying endpoint: ETL\DataEndpoint\JsonFile (name=processor buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/processor_buckets.json) 2022-04-05 15:07:55 [info] Create action xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables) 2022-04-05 15:07:55 [info] Verifying action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables) 2022-04-05 15:07:55 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Source endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json' 2022-04-05 15:07:55 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/job_request_info.json|40cd750bba9870f18aada2478b24840a in 0.001313s 2022-04-05 15:07:55 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json' 2022-04-05 15:07:55 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/job_task_countable.json|40cd750bba9870f18aada2478b24840a in 0.000948s 2022-04-05 15:07:55 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json' 2022-04-05 15:07:55 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/meta_job.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/meta_job.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/meta_job.json|40cd750bba9870f18aada2478b24840a in 0.001114s 2022-04-05 15:07:55 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json' 2022-04-05 15:07:55 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/jobs/meta_job_record.json|40cd750bba9870f18aada2478b24840a in 0.001104s 2022-04-05 15:07:55 [info] Create action xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:55 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/countable_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/countable_type.json' 2022-04-05 15:07:55 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/countable_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001668s 2022-04-05 15:07:55 [info] Verifying action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:55 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Countable types, path=/etc/xdmod/etl/etl_data.d/jobs/countable_type.json) 2022-04-05 15:07:55 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:55 [debug] Created ETL destination table object for table definition key 'countable_type' 2022-04-05 15:07:55 [info] Create action xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:55 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json 2022-04-05 15:07:55 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json 2022-04-05 15:07:55 [debug] Parsed 1 records 2022-04-05 15:07:55 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/job_record_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/job_record_type.json' 2022-04-05 15:07:55 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/job_record_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001625s 2022-04-05 15:07:55 [info] Verifying action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:55 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Job record types, path=/etc/xdmod/etl/etl_data.d/jobs/job_record_type.json) 2022-04-05 15:07:56 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'job_record_type' 2022-04-05 15:07:56 [info] Create action xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/job_task_type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/job_task_type.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/job_task_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001730s 2022-04-05 15:07:56 [info] Verifying action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Job task types, path=/etc/xdmod/etl/etl_data.d/jobs/job_task_type.json) 2022-04-05 15:07:56 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'job_task_type' 2022-04-05 15:07:56 [info] Create action xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/unit.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/unit.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/unit.json' to '/etc/xdmod/etl/etl_tables.d/jobs/unit.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/unit.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/unit.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001695s 2022-04-05 15:07:56 [info] Verifying action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Units of measure, path=/etc/xdmod/etl/etl_data.d/jobs/unit.json) 2022-04-05 15:07:56 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'unit' 2022-04-05 15:07:56 [info] Create action xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/submission_venue.json' to '/etc/xdmod/etl/etl_tables.d/jobs/submission_venue.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/submission_venue.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001797s 2022-04-05 15:07:56 [info] Verifying action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Submission venues, path=/etc/xdmod/etl/etl_data.d/jobs/submission_venue.json) 2022-04-05 15:07:56 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'submission_venue' 2022-04-05 15:07:56 [info] Create action xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/organization.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/organization.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/organization.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001814s 2022-04-05 15:07:56 [info] Verifying action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Organization configuration, path=/etc/xdmod/etl/../organization.json, 1 filters) 2022-04-05 15:07:56 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'staging_organization' 2022-04-05 15:07:56 [info] Create action xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/organizations.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/organizations.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/organizations.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001792s 2022-04-05 15:07:56 [info] Verifying action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'hpcdb_organizations' 2022-04-05 15:07:56 [debug] Create ETL source query object 2022-04-05 15:07:56 [debug] No destination_field_map specified 2022-04-05 15:07:56 [debug] Auto-generating destination_field_map using 3 source fields: organization_id, organization_name, organization_abbrev 2022-04-05 15:07:56 [debug] Available fields for table key 'hpcdb_organizations': organization_id, organization_abbrev, organization_name 2022-04-05 15:07:56 [debug] Generated destination_field_map: Table: hpcdb_organizations organization_id -> organization_id organization_abbrev -> organization_abbrev organization_name -> organization_name 2022-04-05 15:07:56 [info] Create action xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/organization.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/organization.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/organization.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001392s 2022-04-05 15:07:56 [info] Verifying action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'organization' 2022-04-05 15:07:56 [debug] Create ETL source query object 2022-04-05 15:07:56 [debug] No destination_field_map specified 2022-04-05 15:07:56 [debug] Auto-generating destination_field_map using 7 source fields: id, abbrev, name, short_name, long_name, order_id, organization_origin_id 2022-04-05 15:07:56 [debug] Available fields for table key 'organization': id, organizationtype_id, abbrev, name, url, phone, nsf_org_code, is_reconciled, amie_name, country_id, state_id, latitude, longitude, short_name, long_name, federation_instance_id, organization_origin_id, order_id 2022-04-05 15:07:56 [debug] Generated destination_field_map: Table: organization id -> id abbrev -> abbrev name -> name short_name -> short_name long_name -> long_name organization_origin_id -> organization_origin_id order_id -> order_id 2022-04-05 15:07:56 [info] Create action xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables) 2022-04-05 15:07:56 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json|40cd750bba9870f18aada2478b24840a in 0.001301s 2022-04-05 15:07:56 [info] Create action xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql) 2022-04-05 15:07:56 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Create action xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001690s 2022-04-05 15:07:56 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Resource types, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-04-05 15:07:56 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'staging_resource_type' 2022-04-05 15:07:56 [info] Create action xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-types.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-types.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-types.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001898s 2022-04-05 15:07:56 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [debug] Created ETL destination table object for table definition key 'hpcdb_resource_types' 2022-04-05 15:07:56 [debug] Create ETL source query object 2022-04-05 15:07:56 [debug] No destination_field_map specified 2022-04-05 15:07:56 [debug] Auto-generating destination_field_map using 3 source fields: type_id, type_abbr, type_desc 2022-04-05 15:07:56 [debug] Available fields for table key 'hpcdb_resource_types': type_id, type_abbr, type_desc 2022-04-05 15:07:56 [debug] Generated destination_field_map: Table: hpcdb_resource_types type_id -> type_id type_abbr -> type_abbr type_desc -> type_desc 2022-04-05 15:07:56 [info] Create action xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:56 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json 2022-04-05 15:07:56 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json 2022-04-05 15:07:56 [debug] Parsed 1 records 2022-04-05 15:07:56 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-type.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-type.json' 2022-04-05 15:07:56 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001805s 2022-04-05 15:07:56 [info] Verifying action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:56 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:56 [info] Source endpoint: ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [debug] Created ETL destination table object for table definition key 'resourcetype' 2022-04-05 15:07:57 [debug] Create ETL source query object 2022-04-05 15:07:57 [debug] No destination_field_map specified 2022-04-05 15:07:57 [debug] Auto-generating destination_field_map using 3 source fields: id, description, abbrev 2022-04-05 15:07:57 [debug] Available fields for table key 'resourcetype': id, description, abbrev 2022-04-05 15:07:57 [debug] Generated destination_field_map: Table: resourcetype id -> id description -> description abbrev -> abbrev 2022-04-05 15:07:57 [info] Create action xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-config.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-config.json' 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-config.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001875s 2022-04-05 15:07:57 [info] Verifying action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Resources configuration, path=/etc/xdmod/etl/../resources.json) 2022-04-05 15:07:57 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [debug] Created ETL destination table object for table definition key 'staging_resource_config' 2022-04-05 15:07:57 [info] Create action xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource.json' 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001827s 2022-04-05 15:07:57 [info] Verifying action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [debug] Created ETL destination table object for table definition key 'staging_resource' 2022-04-05 15:07:57 [debug] Create ETL source query object 2022-04-05 15:07:57 [debug] No destination_field_map specified 2022-04-05 15:07:57 [debug] Auto-generating destination_field_map using 1 source fields: resource_name 2022-04-05 15:07:57 [debug] Available fields for table key 'staging_resource': resource_id, resource_name 2022-04-05 15:07:57 [debug] Generated destination_field_map: Table: staging_resource resource_name -> resource_name 2022-04-05 15:07:57 [info] Create action xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resources.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resources.json' 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resources.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002431s 2022-04-05 15:07:57 [info] Verifying action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Destination endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [debug] Created ETL destination table object for table definition key 'hpcdb_resources' 2022-04-05 15:07:57 [debug] Create ETL source query object 2022-04-05 15:07:57 [debug] No destination_field_map specified 2022-04-05 15:07:57 [debug] Auto-generating destination_field_map using 8 source fields: resource_id, resource_type_id, organization_id, resource_name, resource_code, resource_description, resource_shared_jobs, resource_timezone 2022-04-05 15:07:57 [debug] Available fields for table key 'hpcdb_resources': resource_id, resource_type_id, organization_id, resource_name, resource_code, resource_description, resource_shared_jobs, resource_timezone 2022-04-05 15:07:57 [debug] Generated destination_field_map: Table: hpcdb_resources resource_id -> resource_id resource_type_id -> resource_type_id organization_id -> organization_id resource_name -> resource_name resource_code -> resource_code resource_description -> resource_description resource_shared_jobs -> resource_shared_jobs resource_timezone -> resource_timezone 2022-04-05 15:07:57 [info] Create action xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-fact.json' to '/etc/xdmod/etl/etl_tables.d/jobs/xdw/resource-fact.json' 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/jobs/xdw/resource-fact.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002444s 2022-04-05 15:07:57 [info] Verifying action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Source endpoint: ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Destination endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [debug] Created ETL destination table object for table definition key 'resourcefact' 2022-04-05 15:07:57 [debug] Create ETL source query object 2022-04-05 15:07:57 [debug] No destination_field_map specified 2022-04-05 15:07:57 [debug] Auto-generating destination_field_map using 9 source fields: id, resourcetype_id, organization_id, name, code, description, shared_jobs, timezone, resource_origin_id 2022-04-05 15:07:57 [debug] Available fields for table key 'resourcefact': id, resourcetype_id, organization_id, name, code, description, start_date, start_date_ts, end_date, end_date_ts, shared_jobs, timezone, resource_origin_id 2022-04-05 15:07:57 [debug] Generated destination_field_map: Table: resourcefact id -> id resourcetype_id -> resourcetype_id organization_id -> organization_id name -> name code -> code description -> description shared_jobs -> shared_jobs timezone -> timezone resource_origin_id -> resource_origin_id 2022-04-05 15:07:57 [info] Create action xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type-realms.json' to '/etc/xdmod/etl/etl_tables.d/common/staging/resource-type-realms.json' 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/staging/resource-type-realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001531s 2022-04-05 15:07:57 [info] Verifying action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Source endpoint: ETL\DataEndpoint\ConfigurationFileEndpoint (name=Resource Types configuration, path=/etc/xdmod/etl/../resource_types.json, 1 filters) 2022-04-05 15:07:57 [info] Destination endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [debug] Created ETL destination table object for table definition key 'staging_resource_type_realms' 2022-04-05 15:07:57 [info] Create action xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/acls/realms.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/acls/realms.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/acls/realms.json' to '/etc/xdmod/etl/etl_tables.d/acls/realms.json' 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/acls/realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/acls/realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001947s 2022-04-05 15:07:57 [info] Verifying action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Destination endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [debug] Created ETL destination table object for table definition key 'realms' 2022-04-05 15:07:57 [debug] Create ETL source query object 2022-04-05 15:07:57 [debug] No destination_field_map specified 2022-04-05 15:07:57 [debug] Auto-generating destination_field_map using 3 source fields: display, name, module_id 2022-04-05 15:07:57 [debug] Available fields for table key 'realms': realm_id, module_id, name, display 2022-04-05 15:07:57 [debug] Generated destination_field_map: Table: realms module_id -> module_id name -> name display -> display 2022-04-05 15:07:57 [info] Create action xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-type-realms.json' to '/etc/xdmod/etl/etl_tables.d/common/hpcdb/resource-type-realms.json' 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/common/hpcdb/resource-type-realms.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001848s 2022-04-05 15:07:57 [info] Verifying action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor) 2022-04-05 15:07:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Source endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Destination endpoint: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [debug] Created ETL destination table object for table definition key 'resource_type_realms' 2022-04-05 15:07:57 [debug] Create ETL source query object 2022-04-05 15:07:57 [debug] No destination_field_map specified 2022-04-05 15:07:57 [debug] Auto-generating destination_field_map using 2 source fields: resource_type_id, realm_id 2022-04-05 15:07:57 [debug] Available fields for table key 'resource_type_realms': resource_type_realm_id, resource_type_id, realm_id 2022-04-05 15:07:57 [debug] Generated destination_field_map: Table: resource_type_realms resource_type_id -> resource_type_id realm_id -> realm_id 2022-04-05 15:07:57 [info] Create action xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables) 2022-04-05 15:07:57 [info] Verifying action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables) 2022-04-05 15:07:57 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Source endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/account.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/account.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/account.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/account.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/account.json|40cd750bba9870f18aada2478b24840a in 0.001477s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/event.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/event.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/event.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/event.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/event.json|40cd750bba9870f18aada2478b24840a in 0.001769s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/asset.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/asset.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/asset.json|40cd750bba9870f18aada2478b24840a in 0.001581s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/host.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/host.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/host.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/host.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/host.json|40cd750bba9870f18aada2478b24840a in 0.001155s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/instance_data.json|40cd750bba9870f18aada2478b24840a in 0.001259s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/event_asset.json|40cd750bba9870f18aada2478b24840a in 0.001180s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/raw_resource_specs.json|40cd750bba9870f18aada2478b24840a in 0.001186s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/staging_resource_specifications.json|40cd750bba9870f18aada2478b24840a in 0.001187s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/cloud_resource_specs.json|40cd750bba9870f18aada2478b24840a in 0.001341s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json|40cd750bba9870f18aada2478b24840a in 0.001966s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json|40cd750bba9870f18aada2478b24840a in 0.001336s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_volume.json|40cd750bba9870f18aada2478b24840a in 0.001542s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_event.json|40cd750bba9870f18aada2478b24840a in 0.001518s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_instance_type.json|40cd750bba9870f18aada2478b24840a in 0.001303s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json' 2022-04-05 15:07:57 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json 2022-04-05 15:07:57 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json 2022-04-05 15:07:57 [debug] Parsed 1 records 2022-04-05 15:07:57 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_generic/raw_volume.json|40cd750bba9870f18aada2478b24840a in 0.001558s 2022-04-05 15:07:57 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/domains.json' 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/domains.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/domains.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/domains.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/domains.json|40cd750bba9870f18aada2478b24840a in 0.001291s 2022-04-05 15:07:58 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json' 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue_staging.json|40cd750bba9870f18aada2478b24840a in 0.001075s 2022-04-05 15:07:58 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json' 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_openstack/domain_submission_venue.json|40cd750bba9870f18aada2478b24840a in 0.001151s 2022-04-05 15:07:58 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json' 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/staging_cloud_project_to_pi.json|40cd750bba9870f18aada2478b24840a in 0.001074s 2022-04-05 15:07:58 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json' 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/session_records.json|40cd750bba9870f18aada2478b24840a in 0.002067s 2022-04-05 15:07:58 [info] Parse table definition: '/etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json' 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_tables.d/cloud_common/cloudfact_by_day_sessionlist.json|40cd750bba9870f18aada2478b24840a in 0.001118s 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/asset_type.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/asset_type.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/asset_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/asset_type.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/asset_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/asset_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001660s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud asset types, path=/etc/xdmod/etl/etl_data.d/cloud_common/asset_type.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'asset_type' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/event_type.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/event_type.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/event_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/event_type.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/event_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/event_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001600s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud event types, path=/etc/xdmod/etl/etl_data.d/cloud_common/event_type.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'event_type' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/record_type.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/record_type.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/record_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/record_type.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/record_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/record_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001620s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/record_type.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'record_type' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/host.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/host.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/host.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/host.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/host.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/host.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001794s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/host.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'host' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/account.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/account.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/account.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/account.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/account.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/account.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001803s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/account.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'account' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/instance.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/instance.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/instance.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/instance.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/instance.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/instance.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001794s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'instance' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/instance_type.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/instance_type.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/instance_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/instance_type.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/instance_type.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/instance_type.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.002060s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/instance_type.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'instance_type' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/image.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/image.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/image.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/image.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/image.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/image.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001694s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=Cloud record types, path=/etc/xdmod/etl/etl_data.d/cloud_common/image.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'image' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/memory_buckets.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/memory_buckets.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/memory_buckets.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/memory_buckets.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/memory_buckets.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/memory_buckets.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001656s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=memory buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/memory_buckets.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'memory_buckets' 2022-04-05 15:07:58 [info] Create action xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_common/processor_buckets.json 2022-04-05 15:07:58 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_common/processor_buckets.json 2022-04-05 15:07:58 [debug] Parsed 1 records 2022-04-05 15:07:58 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_common/processor_buckets.json' to '/etc/xdmod/etl/etl_tables.d/cloud_common/processor_buckets.json' 2022-04-05 15:07:58 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_common/processor_buckets.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_common/processor_buckets.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.001717s 2022-04-05 15:07:58 [info] Verifying action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:07:58 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [info] Source endpoint: ETL\DataEndpoint\JsonFile (name=processor buckets data, path=/etc/xdmod/etl/etl_data.d/cloud_common/processor_buckets.json) 2022-04-05 15:07:58 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:58 [debug] Created ETL destination table object for table definition key 'processor_buckets' 2022-04-05 15:07:58 [info] Empty lock directory specified, using temp directory: /tmp 2022-04-05 15:07:58 [info] Obtaining lock file '/tmp/etlv2_34' 2022-04-05 15:07:58 [notice] Start processing section 'xdmod.jobs-common' 2022-04-05 15:07:58 [info] start (action_name: xdmod.jobs-common.JobsTableManagement, action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables), start_date: , end_date: ) 2022-04-05 15:07:59 [debug] Discover table 'modw.job_request_info' 2022-04-05 15:07:59 [debug] Discover table 'modw.job_task_countable' 2022-04-05 15:07:59 [debug] Discover table 'modw.meta_job' 2022-04-05 15:07:59 [debug] Discover table 'modw.meta_job_record' 2022-04-05 15:07:59 [notice] (action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables), start_time: 1649171279.0125, end_time: 1649171279.0699, elapsed_time: 0.05741) 2022-04-05 15:07:59 [info] end (action_name: xdmod.jobs-common.JobsTableManagement, action: xdmod.jobs-common.JobsTableManagement (ETL\Maintenance\ManageTables)) 2022-04-05 15:07:59 [info] start (action_name: xdmod.jobs-common.CountableTypeIngestor, action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:07:59 [info] Truncate destination table: modw.countable_type 2022-04-05 15:07:59 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.countable_type 2022-04-05 15:07:59 [debug] Discover table 'modw.countable_type' 2022-04-05 15:07:59 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:07:59 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:07:59 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/countable_type.json 2022-04-05 15:07:59 [debug] Parsed 6 records 2022-04-05 15:07:59 [debug] Requested 5 record fields: countable_type_id, unit_id, countable_type, display, description 2022-04-05 15:07:59 [debug] No destination_field_map specified 2022-04-05 15:07:59 [debug] Auto-generating destination_field_map using 5 source fields: countable_type_id, unit_id, countable_type, display, description 2022-04-05 15:07:59 [debug] Available fields for table key 'countable_type': countable_type_id, unit_id, countable_type, display, description 2022-04-05 15:07:59 [debug] Generated destination_field_map: Table: countable_type countable_type_id -> countable_type_id unit_id -> unit_id countable_type -> countable_type display -> display description -> description 2022-04-05 15:07:59 [debug] Processing destination field map 2022-04-05 15:07:59 [debug] Insert SQL for table key 'countable_type': INSERT INTO modw.countable_type (countable_type_id, unit_id, countable_type, display, description) VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE countable_type_id = COALESCE(VALUES(countable_type_id), countable_type_id), unit_id = COALESCE(VALUES(unit_id), unit_id), countable_type = COALESCE(VALUES(countable_type), countable_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-04-05 15:07:59 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:59 [debug] ANALYZE TABLE modw.countable_type 2022-04-05 15:07:59 [debug] Completed in 0.019950s 2022-04-05 15:07:59 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 5 records (Time Taken: 0.18 s) 2022-04-05 15:07:59 [notice] (action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171279.1188, end_time: 1649171279.2965, elapsed_time: 0.17762, records_examined: 5, records_loaded: 5) 2022-04-05 15:07:59 [info] end (action_name: xdmod.jobs-common.CountableTypeIngestor, action: xdmod.jobs-common.CountableTypeIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:07:59 [info] start (action_name: xdmod.jobs-common.JobRecordTypeIngestor, action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:07:59 [info] Truncate destination table: modw.job_record_type 2022-04-05 15:07:59 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.job_record_type 2022-04-05 15:07:59 [debug] Discover table 'modw.job_record_type' 2022-04-05 15:07:59 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:07:59 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:07:59 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/job_record_type.json 2022-04-05 15:07:59 [debug] Parsed 6 records 2022-04-05 15:07:59 [debug] Requested 4 record fields: job_record_type_id, job_record_type, display, description 2022-04-05 15:07:59 [debug] No destination_field_map specified 2022-04-05 15:07:59 [debug] Auto-generating destination_field_map using 4 source fields: job_record_type_id, job_record_type, display, description 2022-04-05 15:07:59 [debug] Available fields for table key 'job_record_type': job_record_type_id, job_record_type, display, description 2022-04-05 15:07:59 [debug] Generated destination_field_map: Table: job_record_type job_record_type_id -> job_record_type_id job_record_type -> job_record_type display -> display description -> description 2022-04-05 15:07:59 [debug] Processing destination field map 2022-04-05 15:07:59 [debug] Insert SQL for table key 'job_record_type': INSERT INTO modw.job_record_type (job_record_type_id, job_record_type, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE job_record_type_id = COALESCE(VALUES(job_record_type_id), job_record_type_id), job_record_type = COALESCE(VALUES(job_record_type), job_record_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-04-05 15:07:59 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:59 [debug] ANALYZE TABLE modw.job_record_type 2022-04-05 15:07:59 [debug] Completed in 0.015670s 2022-04-05 15:07:59 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 5 records (Time Taken: 0.19 s) 2022-04-05 15:07:59 [notice] (action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171279.3573, end_time: 1649171279.5451, elapsed_time: 0.18774, records_examined: 5, records_loaded: 5) 2022-04-05 15:07:59 [info] end (action_name: xdmod.jobs-common.JobRecordTypeIngestor, action: xdmod.jobs-common.JobRecordTypeIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:07:59 [info] start (action_name: xdmod.jobs-common.JobTaskTypeIngestor, action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:07:59 [info] Truncate destination table: modw.job_task_type 2022-04-05 15:07:59 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.job_task_type 2022-04-05 15:07:59 [debug] Discover table 'modw.job_task_type' 2022-04-05 15:07:59 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:07:59 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:07:59 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/job_task_type.json 2022-04-05 15:07:59 [debug] Parsed 6 records 2022-04-05 15:07:59 [debug] Requested 4 record fields: job_task_type_id, job_task_type, display, description 2022-04-05 15:07:59 [debug] No destination_field_map specified 2022-04-05 15:07:59 [debug] Auto-generating destination_field_map using 4 source fields: job_task_type_id, job_task_type, display, description 2022-04-05 15:07:59 [debug] Available fields for table key 'job_task_type': job_task_type_id, job_record_type_id, job_task_type, display, description 2022-04-05 15:07:59 [debug] Generated destination_field_map: Table: job_task_type job_task_type_id -> job_task_type_id job_task_type -> job_task_type display -> display description -> description 2022-04-05 15:07:59 [debug] Processing destination field map 2022-04-05 15:07:59 [debug] Insert SQL for table key 'job_task_type': INSERT INTO modw.job_task_type (job_task_type_id, job_task_type, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE job_task_type_id = COALESCE(VALUES(job_task_type_id), job_task_type_id), job_task_type = COALESCE(VALUES(job_task_type), job_task_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-04-05 15:07:59 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:07:59 [debug] ANALYZE TABLE modw.job_task_type 2022-04-05 15:07:59 [debug] Completed in 0.014600s 2022-04-05 15:07:59 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 5 records (Time Taken: 0.19 s) 2022-04-05 15:07:59 [notice] (action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171279.6047, end_time: 1649171279.7926, elapsed_time: 0.18781, records_examined: 5, records_loaded: 5) 2022-04-05 15:07:59 [info] end (action_name: xdmod.jobs-common.JobTaskTypeIngestor, action: xdmod.jobs-common.JobTaskTypeIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:07:59 [info] start (action_name: xdmod.jobs-common.UnitIngestor, action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:07:59 [info] Truncate destination table: modw.unit 2022-04-05 15:07:59 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.unit 2022-04-05 15:07:59 [debug] Discover table 'modw.unit' 2022-04-05 15:07:59 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:07:59 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:07:59 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/unit.json 2022-04-05 15:07:59 [debug] Parsed 7 records 2022-04-05 15:07:59 [debug] Requested 4 record fields: unit_id, unit, display, description 2022-04-05 15:07:59 [debug] No destination_field_map specified 2022-04-05 15:07:59 [debug] Auto-generating destination_field_map using 4 source fields: unit_id, unit, display, description 2022-04-05 15:07:59 [debug] Available fields for table key 'unit': unit_id, unit, display, description 2022-04-05 15:07:59 [debug] Generated destination_field_map: Table: unit unit_id -> unit_id unit -> unit display -> display description -> description 2022-04-05 15:07:59 [debug] Processing destination field map 2022-04-05 15:07:59 [debug] Insert SQL for table key 'unit': INSERT INTO modw.unit (unit_id, unit, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE unit_id = COALESCE(VALUES(unit_id), unit_id), unit = COALESCE(VALUES(unit), unit), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-04-05 15:08:00 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:00 [debug] ANALYZE TABLE modw.unit 2022-04-05 15:08:00 [debug] Completed in 0.013460s 2022-04-05 15:08:00 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 6 records (Time Taken: 0.18 s) 2022-04-05 15:08:00 [notice] (action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171279.8575, end_time: 1649171280.04, elapsed_time: 0.18251, records_examined: 6, records_loaded: 6) 2022-04-05 15:08:00 [info] end (action_name: xdmod.jobs-common.UnitIngestor, action: xdmod.jobs-common.UnitIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:00 [info] start (action_name: xdmod.jobs-common.SubmissionVenueIngestor, action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:00 [info] Truncate destination table: modw.submission_venue 2022-04-05 15:08:00 [debug] Truncate destination task ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): TRUNCATE TABLE modw.submission_venue 2022-04-05 15:08:00 [debug] Discover table 'modw.submission_venue' 2022-04-05 15:08:00 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:00 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:00 [debug] Parsing /etc/xdmod/etl/etl_data.d/jobs/submission_venue.json 2022-04-05 15:08:00 [debug] Parsed 5 records 2022-04-05 15:08:00 [debug] Requested 5 record fields: submission_venue_id, submission_venue, display, description, order_id 2022-04-05 15:08:00 [debug] No destination_field_map specified 2022-04-05 15:08:00 [debug] Auto-generating destination_field_map using 5 source fields: submission_venue_id, submission_venue, display, description, order_id 2022-04-05 15:08:00 [debug] Available fields for table key 'submission_venue': submission_venue_id, submission_venue, display, description, order_id 2022-04-05 15:08:00 [debug] Generated destination_field_map: Table: submission_venue submission_venue_id -> submission_venue_id submission_venue -> submission_venue display -> display description -> description order_id -> order_id 2022-04-05 15:08:00 [debug] Processing destination field map 2022-04-05 15:08:00 [debug] Insert SQL for table key 'submission_venue': INSERT INTO modw.submission_venue (submission_venue_id, submission_venue, display, description, order_id) VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE submission_venue_id = COALESCE(VALUES(submission_venue_id), submission_venue_id), submission_venue = COALESCE(VALUES(submission_venue), submission_venue), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description), order_id = COALESCE(VALUES(order_id), order_id) 2022-04-05 15:08:00 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:00 [debug] ANALYZE TABLE modw.submission_venue 2022-04-05 15:08:00 [debug] Completed in 0.013610s 2022-04-05 15:08:00 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 4 records (Time Taken: 0.17 s) 2022-04-05 15:08:00 [notice] (action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171280.0963, end_time: 1649171280.2653, elapsed_time: 0.16891, records_examined: 4, records_loaded: 4) 2022-04-05 15:08:00 [info] end (action_name: xdmod.jobs-common.SubmissionVenueIngestor, action: xdmod.jobs-common.SubmissionVenueIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:00 [notice] Finished processing section 'xdmod.jobs-common' 2022-04-05 15:08:00 [notice] Start processing section 'xdmod.ingest-organizations' 2022-04-05 15:08:00 [info] start (action_name: xdmod.ingest-organizations.IngestOrganizationConfig, action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:00 [debug] Discover table 'mod_shredder.staging_organization' 2022-04-05 15:08:00 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:00 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:00 [debug] Parsing /etc/xdmod/etl/../organization.json 2022-04-05 15:08:00 [debug] Registering filter: xdmod.external_process 2022-04-05 15:08:00 [debug] Creating filter xdmod.external_process: jq '{id:1,name,abbrev}' 2022-04-05 15:08:00 [debug] Adding filter xdmod.external_process to stream: jq 2022-04-05 15:08:00 [debug] Parsed 1 records 2022-04-05 15:08:00 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/common/organization.schema.json 2022-04-05 15:08:00 [debug] Requested 3 record fields: id, name, abbrev 2022-04-05 15:08:00 [debug] Processing destination field map 2022-04-05 15:08:00 [debug] Insert SQL for table key 'staging_organization': INSERT INTO mod_shredder.staging_organization (organization_id, organization_name, organization_abbrev) VALUES (?, ?, ?) ON DUPLICATE KEY UPDATE organization_id = COALESCE(VALUES(organization_id), organization_id), organization_name = COALESCE(VALUES(organization_name), organization_name), organization_abbrev = COALESCE(VALUES(organization_abbrev), organization_abbrev) 2022-04-05 15:08:00 [info] Execute Post-execute tasks: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:00 [debug] ANALYZE TABLE mod_shredder.staging_organization 2022-04-05 15:08:00 [debug] Completed in 0.014110s 2022-04-05 15:08:00 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 0.12 s) 2022-04-05 15:08:00 [notice] (action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171280.3612, end_time: 1649171280.4772, elapsed_time: 0.11601, records_examined: 1, records_loaded: 1) 2022-04-05 15:08:00 [info] end (action_name: xdmod.ingest-organizations.IngestOrganizationConfig, action: xdmod.ingest-organizations.IngestOrganizationConfig (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:00 [info] start (action_name: xdmod.ingest-organizations.IngestOrganizationHpcdb, action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:00 [debug] Discover table 'mod_hpcdb.hpcdb_organizations' 2022-04-05 15:08:00 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:00 [debug] Available Variables: DESTINATION_SCHEMA='mod_hpcdb', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:00 [debug] Allowing same-server SQL optimizations 2022-04-05 15:08:00 [info] Single-database ingest into ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:00 [debug] INSERT INTO mod_hpcdb.hpcdb_organizations (organization_id,organization_name,organization_abbrev) SELECT o.organization_id AS organization_id, o.organization_name AS organization_name, o.organization_abbrev AS organization_abbrev FROM mod_shredder.staging_organization AS o ON DUPLICATE KEY UPDATE organization_id=VALUES(organization_id),organization_name=VALUES(organization_name),organization_abbrev=VALUES(organization_abbrev) 2022-04-05 15:08:00 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-04-05 15:08:00 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.07 s) 2022-04-05 15:08:00 [notice] (action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor), start_time: 1649171280.5351, end_time: 1649171280.6071, elapsed_time: 0.07193, records_examined: 0, records_loaded: 0) 2022-04-05 15:08:00 [info] end (action_name: xdmod.ingest-organizations.IngestOrganizationHpcdb, action: xdmod.ingest-organizations.IngestOrganizationHpcdb (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:00 [info] start (action_name: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse, action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:00 [debug] Discover table 'modw.organization' 2022-04-05 15:08:00 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:00 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_hpcdb', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:00 [debug] Query optimization disabled 2022-04-05 15:08:00 [debug] Using multi-database ingest 2022-04-05 15:08:00 [debug] Using temporary file '/tmp/modw.organization.data.ts_1649171280.141827548uggKSA' for destination table key 'organization' 2022-04-05 15:08:00 [debug] LOAD statement for destination table key 'organization' ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): CREATE TABLE modw.tmp_organization_1649171280 LIKE modw.organization; ALTER TABLE modw.tmp_organization_1649171280 DISABLE KEYS; LOAD DATA LOCAL INFILE '/tmp/modw.organization.data.ts_1649171280.141827548uggKSA' INTO TABLE modw.tmp_organization_1649171280 FIELDS TERMINATED BY 0x1e OPTIONALLY ENCLOSED BY 0x1f ESCAPED BY 0x5c LINES TERMINATED BY 0x1d (id,abbrev,name,short_name,long_name,organization_origin_id,order_id); SHOW WARNINGS; INSERT INTO modw.organization (id,abbrev,name,short_name,long_name,organization_origin_id,order_id) SELECT id,abbrev,name,short_name,long_name,organization_origin_id,order_id FROM modw.tmp_organization_1649171280 ON DUPLICATE KEY UPDATE id=VALUES(id),abbrev=VALUES(abbrev),name=VALUES(name),short_name=VALUES(short_name),long_name=VALUES(long_name),organization_origin_id=VALUES(organization_origin_id),order_id=VALUES(order_id); DROP TABLE modw.tmp_organization_1649171280; 2022-04-05 15:08:00 [info] Multi-database ingest into ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:00 [info] Switching to un-buffered query mode 2022-04-05 15:08:00 [debug] Current net_write_timeout = 60 2022-04-05 15:08:00 [info] ETL\Ingestor\DatabaseIngestor: Querying ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:00 [debug] Source query ('HPCDB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=mod_hpcdb, host=mariadb:3306, user=xdmod): SELECT organization_id AS id, organization_abbrev AS abbrev, organization_name AS name, COALESCE(organization_abbrev, organization_name) AS short_name, CASE WHEN ISNULL(organization_abbrev) THEN organization_name ELSE CONCAT(organization_abbrev, ' - ', organization_name) END AS long_name, NULL AS order_id, organization_id AS organization_origin_id FROM mod_hpcdb.hpcdb_organizations 2022-04-05 15:08:00 [debug] Loaded 1 records into 'organization' 2022-04-05 15:08:00 [debug] Loaded 1 files in 0s 2022-04-05 15:08:00 [info] ETL\Ingestor\DatabaseIngestor: Processed 1 records (1 source records) in 0s 2022-04-05 15:08:00 [info] Returning buffered query mode to: true 2022-04-05 15:08:00 [info] Execute Post-execute tasks: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:00 [debug] ANALYZE TABLE modw.organization 2022-04-05 15:08:00 [debug] Completed in 0.015030s 2022-04-05 15:08:00 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 1 records (Time Taken: 0.27 s) 2022-04-05 15:08:00 [notice] (action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_time: 1649171280.7018, end_time: 1649171280.9733, elapsed_time: 0.2715, records_examined: 1, records_loaded: 1) 2022-04-05 15:08:01 [info] end (action_name: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse, action: xdmod.ingest-organizations.IngestOrgranizationDatawarehouse (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:01 [notice] Finished processing section 'xdmod.ingest-organizations' 2022-04-05 15:08:01 [notice] Start processing section 'xdmod.ingest-resource-types' 2022-04-05 15:08:01 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement, action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables), start_date: , end_date: ) 2022-04-05 15:08:01 [debug] Discover table 'modw.staging_resource_type' 2022-04-05 15:08:01 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables), start_time: 1649171281.064, end_time: 1649171281.0845, elapsed_time: 0.02053) 2022-04-05 15:08:01 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement, action: xdmod.ingest-resource-types.ResourceTypesStagingTableManagement (ETL\Maintenance\ManageTables)) 2022-04-05 15:08:01 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesStagingUnknown, action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_date: , end_date: ) 2022-04-05 15:08:01 [notice] Processing SQL file '/etc/xdmod/etl/etl_sql.d/cloud_openstack/unknown_resource_type.sql' using delimiter '//' containing 2 statements 2022-04-05 15:08:01 [info] Executing statement ( 1 / 2) (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)-sql-1, endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod), sql: SET SESSION sql_mode='NO_AUTO_VALUE_ON_ZERO') 2022-04-05 15:08:01 [info] Finished executing statement ( 1 / 2) (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)-sql-1, rows: 0, start_time: 1649171281.1432, end_time: 1649171281.1601, elapsed_time: 0.016932964324951) 2022-04-05 15:08:01 [info] Executing statement ( 2 / 2) (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)-sql-2, endpoint: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod), sql: INSERT IGNORE INTO mod_shredder.staging_resource_type (resource_type_id, resource_type_description, resource_type_abbrev) VALUES ('0', 'Unknown Resource Type', 'UNK')) 2022-04-05 15:08:01 [info] Finished executing statement ( 2 / 2) (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)-sql-2, rows: 0, start_time: 1649171281.1749, end_time: 1649171281.1898, elapsed_time: 0.014877080917358) 2022-04-05 15:08:01 [notice] Finished Processing 2 SQL statements 2022-04-05 15:08:01 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql), start_time: 1649171281.1286, end_time: 1649171281.2211, elapsed_time: 0.09246) 2022-04-05 15:08:01 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesStagingUnknown, action: xdmod.ingest-resource-types.ResourceTypesStagingUnknown (ETL\Maintenance\ExecuteSql)) 2022-04-05 15:08:01 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesStaging, action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:01 [debug] Discover table 'mod_shredder.staging_resource_type' 2022-04-05 15:08:01 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:01 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:01 [debug] Parsing /etc/xdmod/etl/../resource_types.json 2022-04-05 15:08:01 [debug] Registering filter: xdmod.external_process 2022-04-05 15:08:01 [debug] Creating filter xdmod.external_process: jq '[ .resource_types | to_entries[]| {abbrev: .key, description:.value.description} ] ' 2022-04-05 15:08:01 [debug] Adding filter xdmod.external_process to stream: jq 2022-04-05 15:08:01 [debug] Parsed 12 records 2022-04-05 15:08:01 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/common/resource-types.schema.json 2022-04-05 15:08:01 [debug] Requested 2 record fields: abbrev, description 2022-04-05 15:08:01 [debug] Processing destination field map 2022-04-05 15:08:01 [debug] Insert SQL for table key 'staging_resource_type': INSERT INTO mod_shredder.staging_resource_type (resource_type_description, resource_type_abbrev) VALUES (?, ?) ON DUPLICATE KEY UPDATE resource_type_description = COALESCE(VALUES(resource_type_description), resource_type_description), resource_type_abbrev = COALESCE(VALUES(resource_type_abbrev), resource_type_abbrev) 2022-04-05 15:08:01 [info] Execute Post-execute tasks: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:01 [debug] ANALYZE TABLE mod_shredder.staging_resource_type 2022-04-05 15:08:01 [debug] Completed in 0.014340s 2022-04-05 15:08:01 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 12 records (Time Taken: 0.12 s) 2022-04-05 15:08:01 [notice] (action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171281.2664, end_time: 1649171281.382, elapsed_time: 0.11559, records_examined: 12, records_loaded: 12) 2022-04-05 15:08:01 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesStaging, action: xdmod.ingest-resource-types.ResourceTypesStaging (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:01 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesHpcdb, action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:01 [debug] Discover table 'mod_hpcdb.hpcdb_resource_types' 2022-04-05 15:08:01 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:01 [debug] Available Variables: DESTINATION_SCHEMA='mod_hpcdb', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:01 [debug] Allowing same-server SQL optimizations 2022-04-05 15:08:01 [info] Single-database ingest into ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:01 [debug] INSERT INTO mod_hpcdb.hpcdb_resource_types (type_id,type_abbr,type_desc) SELECT rt.resource_type_id AS type_id, rt.resource_type_abbrev AS type_abbr, rt.resource_type_description AS type_desc FROM mod_shredder.staging_resource_type AS rt ON DUPLICATE KEY UPDATE type_id=VALUES(type_id),type_abbr=VALUES(type_abbr),type_desc=VALUES(type_desc) 2022-04-05 15:08:01 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-04-05 15:08:01 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.07 s) 2022-04-05 15:08:01 [notice] (action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor), start_time: 1649171281.4388, end_time: 1649171281.5066, elapsed_time: 0.06781, records_examined: 0, records_loaded: 0) 2022-04-05 15:08:01 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesHpcdb, action: xdmod.ingest-resource-types.ResourceTypesHpcdb (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:01 [info] start (action_name: xdmod.ingest-resource-types.ResourceTypesDatawarehouse, action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:01 [debug] Discover table 'modw.resourcetype' 2022-04-05 15:08:01 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:01 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_hpcdb', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:01 [debug] Allowing same-server SQL optimizations 2022-04-05 15:08:01 [info] Single-database ingest into ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:01 [debug] INSERT INTO modw.resourcetype (id,description,abbrev) SELECT rtf.type_id AS id, rtf.type_desc AS description, rtf.type_abbr AS abbrev FROM mod_hpcdb.hpcdb_resource_types AS rtf ON DUPLICATE KEY UPDATE id=VALUES(id),description=VALUES(description),abbrev=VALUES(abbrev) 2022-04-05 15:08:01 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-04-05 15:08:01 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.07 s) 2022-04-05 15:08:01 [notice] (action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor), start_time: 1649171281.5651, end_time: 1649171281.6362, elapsed_time: 0.07108, records_examined: 0, records_loaded: 0) 2022-04-05 15:08:01 [info] end (action_name: xdmod.ingest-resource-types.ResourceTypesDatawarehouse, action: xdmod.ingest-resource-types.ResourceTypesDatawarehouse (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:01 [notice] Finished processing section 'xdmod.ingest-resource-types' 2022-04-05 15:08:01 [notice] Start processing section 'xdmod.ingest-resources' 2022-04-05 15:08:01 [info] start (action_name: xdmod.ingest-resources.IngestResourceConfig, action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:01 [debug] Discover table 'mod_shredder.staging_resource_config' 2022-04-05 15:08:01 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:01 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:01 [debug] Parsing /etc/xdmod/etl/../resources.json 2022-04-05 15:08:01 [debug] Parsed 1 records 2022-04-05 15:08:01 [debug] Validating data against schema /etc/xdmod/etl/etl_schemas.d/common/resources.schema.json 2022-04-05 15:08:01 [debug] Requested 7 record fields: resource, name, description, resource_type, pi_column, shared_jobs, timezone 2022-04-05 15:08:01 [debug] Processing destination field map 2022-04-05 15:08:01 [debug] Insert SQL for table key 'staging_resource_config': INSERT INTO mod_shredder.staging_resource_config (resource, name, description, shared_jobs, timezone, type_abbrev) VALUES (?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE resource = COALESCE(VALUES(resource), resource), name = COALESCE(VALUES(name), name), description = COALESCE(VALUES(description), description), shared_jobs = COALESCE(VALUES(shared_jobs), shared_jobs), timezone = COALESCE(VALUES(timezone), timezone), type_abbrev = COALESCE(VALUES(type_abbrev), type_abbrev) 2022-04-05 15:08:01 [info] Execute Post-execute tasks: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:01 [debug] ANALYZE TABLE mod_shredder.staging_resource_config 2022-04-05 15:08:01 [debug] Completed in 0.015220s 2022-04-05 15:08:01 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 0.07 s) 2022-04-05 15:08:01 [notice] (action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171281.731, end_time: 1649171281.8024, elapsed_time: 0.0714, records_examined: 1, records_loaded: 1) 2022-04-05 15:08:01 [info] end (action_name: xdmod.ingest-resources.IngestResourceConfig, action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:01 [info] start (action_name: xdmod.ingest-resources.IngestResourcesStaging, action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:01 [debug] Discover table 'mod_shredder.staging_resource' 2022-04-05 15:08:01 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:01 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:01 [debug] Allowing same-server SQL optimizations 2022-04-05 15:08:01 [info] Single-database ingest into ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:01 [debug] INSERT INTO mod_shredder.staging_resource (resource_name) SELECT DISTINCT rc.resource AS resource_name FROM mod_shredder.staging_resource_config AS rc ON DUPLICATE KEY UPDATE resource_name=VALUES(resource_name) 2022-04-05 15:08:01 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-04-05 15:08:01 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.07 s) 2022-04-05 15:08:01 [notice] (action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_time: 1649171281.8698, end_time: 1649171281.9407, elapsed_time: 0.07085, records_examined: 0, records_loaded: 0) 2022-04-05 15:08:01 [info] end (action_name: xdmod.ingest-resources.IngestResourcesStaging, action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:01 [info] start (action_name: xdmod.ingest-resources.HpcdbIngestResources, action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:02 [debug] Discover table 'mod_hpcdb.hpcdb_resources' 2022-04-05 15:08:02 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:02 [debug] Available Variables: DESTINATION_SCHEMA='mod_hpcdb', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:02 [debug] Allowing same-server SQL optimizations 2022-04-05 15:08:02 [info] Single-database ingest into ('HPCDB Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_hpcdb, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:02 [debug] INSERT INTO mod_hpcdb.hpcdb_resources (resource_id,resource_type_id,organization_id,resource_name,resource_code,resource_description,resource_shared_jobs,resource_timezone) SELECT r.resource_id AS resource_id, COALESCE(srt.resource_type_id, 0) AS resource_type_id, 1 AS organization_id, rc.name AS resource_name, rc.resource AS resource_code, rc.description AS resource_description, COALESCE(rc.shared_jobs, FALSE) AS resource_shared_jobs, COALESCE(rc.timezone, 'UTC') AS resource_timezone FROM mod_shredder.staging_resource AS r LEFT OUTER JOIN mod_shredder.staging_resource_config AS rc ON r.resource_name = rc.resource LEFT JOIN mod_shredder.staging_resource_type AS srt ON rc.type_abbrev = srt.resource_type_abbrev ON DUPLICATE KEY UPDATE resource_id=VALUES(resource_id),resource_type_id=VALUES(resource_type_id),organization_id=VALUES(organization_id),resource_name=VALUES(resource_name),resource_code=VALUES(resource_code),resource_description=VALUES(resource_description),resource_shared_jobs=VALUES(resource_shared_jobs),resource_timezone=VALUES(resource_timezone) 2022-04-05 15:08:02 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-04-05 15:08:02 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.07 s) 2022-04-05 15:08:02 [notice] (action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor), start_time: 1649171282.0052, end_time: 1649171282.0797, elapsed_time: 0.07446, records_examined: 0, records_loaded: 0) 2022-04-05 15:08:02 [info] end (action_name: xdmod.ingest-resources.HpcdbIngestResources, action: xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:02 [info] start (action_name: xdmod.ingest-resources.IngestResourcefact, action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:02 [debug] Discover table 'modw.resourcefact' 2022-04-05 15:08:02 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:02 [debug] Available Variables: DESTINATION_SCHEMA='modw', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_hpcdb', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:02 [debug] Allowing same-server SQL optimizations 2022-04-05 15:08:02 [info] Single-database ingest into ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:02 [debug] INSERT INTO modw.resourcefact (id,resourcetype_id,organization_id,name,code,description,shared_jobs,timezone,resource_origin_id) SELECT r.resource_id AS id, r.resource_type_id AS resourcetype_id, r.organization_id AS organization_id, r.resource_name AS name, r.resource_code AS code, r.resource_description AS description, r.resource_shared_jobs AS shared_jobs, r.resource_timezone AS timezone, r.resource_id AS resource_origin_id FROM mod_hpcdb.hpcdb_resources AS r ORDER BY r.resource_id ON DUPLICATE KEY UPDATE id=VALUES(id),resourcetype_id=VALUES(resourcetype_id),organization_id=VALUES(organization_id),name=VALUES(name),code=VALUES(code),description=VALUES(description),shared_jobs=VALUES(shared_jobs),timezone=VALUES(timezone),resource_origin_id=VALUES(resource_origin_id) 2022-04-05 15:08:02 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-04-05 15:08:02 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.07 s) 2022-04-05 15:08:02 [notice] (action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor), start_time: 1649171282.1439, end_time: 1649171282.217, elapsed_time: 0.07317, records_examined: 0, records_loaded: 0) 2022-04-05 15:08:02 [info] end (action_name: xdmod.ingest-resources.IngestResourcefact, action: xdmod.ingest-resources.IngestResourcefact (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:02 [info] start (action_name: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging, action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:02 [debug] Discover table 'mod_shredder.staging_resource_type_realms' 2022-04-05 15:08:02 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:02 [debug] Available Variables: DESTINATION_SCHEMA='mod_shredder', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:02 [debug] Loading configuration file /etc/xdmod/resource_types.json 2022-04-05 15:08:02 [debug] Parsing /etc/xdmod/resource_types.json 2022-04-05 15:08:02 [debug] Parsed 1 records 2022-04-05 15:08:02 [debug] Stored object Configuration\XdmodConfiguration (/etc/xdmod/etl/../resource_types.json) in APCu cache with key Configuration\XdmodConfiguration|/etc/xdmod/etl/../resource_types.json|40cd750bba9870f18aada2478b24840a in 0.001685s 2022-04-05 15:08:02 [debug] Parsing /tmp/etl-xdmod-config-jsonUHZPuP 2022-04-05 15:08:02 [debug] Registering filter: xdmod.external_process 2022-04-05 15:08:02 [debug] Creating filter xdmod.external_process: jq '[(.resource_types | keys) as $a | .resource_types as $rt | $a[] | {abbrev: . , realm: $rt[.].realms[]}]' 2022-04-05 15:08:02 [debug] Adding filter xdmod.external_process to stream: jq 2022-04-05 15:08:02 [debug] Parsed 9 records 2022-04-05 15:08:02 [debug] Requested 2 record fields: abbrev, realm 2022-04-05 15:08:02 [debug] No destination_field_map specified 2022-04-05 15:08:02 [debug] Auto-generating destination_field_map using 2 source fields: abbrev, realm 2022-04-05 15:08:02 [debug] Available fields for table key 'staging_resource_type_realms': staging_resource_type_realm_id, abbrev, realm 2022-04-05 15:08:02 [debug] Generated destination_field_map: Table: staging_resource_type_realms abbrev -> abbrev realm -> realm 2022-04-05 15:08:02 [debug] Processing destination field map 2022-04-05 15:08:02 [debug] Insert SQL for table key 'staging_resource_type_realms': INSERT INTO mod_shredder.staging_resource_type_realms (abbrev, realm) VALUES (?, ?) ON DUPLICATE KEY UPDATE abbrev = COALESCE(VALUES(abbrev), abbrev), realm = COALESCE(VALUES(realm), realm) 2022-04-05 15:08:02 [info] Execute Post-execute tasks: ('Shredder/Staging Database', class=ETL\DataEndpoint\Mysql, config=database, schema=mod_shredder, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:02 [debug] ANALYZE TABLE mod_shredder.staging_resource_type_realms 2022-04-05 15:08:02 [debug] Completed in 0.015940s 2022-04-05 15:08:02 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 9 records (Time Taken: 0.20 s) 2022-04-05 15:08:02 [notice] (action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171282.278, end_time: 1649171282.4828, elapsed_time: 0.20484, records_examined: 9, records_loaded: 9) 2022-04-05 15:08:02 [info] end (action_name: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging, action: xdmod.ingest-resources.IngestResourceTypeRealmRelationsStaging (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:02 [info] start (action_name: xdmod.ingest-resources.IngestInitialRealms, action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:02 [debug] Discover table 'moddb.realms' 2022-04-05 15:08:02 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:02 [debug] Available Variables: DESTINATION_SCHEMA='moddb', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:02 [debug] Allowing same-server SQL optimizations 2022-04-05 15:08:02 [info] Single-database ingest into ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:02 [debug] INSERT INTO moddb.realms (display,name,module_id) SELECT DISTINCT srtr.realm AS display, LOWER(srtr.realm) AS name, 1 AS module_id FROM mod_shredder.staging_resource_type_realms AS srtr ON DUPLICATE KEY UPDATE display=VALUES(display),name=VALUES(name),module_id=VALUES(module_id) 2022-04-05 15:08:02 [info] ETL\Ingestor\DatabaseIngestor: Processed 0 records 2022-04-05 15:08:02 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 0 records (Time Taken: 0.08 s) 2022-04-05 15:08:02 [notice] (action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor), start_time: 1649171282.5469, end_time: 1649171282.6297, elapsed_time: 0.08272, records_examined: 0, records_loaded: 0) 2022-04-05 15:08:02 [info] end (action_name: xdmod.ingest-resources.IngestInitialRealms, action: xdmod.ingest-resources.IngestInitialRealms (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:02 [info] start (action_name: xdmod.ingest-resources.IngestResourcefactRealmRelations, action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor), start_date: , end_date: ) 2022-04-05 15:08:02 [debug] Discover table 'moddb.resource_type_realms' 2022-04-05 15:08:02 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:02 [debug] Available Variables: DESTINATION_SCHEMA='moddb', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', SOURCE_SCHEMA='mod_shredder', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:02 [debug] Allowing same-server SQL optimizations 2022-04-05 15:08:02 [info] Single-database ingest into ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:02 [debug] INSERT INTO moddb.resource_type_realms (resource_type_id,realm_id) SELECT rt.id AS resource_type_id, r.realm_id AS realm_id FROM mod_shredder.staging_resource_type_realms AS srtr JOIN modw.resourcetype AS rt ON rt.abbrev = srtr.abbrev JOIN moddb.realms AS r ON r.display = srtr.realm ON DUPLICATE KEY UPDATE resource_type_id=VALUES(resource_type_id),realm_id=VALUES(realm_id) 2022-04-05 15:08:02 [info] ETL\Ingestor\DatabaseIngestor: Processed 45 records 2022-04-05 15:08:02 [info] Execute Post-execute tasks: ('Database', class=ETL\DataEndpoint\Mysql, config=database, schema=moddb, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:02 [debug] ANALYZE TABLE moddb.resource_type_realms 2022-04-05 15:08:02 [debug] Completed in 0.015870s 2022-04-05 15:08:02 [info] ETL\Ingestor\DatabaseIngestor: Rows Processed: 45 records (Time Taken: 0.13 s) 2022-04-05 15:08:02 [notice] (action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor), start_time: 1649171282.6957, end_time: 1649171282.8214, elapsed_time: 0.12572, records_examined: 45, records_loaded: 45) 2022-04-05 15:08:02 [info] end (action_name: xdmod.ingest-resources.IngestResourcefactRealmRelations, action: xdmod.ingest-resources.IngestResourcefactRealmRelations (ETL\Ingestor\DatabaseIngestor)) 2022-04-05 15:08:02 [notice] Finished processing section 'xdmod.ingest-resources' 2022-04-05 15:08:02 [notice] Start processing section 'xdmod.jobs-cloud-common' 2022-04-05 15:08:02 [info] start (action_name: xdmod.jobs-cloud-common.CloudTableManagement, action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_date: , end_date: ) 2022-04-05 15:08:02 [debug] Discover table 'modw_cloud.account' 2022-04-05 15:08:02 [debug] Discover table 'modw_cloud.event' 2022-04-05 15:08:02 [debug] Column last_modified: values for "timestamp" differ ("CURRENT_TIMESTAMP on update current_timestamp" != "current_timestamp() on update current_timestamp()") 2022-04-05 15:08:02 [notice] Altering table modw_cloud.event 2022-04-05 15:08:02 [debug] Alter table SQL ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod): ALTER TABLE modw_cloud.event CHANGE COLUMN last_modified last_modified timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP on update current_timestamp ; 2022-04-05 15:08:02 [debug] Discover table 'modw_cloud.asset' 2022-04-05 15:08:02 [debug] Discover table 'modw_cloud.host' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.instance_data' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.event_asset' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.raw_resource_specs' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.staging_resource_specifications' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.cloud_resource_specs' 2022-04-05 15:08:03 [debug] Column last_modified: values for "timestamp" differ ("CURRENT_TIMESTAMP on update current_timestamp" != "current_timestamp() on update current_timestamp()") 2022-04-05 15:08:03 [notice] Altering table modw_cloud.cloud_resource_specs 2022-04-05 15:08:03 [debug] Alter table SQL ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod): ALTER TABLE modw_cloud.cloud_resource_specs CHANGE COLUMN last_modified last_modified timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP on update current_timestamp ; 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.openstack_raw_event' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.openstack_raw_instance_type' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.openstack_raw_volume' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.generic_cloud_raw_event' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.generic_cloud_raw_instance_type' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.generic_cloud_raw_volume' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.domains' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.domain_submission_venues_staging' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.domain_submission_venues' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.staging_pi_to_project' 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.session_records' 2022-04-05 15:08:03 [debug] Column last_modified: values for "timestamp" differ ("CURRENT_TIMESTAMP on update current_timestamp" != "current_timestamp() on update current_timestamp()") 2022-04-05 15:08:03 [notice] Altering table modw_cloud.session_records 2022-04-05 15:08:03 [debug] Alter table SQL ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod): ALTER TABLE modw_cloud.session_records CHANGE COLUMN last_modified last_modified timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP on update current_timestamp ; 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.cloudfact_by_day_sessionlist' 2022-04-05 15:08:03 [notice] (action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables), start_time: 1649171282.9162, end_time: 1649171283.2678, elapsed_time: 0.35161) 2022-04-05 15:08:03 [info] end (action_name: xdmod.jobs-cloud-common.CloudTableManagement, action: xdmod.jobs-cloud-common.CloudTableManagement (ETL\Maintenance\ManageTables)) 2022-04-05 15:08:03 [info] start (action_name: xdmod.jobs-cloud-common.CloudAssetTypeInitializer, action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.asset_type' 2022-04-05 15:08:03 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:03 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:03 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/asset_type.json 2022-04-05 15:08:03 [debug] Parsed 6 records 2022-04-05 15:08:03 [debug] Requested 5 record fields: asset_type_id, asset_type, display, description, unit_id 2022-04-05 15:08:03 [debug] No destination_field_map specified 2022-04-05 15:08:03 [debug] Auto-generating destination_field_map using 5 source fields: asset_type_id, asset_type, display, description, unit_id 2022-04-05 15:08:03 [debug] Available fields for table key 'asset_type': asset_type_id, asset_type, display, description, unit_id 2022-04-05 15:08:03 [debug] Generated destination_field_map: Table: asset_type asset_type_id -> asset_type_id asset_type -> asset_type display -> display description -> description unit_id -> unit_id 2022-04-05 15:08:03 [debug] Processing destination field map 2022-04-05 15:08:03 [debug] Insert SQL for table key 'asset_type': INSERT INTO modw_cloud.asset_type (asset_type_id, asset_type, display, description, unit_id) VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE asset_type_id = COALESCE(VALUES(asset_type_id), asset_type_id), asset_type = COALESCE(VALUES(asset_type), asset_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description), unit_id = COALESCE(VALUES(unit_id), unit_id) 2022-04-05 15:08:03 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:03 [debug] ANALYZE TABLE modw_cloud.asset_type 2022-04-05 15:08:03 [debug] Completed in 0.016430s 2022-04-05 15:08:03 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 5 records (Time Taken: 0.07 s) 2022-04-05 15:08:03 [notice] (action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171283.3185, end_time: 1649171283.3909, elapsed_time: 0.0724, records_examined: 5, records_loaded: 5) 2022-04-05 15:08:03 [info] end (action_name: xdmod.jobs-cloud-common.CloudAssetTypeInitializer, action: xdmod.jobs-cloud-common.CloudAssetTypeInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:03 [info] start (action_name: xdmod.jobs-cloud-common.CloudEventTypeInitializer, action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.event_type' 2022-04-05 15:08:03 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:03 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:03 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/event_type.json 2022-04-05 15:08:03 [debug] Parsed 66 records 2022-04-05 15:08:03 [debug] Requested 4 record fields: event_type_id, event_type, display, description 2022-04-05 15:08:03 [debug] No destination_field_map specified 2022-04-05 15:08:03 [debug] Auto-generating destination_field_map using 4 source fields: event_type_id, event_type, display, description 2022-04-05 15:08:03 [debug] Available fields for table key 'event_type': event_type_id, event_type, display, description 2022-04-05 15:08:03 [debug] Generated destination_field_map: Table: event_type event_type_id -> event_type_id event_type -> event_type display -> display description -> description 2022-04-05 15:08:03 [debug] Processing destination field map 2022-04-05 15:08:03 [debug] Insert SQL for table key 'event_type': INSERT INTO modw_cloud.event_type (event_type_id, event_type, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE event_type_id = COALESCE(VALUES(event_type_id), event_type_id), event_type = COALESCE(VALUES(event_type), event_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-04-05 15:08:03 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:03 [debug] ANALYZE TABLE modw_cloud.event_type 2022-04-05 15:08:03 [debug] Completed in 0.013910s 2022-04-05 15:08:03 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 65 records (Time Taken: 0.09 s) 2022-04-05 15:08:03 [notice] (action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171283.457, end_time: 1649171283.5425, elapsed_time: 0.08546, records_examined: 65, records_loaded: 65) 2022-04-05 15:08:03 [info] end (action_name: xdmod.jobs-cloud-common.CloudEventTypeInitializer, action: xdmod.jobs-cloud-common.CloudEventTypeInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:03 [info] start (action_name: xdmod.jobs-cloud-common.CloudRecordTypeInitializer, action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.record_type' 2022-04-05 15:08:03 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:03 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:03 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/record_type.json 2022-04-05 15:08:03 [debug] Parsed 5 records 2022-04-05 15:08:03 [debug] Requested 4 record fields: record_type_id, record_type, display, description 2022-04-05 15:08:03 [debug] No destination_field_map specified 2022-04-05 15:08:03 [debug] Auto-generating destination_field_map using 4 source fields: record_type_id, record_type, display, description 2022-04-05 15:08:03 [debug] Available fields for table key 'record_type': record_type_id, record_type, display, description 2022-04-05 15:08:03 [debug] Generated destination_field_map: Table: record_type record_type_id -> record_type_id record_type -> record_type display -> display description -> description 2022-04-05 15:08:03 [debug] Processing destination field map 2022-04-05 15:08:03 [debug] Insert SQL for table key 'record_type': INSERT INTO modw_cloud.record_type (record_type_id, record_type, display, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE record_type_id = COALESCE(VALUES(record_type_id), record_type_id), record_type = COALESCE(VALUES(record_type), record_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description) 2022-04-05 15:08:03 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:03 [debug] ANALYZE TABLE modw_cloud.record_type 2022-04-05 15:08:03 [debug] Completed in 0.015380s 2022-04-05 15:08:03 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 4 records (Time Taken: 0.06 s) 2022-04-05 15:08:03 [notice] (action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171283.6159, end_time: 1649171283.6802, elapsed_time: 0.06434, records_examined: 4, records_loaded: 4) 2022-04-05 15:08:03 [info] end (action_name: xdmod.jobs-cloud-common.CloudRecordTypeInitializer, action: xdmod.jobs-cloud-common.CloudRecordTypeInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:03 [info] start (action_name: xdmod.jobs-cloud-common.CloudHostUnknownInitializer, action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.host' 2022-04-05 15:08:03 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:03 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:03 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/host.json 2022-04-05 15:08:03 [debug] Parsed 2 records 2022-04-05 15:08:03 [debug] Requested 3 record fields: resource_id, host_id, hostname 2022-04-05 15:08:03 [debug] No destination_field_map specified 2022-04-05 15:08:03 [debug] Auto-generating destination_field_map using 3 source fields: resource_id, host_id, hostname 2022-04-05 15:08:03 [debug] Available fields for table key 'host': resource_id, host_id, hostname 2022-04-05 15:08:03 [debug] Generated destination_field_map: Table: host resource_id -> resource_id host_id -> host_id hostname -> hostname 2022-04-05 15:08:03 [debug] Processing destination field map 2022-04-05 15:08:03 [debug] Insert SQL for table key 'host': INSERT INTO modw_cloud.host (resource_id, host_id, hostname) VALUES (?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), host_id = COALESCE(VALUES(host_id), host_id), hostname = COALESCE(VALUES(hostname), hostname) 2022-04-05 15:08:03 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:03 [debug] ANALYZE TABLE modw_cloud.host 2022-04-05 15:08:03 [debug] Completed in 0.018600s 2022-04-05 15:08:03 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 0.08 s) 2022-04-05 15:08:03 [notice] (action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171283.7433, end_time: 1649171283.8197, elapsed_time: 0.07633, records_examined: 1, records_loaded: 1) 2022-04-05 15:08:03 [info] end (action_name: xdmod.jobs-cloud-common.CloudHostUnknownInitializer, action: xdmod.jobs-cloud-common.CloudHostUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:03 [info] start (action_name: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer, action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:03 [debug] Discover table 'modw_cloud.account' 2022-04-05 15:08:03 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:03 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:03 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/account.json 2022-04-05 15:08:03 [debug] Parsed 2 records 2022-04-05 15:08:03 [debug] Requested 4 record fields: resource_id, account_id, provider_account, display 2022-04-05 15:08:03 [debug] No destination_field_map specified 2022-04-05 15:08:03 [debug] Auto-generating destination_field_map using 4 source fields: resource_id, account_id, provider_account, display 2022-04-05 15:08:03 [debug] Available fields for table key 'account': resource_id, account_id, provider_account, display, principalinvestigator_person_id, fos_id 2022-04-05 15:08:03 [debug] Generated destination_field_map: Table: account resource_id -> resource_id account_id -> account_id provider_account -> provider_account display -> display 2022-04-05 15:08:03 [debug] Processing destination field map 2022-04-05 15:08:03 [debug] Insert SQL for table key 'account': INSERT INTO modw_cloud.account (resource_id, account_id, provider_account, display) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), account_id = COALESCE(VALUES(account_id), account_id), provider_account = COALESCE(VALUES(provider_account), provider_account), display = COALESCE(VALUES(display), display) 2022-04-05 15:08:03 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:03 [debug] ANALYZE TABLE modw_cloud.account 2022-04-05 15:08:03 [debug] Completed in 0.016540s 2022-04-05 15:08:03 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 0.08 s) 2022-04-05 15:08:03 [notice] (action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171283.8862, end_time: 1649171283.9625, elapsed_time: 0.07629, records_examined: 1, records_loaded: 1) 2022-04-05 15:08:04 [info] end (action_name: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer, action: xdmod.jobs-cloud-common.CloudAccountUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:04 [info] start (action_name: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer, action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:04 [debug] Discover table 'modw_cloud.instance' 2022-04-05 15:08:04 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:04 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/instance.json 2022-04-05 15:08:04 [debug] Parsed 2 records 2022-04-05 15:08:04 [debug] Requested 5 record fields: resource_id, instance_id, account_id, provider_identifier, person_id 2022-04-05 15:08:04 [debug] No destination_field_map specified 2022-04-05 15:08:04 [debug] Auto-generating destination_field_map using 5 source fields: resource_id, instance_id, account_id, provider_identifier, person_id 2022-04-05 15:08:04 [debug] Available fields for table key 'instance': resource_id, instance_id, account_id, provider_identifier, person_id, service_provider 2022-04-05 15:08:04 [debug] Generated destination_field_map: Table: instance resource_id -> resource_id instance_id -> instance_id account_id -> account_id provider_identifier -> provider_identifier person_id -> person_id 2022-04-05 15:08:04 [debug] Processing destination field map 2022-04-05 15:08:04 [debug] Insert SQL for table key 'instance': INSERT INTO modw_cloud.instance (resource_id, instance_id, account_id, provider_identifier, person_id) VALUES (?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), instance_id = COALESCE(VALUES(instance_id), instance_id), account_id = COALESCE(VALUES(account_id), account_id), provider_identifier = COALESCE(VALUES(provider_identifier), provider_identifier), person_id = COALESCE(VALUES(person_id), person_id) 2022-04-05 15:08:04 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:04 [debug] ANALYZE TABLE modw_cloud.instance 2022-04-05 15:08:04 [debug] Completed in 0.020090s 2022-04-05 15:08:04 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 0.07 s) 2022-04-05 15:08:04 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171284.0347, end_time: 1649171284.109, elapsed_time: 0.07433, records_examined: 1, records_loaded: 1) 2022-04-05 15:08:04 [info] end (action_name: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer, action: xdmod.jobs-cloud-common.CloudInstanceUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:04 [info] start (action_name: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer, action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:04 [debug] Discover table 'modw_cloud.instance_type' 2022-04-05 15:08:04 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:04 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/instance_type.json 2022-04-05 15:08:04 [debug] Parsed 2 records 2022-04-05 15:08:04 [debug] Requested 10 record fields: resource_id, instance_type_id, instance_type, display, description, num_cores, memory_mb, disk_gb, start_time, end_time 2022-04-05 15:08:04 [debug] No destination_field_map specified 2022-04-05 15:08:04 [debug] Auto-generating destination_field_map using 10 source fields: resource_id, instance_type_id, instance_type, display, description, num_cores, memory_mb, disk_gb, start_time, end_time 2022-04-05 15:08:04 [debug] Available fields for table key 'instance_type': resource_id, instance_type_id, instance_type, display, description, num_cores, memory_mb, disk_gb, start_time, end_time 2022-04-05 15:08:04 [debug] Generated destination_field_map: Table: instance_type resource_id -> resource_id instance_type_id -> instance_type_id instance_type -> instance_type display -> display description -> description num_cores -> num_cores memory_mb -> memory_mb disk_gb -> disk_gb start_time -> start_time end_time -> end_time 2022-04-05 15:08:04 [debug] Processing destination field map 2022-04-05 15:08:04 [debug] Insert SQL for table key 'instance_type': INSERT INTO modw_cloud.instance_type (resource_id, instance_type_id, instance_type, display, description, num_cores, memory_mb, disk_gb, start_time, end_time) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), instance_type_id = COALESCE(VALUES(instance_type_id), instance_type_id), instance_type = COALESCE(VALUES(instance_type), instance_type), display = COALESCE(VALUES(display), display), description = COALESCE(VALUES(description), description), num_cores = COALESCE(VALUES(num_cores), num_cores), memory_mb = COALESCE(VALUES(memory_mb), memory_mb), disk_gb = COALESCE(VALUES(disk_gb), disk_gb), start_time = COALESCE(VALUES(start_time), start_time), end_time = COALESCE(VALUES(end_time), end_time) 2022-04-05 15:08:04 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:04 [debug] ANALYZE TABLE modw_cloud.instance_type 2022-04-05 15:08:04 [debug] Completed in 0.014460s 2022-04-05 15:08:04 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 0.07 s) 2022-04-05 15:08:04 [notice] (action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171284.1768, end_time: 1649171284.2472, elapsed_time: 0.07037, records_examined: 1, records_loaded: 1) 2022-04-05 15:08:04 [info] end (action_name: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer, action: xdmod.jobs-cloud-common.CloudInstanceTypeUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:04 [info] start (action_name: xdmod.jobs-cloud-common.CloudImageUnknownInitializer, action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:04 [debug] Discover table 'modw_cloud.image' 2022-04-05 15:08:04 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:04 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/image.json 2022-04-05 15:08:04 [debug] Parsed 2 records 2022-04-05 15:08:04 [debug] Requested 3 record fields: resource_id, image_id, image 2022-04-05 15:08:04 [debug] No destination_field_map specified 2022-04-05 15:08:04 [debug] Auto-generating destination_field_map using 3 source fields: resource_id, image_id, image 2022-04-05 15:08:04 [debug] Available fields for table key 'image': resource_id, image_id, image 2022-04-05 15:08:04 [debug] Generated destination_field_map: Table: image resource_id -> resource_id image_id -> image_id image -> image 2022-04-05 15:08:04 [debug] Processing destination field map 2022-04-05 15:08:04 [debug] Insert SQL for table key 'image': INSERT INTO modw_cloud.image (resource_id, image_id, image) VALUES (?, ?, ?) ON DUPLICATE KEY UPDATE resource_id = COALESCE(VALUES(resource_id), resource_id), image_id = COALESCE(VALUES(image_id), image_id), image = COALESCE(VALUES(image), image) 2022-04-05 15:08:04 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:04 [debug] ANALYZE TABLE modw_cloud.image 2022-04-05 15:08:04 [debug] Completed in 0.052820s 2022-04-05 15:08:04 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 1 records (Time Taken: 0.19 s) 2022-04-05 15:08:04 [notice] (action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171284.3144, end_time: 1649171284.5014, elapsed_time: 0.18695, records_examined: 1, records_loaded: 1) 2022-04-05 15:08:04 [info] end (action_name: xdmod.jobs-cloud-common.CloudImageUnknownInitializer, action: xdmod.jobs-cloud-common.CloudImageUnknownInitializer (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:04 [info] start (action_name: xdmod.jobs-cloud-common.MemoryBucketGenerator, action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:04 [debug] Discover table 'modw_cloud.memory_buckets' 2022-04-05 15:08:04 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:04 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/memory_buckets.json 2022-04-05 15:08:04 [debug] Parsed 9 records 2022-04-05 15:08:04 [debug] Requested 4 record fields: id, min_memory, max_memory, description 2022-04-05 15:08:04 [debug] No destination_field_map specified 2022-04-05 15:08:04 [debug] Auto-generating destination_field_map using 4 source fields: id, min_memory, max_memory, description 2022-04-05 15:08:04 [debug] Available fields for table key 'memory_buckets': id, min_memory, max_memory, description 2022-04-05 15:08:04 [debug] Generated destination_field_map: Table: memory_buckets id -> id min_memory -> min_memory max_memory -> max_memory description -> description 2022-04-05 15:08:04 [debug] Processing destination field map 2022-04-05 15:08:04 [debug] Insert SQL for table key 'memory_buckets': INSERT INTO modw_cloud.memory_buckets (id, min_memory, max_memory, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE id = COALESCE(VALUES(id), id), min_memory = COALESCE(VALUES(min_memory), min_memory), max_memory = COALESCE(VALUES(max_memory), max_memory), description = COALESCE(VALUES(description), description) 2022-04-05 15:08:04 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:04 [debug] ANALYZE TABLE modw_cloud.memory_buckets 2022-04-05 15:08:04 [debug] Completed in 0.016710s 2022-04-05 15:08:04 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 8 records (Time Taken: 0.08 s) 2022-04-05 15:08:04 [notice] (action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171284.5772, end_time: 1649171284.6549, elapsed_time: 0.07777, records_examined: 8, records_loaded: 8) 2022-04-05 15:08:04 [info] end (action_name: xdmod.jobs-cloud-common.MemoryBucketGenerator, action: xdmod.jobs-cloud-common.MemoryBucketGenerator (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:04 [info] start (action_name: xdmod.jobs-cloud-common.ProcessorBucketGenerator, action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:04 [debug] Discover table 'modw_cloud.processor_buckets' 2022-04-05 15:08:04 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:04 [debug] Available Variables: DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl_data.d/cloud_common/processor_buckets.json 2022-04-05 15:08:04 [debug] Parsed 8 records 2022-04-05 15:08:04 [debug] Requested 4 record fields: id, min_processors, max_processors, description 2022-04-05 15:08:04 [debug] No destination_field_map specified 2022-04-05 15:08:04 [debug] Auto-generating destination_field_map using 4 source fields: id, min_processors, max_processors, description 2022-04-05 15:08:04 [debug] Available fields for table key 'processor_buckets': id, min_processors, max_processors, description 2022-04-05 15:08:04 [debug] Generated destination_field_map: Table: processor_buckets id -> id min_processors -> min_processors max_processors -> max_processors description -> description 2022-04-05 15:08:04 [debug] Processing destination field map 2022-04-05 15:08:04 [debug] Insert SQL for table key 'processor_buckets': INSERT INTO modw_cloud.processor_buckets (id, min_processors, max_processors, description) VALUES (?, ?, ?, ?) ON DUPLICATE KEY UPDATE id = COALESCE(VALUES(id), id), min_processors = COALESCE(VALUES(min_processors), min_processors), max_processors = COALESCE(VALUES(max_processors), max_processors), description = COALESCE(VALUES(description), description) 2022-04-05 15:08:04 [info] Execute Post-execute tasks: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:04 [debug] ANALYZE TABLE modw_cloud.processor_buckets 2022-04-05 15:08:04 [debug] Completed in 0.015200s 2022-04-05 15:08:04 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 7 records (Time Taken: 0.08 s) 2022-04-05 15:08:04 [notice] (action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171284.7222, end_time: 1649171284.7984, elapsed_time: 0.07626, records_examined: 7, records_loaded: 7) 2022-04-05 15:08:04 [info] end (action_name: xdmod.jobs-cloud-common.ProcessorBucketGenerator, action: xdmod.jobs-cloud-common.ProcessorBucketGenerator (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:04 [notice] Finished processing section 'xdmod.jobs-cloud-common' 2022-04-05 15:08:04 [info] Releasing lock file '/tmp/etlv2_34' 2022-04-05 15:08:04 [debug] Running ETL pipeline "jobs-cloud-ingest-openstack" with parameters {"include-only-resource-codes":"Cloud","variable-overrides":{"CLOUD_EVENT_LOG_DIRECTORY":"\/root\/test\/openstack"}} 2022-04-05 15:08:04 [debug] Loading configuration file /etc/xdmod/etl/etl.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-import.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/acls-import.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/acls-xdmod-management.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/action_state_setup.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/action_state_setup.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_ingest_resource_specs.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [info] ETL\DataEndpoint\DirectoryScanner (name=Open Stack resource specifications, path=${CLOUD_RESOURCE_SPECS_DIRECTORY}): Relative path provided, absolute path recommended 2022-04-05 15:08:04 [debug] Qualifying relative path ${CLOUD_RESOURCE_SPECS_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/cloud_state_machine.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/gateways.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/gateways.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb-xdw.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/hpcdb.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/hpcdb.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/ingest_resources.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/ingest_resources.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/jobs.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-04-05 15:08:04 [debug] (Configuration\JsonReferenceTransformer) Resolved reference 'etl_pipelines.d/jobs-xdw.json' to '/etc/xdmod/etl/etl_pipelines.d/jobs-xdw.json' 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_common.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_generic.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_cloud_openstack.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/jobs_common.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/jobs_common.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/organizations.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/organizations.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/resource_types.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/resource_types.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/shredder.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/shredder.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/staging.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/staging.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [info] ETL\DataEndpoint\DirectoryScanner (name=usage-directory, path=${STORAGE_LOG_DIRECTORY}): Relative path provided, absolute path recommended 2022-04-05 15:08:04 [debug] Qualifying relative path ${STORAGE_LOG_DIRECTORY} with /etc/xdmod/etl/etl_data.d 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/storage.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/storage.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:04 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/test_suite.json 2022-04-05 15:08:04 [debug] Parsing /etc/xdmod/etl/etl.d/test_suite.json 2022-04-05 15:08:04 [debug] Parsed 1 records 2022-04-05 15:08:05 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/verify.json 2022-04-05 15:08:05 [debug] Parsing /etc/xdmod/etl/etl.d/verify.json 2022-04-05 15:08:05 [debug] Parsed 1 records 2022-04-05 15:08:05 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdb.json 2022-04-05 15:08:05 [debug] Parsing /etc/xdmod/etl/etl.d/xdb.json 2022-04-05 15:08:05 [debug] Parsed 1 records 2022-04-05 15:08:05 [debug] Loading local configuration file /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-04-05 15:08:05 [debug] Parsing /etc/xdmod/etl/etl.d/xdmod-migration-9_5_0-10_0_0.json 2022-04-05 15:08:05 [debug] Parsed 1 records 2022-04-05 15:08:05 [debug] Stored object ETL\Configuration\EtlConfiguration (/etc/xdmod/etl/etl.json) in APCu cache with key ETL\Configuration\EtlConfiguration|/etc/xdmod/etl/etl.json|fb0821326dcaf0a4f7340c8d000954c1 in 0.121142s 2022-04-05 15:08:05 [debug] Running ETL pipeline with script options {"default-module-name":"xdmod","process-sections":["jobs-cloud-ingest-openstack"],"include-only-resource-codes":"Cloud","variable-overrides":{"CLOUD_EVENT_LOG_DIRECTORY":"\/root\/test\/openstack"}} 2022-04-05 15:08:05 [debug] Loading resource code to id map ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod): SELECT id, code from modw.resourcefact 2022-04-05 15:08:05 [info] Verifying endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:05 [info] Verifying endpoint: ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=/root/test/openstack, handler=jsonfile) 2022-04-05 15:08:05 [info] Verifying endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:05 [info] Create action xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:08:05 [debug] Loading configuration file /etc/xdmod/etl/etl_action_defs.d/cloud_openstack/raw_cloud_job_logs.json 2022-04-05 15:08:05 [debug] Parsing /etc/xdmod/etl/etl_action_defs.d/cloud_openstack/raw_cloud_job_logs.json 2022-04-05 15:08:05 [debug] Parsed 1 records 2022-04-05 15:08:05 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json' to '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_event.json' 2022-04-05 15:08:05 [debug] (Configuration\JsonReferenceTransformer) Resolved reference '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json' to '/etc/xdmod/etl/etl_tables.d/cloud_openstack/raw_instance_type.json' 2022-04-05 15:08:05 [debug] Stored object Configuration\Configuration (/etc/xdmod/etl/etl_action_defs.d/cloud_openstack/raw_cloud_job_logs.json) in APCu cache with key Configuration\Configuration|/etc/xdmod/etl/etl_action_defs.d/cloud_openstack/raw_cloud_joblogs.json|080e8f8664a3addcdf68a2660e1a6e34 in 0.003704s 2022-04-05 15:08:05 [info] Verifying action: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor) 2022-04-05 15:08:05 [info] Utility endpoint: ('Utility DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:05 [info] Source endpoint: ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=/root/test/openstack, handler=jsonfile) 2022-04-05 15:08:05 [debug] Connecting directory scanner to /root/test/openstack 2022-04-05 15:08:05 [debug] Set max recursion depth: 1 2022-04-05 15:08:05 [info] Applying pattern filters: (directory: null, file: /[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.json/) 2022-04-05 15:08:05 [info] Destination endpoint: ('Cloud DB', class=ETL\DataEndpoint\Mysql, config=datawarehouse, schema=modw_cloud, host=mariadb:3306, user=xdmod) 2022-04-05 15:08:05 [debug] Created ETL destination table object for table definition key 'openstack_raw_event' 2022-04-05 15:08:05 [debug] Created ETL destination table object for table definition key 'openstack_raw_instance_type' 2022-04-05 15:08:05 [info] Empty lock directory specified, using temp directory: /tmp 2022-04-05 15:08:05 [info] Obtaining lock file '/tmp/etlv2_34' 2022-04-05 15:08:05 [notice] Start processing section 'xdmod.jobs-cloud-ingest-openstack' 2022-04-05 15:08:05 [info] start (action_name: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor, action: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor), start_date: , end_date: ) 2022-04-05 15:08:05 [debug] Discover table 'modw_cloud.openstack_raw_event' 2022-04-05 15:08:05 [debug] Discover table 'modw_cloud.openstack_raw_instance_type' 2022-04-05 15:08:05 [info] Process date interval (1/1) (start: none, end: none) 2022-04-05 15:08:05 [debug] Available Variables: CLOUD_EVENT_LOG_DIRECTORY='/root/test/openstack', DESTINATION_SCHEMA='modw_cloud', DW_ETL_LOG_RECIPIENT='robbaron@bu.edu', RESOURCE='Cloud', RESOURCE_ID='1', TIMEZONE='UTC', UTILITY_SCHEMA='modw', action_definition_dir='/etc/xdmod/etl/etl_action_defs.d', base_dir='/etc/xdmod/etl', data_dir='/etc/xdmod/etl/etl_data.d', local_config_dir='/etc/xdmod/etl/etl.d', macro_dir='/etc/xdmod/etl/etl_macros.d', schema_dir='/etc/xdmod/etl/etl_schemas.d', sql_dir='/etc/xdmod/etl/etl_sql.d', table_definition_dir='/etc/xdmod/etl/etl_tables.d' 2022-04-05 15:08:05 [info] Source endpoint ETL\DataEndpoint\DirectoryScanner (name=Open Stack event logs, path=/root/test/openstack, handler=jsonfile) returned 0 records, skipping. 2022-04-05 15:08:05 [info] ETL\Ingestor\StructuredFileIngestor: Rows Processed: 0 records (Time Taken: 0.06 s) 2022-04-05 15:08:05 [notice] (action: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor), start_time: 1649171285.2501, end_time: 1649171285.3097, elapsed_time: 0.0596, records_examined: 0, records_loaded: 0) 2022-04-05 15:08:05 [info] end (action_name: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor, action: xdmod.jobs-cloud-ingest-openstack.OpenStackRawCloudEventIngestor (ETL\Ingestor\StructuredFileIngestor)) 2022-04-05 15:08:05 [notice] Finished processing section 'xdmod.jobs-cloud-ingest-openstack' 2022-04-05 15:08:05 [info] Releasing lock file '/tmp/etlv2_34' 2022-04-05 15:08:05 [notice] Done shredding files 2022-04-05 15:08:05 [notice] xdmod-shredder end (process_end_time: 2022-04-05 15:08:05)

The file is here:

sh-4.2$ pwd /root/test/openstack sh-4.2$ ls -la total 2324 drwxr-xr-x. 2 root root 58 Apr 1 15:32 . drwxr-xr-x. 3 root root 23 Apr 1 15:32 .. -rw-r--r--. 1 root root 2379484 Mar 24 16:17 2018-04-17T00-00-00_2018-04-30T23-59-59.json

And mariadb is:

sh-4.2$ mysql -h mariadb --protocol=tcp -u root -p Enter password: Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 1011 Server version: 10.3.34-MariaDB-1:10.3.34+maria~focal mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select * from modw_cloud.openstack_staging_event; Empty set (0.00 sec)

MariaDB [(none)]> select * from modw_cloud.event; Empty set (0.00 sec)

MariaDB [(none)]> select * from modw_cloud.session_records; Empty set (0.00 sec)

MariaDB [(none)]>

rob-baron commented 1 year ago

Gregary Dean , said over 1 year ago Ticket: https://help.xdmod.org/support/tickets/29597

Hi Robert,

I think your filename is in the wrong format. There should be colons between the hours, seconds, and minutes instead of dashes. Here's what the filename should be, 2018-04-17T00:00:00_2018-04-30T23:59:59.json.

-greg Robert Bartlett Baron , said over 1 year ago The Data is getting in to those tables, but is still not viewable though the interface.

MariaDB [(none)]> select count() from modw_cloud.openstack_staging_event; +----------+ | count() | +----------+ | 1827 | +----------+ 1 row in set (0.00 sec)

MariaDB [(none)]> select count() from modw_cloud.event; +----------+ | count() | +----------+ | 1827 | +----------+ 1 row in set (0.01 sec)

MariaDB [(none)]> select count() from modw_cloud.session_records; +----------+ | count() | +----------+ | 261 | +----------+ 1 row in set (0.00 sec)

Robert Bartlett Baron , said over 1 year ago Ok, now the data is being displayed in the interface.

Robert Bartlett Baron , said over 1 year ago Sorry, I was being bone headed about the specific format of the filename. Interesting design choice. Gregary Dean , said over 1 year ago Ticket: https://help.xdmod.org/support/tickets/29597

Hi Robert,

Great! I'm glad to hear that you're able to see the data.

-greg

Robert Bartlett Baron , said over 1 year ago We can close this ticket. This ticket has been Closed | over 1 year ago Reply