I'm pretty sure this has to do with an organization not existing. When you run xdmod-setup there is an option for setting up your organization in XDMoD, did you do that? It is option number 3 in the xdmod-setup menu. If you did do this step, can you send me the contents of your organization.json file? If you installed XDMoD via the RPM it should be in /etc/xdmod/organization.json. can you also send me an export of the mod_hpcdb.hpcdb_organizations table.
Thanks,
Greg Dean
Robert Bartlett Baron , said over 1 year ago
organization.json
{
"name": "moc",
"abbrev": "moc"
}
select * from hpcdb_organizations;
Empty set (0.00 sec)
After running that command you can check the hpcdb_organizations table again and there should be a row with your organization. If your organization is in that table, you should be able to run your original command without getting that error.
-greg
Robert Bartlett Baron , said over 1 year ago
Now I get the following when I run:
/usr/share/xdmod/tools/etl/etl_overseer.php -p ingest-resources
2022-03-07 18:32:59 [notice] dw_extract_transform_load start (process_start_time: 2022-03-07 18:32:59)
2022-03-07 18:32:59 [notice] Start processing section 'xdmod.ingest-resources'
2022-03-07 18:32:59 [notice] (action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1646677979.3068, end_time: 1646677979.5894, elapsed_time: 0.28256, records_examined: 1, records_loaded: 1)
2022-03-07 18:32:59 [notice] (action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_time: 1646677979.5999, end_time: 1646677979.6298, elapsed_time: 0.02995, records_examined: 0, records_loaded: 0)
2022-03-07 18:32:59 [error] {"message":"xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor): SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINT fk_resource_res_type_upd FOREIGN KEY (resource_type_id) REFERENCES hpcdb_resource_types (type_id) ON DELETE CASCADE ON UPDATE CASCADE) Exception: 'SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINT fk_resource_res_type_upd FOREIGN KEY (resource_type_id) REFERENCES hpcdb_resource_types (type_id) ON DELETE CASCADE ON UPDATE CASCADE)'"}
2022-03-07 18:32:59 [warning] Stopping ETL due to exception in xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor)
2022-03-07 18:32:59 [error] xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor): SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINT fk_resource_res_type_upd FOREIGN KEY (resource_type_id) REFERENCES hpcdb_resource_types (type_id) ON DELETE CASCADE ON UPDATE CASCADE) Exception: 'SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINT fk_resource_res_type_upd FOREIGN KEY (resource_type_id) REFERENCES hpcdb_resource_types (type_id) ON DELETE CASCADE ON UPDATE CASCADE)'
I am assuming that this is using the following json files:
/etc/xdmod/resources.json
[
{
"resource": "Cloud",
"resource_type": "Cloud",
"name": "kaizen"
}
]
Robert Bartlett Baron, reported over 1 year ago Here is what I am seeing:
/usr/share/xdmod/tools/etl/etl_overseer.php -p ingest-resources
2022-03-01 19:25:50 [notice] dw_extract_transform_load start (process_start_time: 2022-03-01 19:25:50) 2022-03-01 19:25:51 [notice] Start processing section 'xdmod.ingest-resources' 2022-03-01 19:25:51 [notice] (action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1646162751.0614, end_time: 1646162751.1048, elapsed_time: 0.04343, records_examined: 1, records_loaded: 1) 2022-03-01 19:25:51 [notice] (action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_time: 1646162751.1136, end_time: 1646162751.1432, elapsed_time: 0.02958, records_examined: 0, records_loaded: 0) 2022-03-01 19:25:51 [error] {"message":"xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor): SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (
mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_org_resou_organizaFOREIGN KEY (organization_id) REFERENCEShpcdb_organizations(organization_id)) Exception: 'SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_org_resou_organizaFOREIGN KEY (organization_id) REFERENCEShpcdb_organizations(organization_id))'"} 2022-03-01 19:25:51 [warning] Stopping ETL due to exception in xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor) 2022-03-01 19:25:51 [error] xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor): SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_org_resou_organizaFOREIGN KEY (organization_id) REFERENCEShpcdb_organizations(organization_id)) Exception: 'SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_org_resou_organizaFOREIGN KEY (organization_id) REFERENCEShpcdb_organizations(organization_id))'0 /usr/share/xdmod/classes/ETL/Ingestor/pdoIngestor.php(544): CCR\Loggable->logAndThrowException('SQLSTATE[23000]...', Array)
1 /usr/share/xdmod/classes/ETL/Ingestor/pdoIngestor.php(459): ETL\Ingestor\pdoIngestor->singleDatabaseIngest()
2 /usr/share/xdmod/classes/ETL/Ingestor/aIngestor.php(126): ETL\Ingestor\pdoIngestor->_execute()
3 /usr/share/xdmod/classes/ETL/EtlOverseer.php(473): ETL\Ingestor\aIngestor->execute(Object(ETL\EtlOverseerOptions))
4 /usr/share/xdmod/classes/ETL/EtlOverseer.php(435): ETL\EtlOverseer->_execute('xdmod.ingest-re...', Object(ETL\Ingestor\DatabaseIngestor))
5 /usr/share/xdmod/tools/etl/etl_overseer.php(608): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))
6 {main}
xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor): SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (
mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_org_resou_organizaFOREIGN KEY (organization_id) REFERENCEShpcdb_organizations(organization_id)) Exception: 'SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_org_resou_organizaFOREIGN KEY (organization_id) REFERENCEShpcdb_organizations(organization_id))'0 /usr/share/xdmod/classes/ETL/Ingestor/pdoIngestor.php(544): CCR\Loggable->logAndThrowException('SQLSTATE[23000]...', Array)
1 /usr/share/xdmod/classes/ETL/Ingestor/pdoIngestor.php(459): ETL\Ingestor\pdoIngestor->singleDatabaseIngest()
2 /usr/share/xdmod/classes/ETL/Ingestor/aIngestor.php(126): ETL\Ingestor\pdoIngestor->_execute()
3 /usr/share/xdmod/classes/ETL/EtlOverseer.php(473): ETL\Ingestor\aIngestor->execute(Object(ETL\EtlOverseerOptions))
4 /usr/share/xdmod/classes/ETL/EtlOverseer.php(435): ETL\EtlOverseer->_execute('xdmod.ingest-re...', Object(ETL\Ingestor\DatabaseIngestor))
5 /usr/share/xdmod/tools/etl/etl_overseer.php(608): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))
6 {main}
Any suggestions as to how to move forward?
Gregary Dean , said over 1 year ago Ticket: https://help.xdmod.org/support/tickets/29490
Hi Robert,
I'm pretty sure this has to do with an organization not existing. When you run xdmod-setup there is an option for setting up your organization in XDMoD, did you do that? It is option number 3 in the xdmod-setup menu. If you did do this step, can you send me the contents of your organization.json file? If you installed XDMoD via the RPM it should be in /etc/xdmod/organization.json. can you also send me an export of the mod_hpcdb.hpcdb_organizations table.
Thanks, Greg Dean Robert Bartlett Baron , said over 1 year ago
organization.json
{
}
select * from hpcdb_organizations; Empty set (0.00 sec)
Gregary Dean , said over 1 year ago Ticket: https://help.xdmod.org/support/tickets/29490
Hi Robert,
Thanks for the information. The following command should work for fixing your issue.
/usr/share/xdmod/tools/etl/etl_overseer.php -p ingest-organizations
After running that command you can check the hpcdb_organizations table again and there should be a row with your organization. If your organization is in that table, you should be able to run your original command without getting that error.
-greg Robert Bartlett Baron , said over 1 year ago Now I get the following when I run:
/usr/share/xdmod/tools/etl/etl_overseer.php -p ingest-resources 2022-03-07 18:32:59 [notice] dw_extract_transform_load start (process_start_time: 2022-03-07 18:32:59) 2022-03-07 18:32:59 [notice] Start processing section 'xdmod.ingest-resources' 2022-03-07 18:32:59 [notice] (action: xdmod.ingest-resources.IngestResourceConfig (ETL\Ingestor\StructuredFileIngestor), start_time: 1646677979.3068, end_time: 1646677979.5894, elapsed_time: 0.28256, records_examined: 1, records_loaded: 1) 2022-03-07 18:32:59 [notice] (action: xdmod.ingest-resources.IngestResourcesStaging (ETL\Ingestor\DatabaseIngestor), start_time: 1646677979.5999, end_time: 1646677979.6298, elapsed_time: 0.02995, records_examined: 0, records_loaded: 0) 2022-03-07 18:32:59 [error] {"message":"xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor): SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (
mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_res_type_updFOREIGN KEY (resource_type_id) REFERENCEShpcdb_resource_types(type_id) ON DELETE CASCADE ON UPDATE CASCADE) Exception: 'SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_res_type_updFOREIGN KEY (resource_type_id) REFERENCEShpcdb_resource_types(type_id) ON DELETE CASCADE ON UPDATE CASCADE)'"} 2022-03-07 18:32:59 [warning] Stopping ETL due to exception in xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor) 2022-03-07 18:32:59 [error] xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor): SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_res_type_updFOREIGN KEY (resource_type_id) REFERENCEShpcdb_resource_types(type_id) ON DELETE CASCADE ON UPDATE CASCADE) Exception: 'SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_res_type_updFOREIGN KEY (resource_type_id) REFERENCEShpcdb_resource_types(type_id) ON DELETE CASCADE ON UPDATE CASCADE)'0 /usr/share/xdmod/classes/ETL/Ingestor/pdoIngestor.php(544): CCR\Loggable->logAndThrowException('SQLSTATE[23000]...', Array)
1 /usr/share/xdmod/classes/ETL/Ingestor/pdoIngestor.php(459): ETL\Ingestor\pdoIngestor->singleDatabaseIngest()
2 /usr/share/xdmod/classes/ETL/Ingestor/aIngestor.php(126): ETL\Ingestor\pdoIngestor->_execute()
3 /usr/share/xdmod/classes/ETL/EtlOverseer.php(473): ETL\Ingestor\aIngestor->execute(Object(ETL\EtlOverseerOptions))
4 /usr/share/xdmod/classes/ETL/EtlOverseer.php(435): ETL\EtlOverseer->_execute('xdmod.ingest-re...', Object(ETL\Ingestor\DatabaseIngestor))
5 /usr/share/xdmod/tools/etl/etl_overseer.php(608): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))
6 {main}
xdmod.ingest-resources.HpcdbIngestResources (ETL\Ingestor\DatabaseIngestor): SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (
mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_res_type_updFOREIGN KEY (resource_type_id) REFERENCEShpcdb_resource_types(type_id) ON DELETE CASCADE ON UPDATE CASCADE) Exception: 'SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (mod_hpcdb.hpcdb_resources, CONSTRAINTfk_resource_res_type_updFOREIGN KEY (resource_type_id) REFERENCEShpcdb_resource_types(type_id) ON DELETE CASCADE ON UPDATE CASCADE)'0 /usr/share/xdmod/classes/ETL/Ingestor/pdoIngestor.php(544): CCR\Loggable->logAndThrowException('SQLSTATE[23000]...', Array)
1 /usr/share/xdmod/classes/ETL/Ingestor/pdoIngestor.php(459): ETL\Ingestor\pdoIngestor->singleDatabaseIngest()
2 /usr/share/xdmod/classes/ETL/Ingestor/aIngestor.php(126): ETL\Ingestor\pdoIngestor->_execute()
3 /usr/share/xdmod/classes/ETL/EtlOverseer.php(473): ETL\Ingestor\aIngestor->execute(Object(ETL\EtlOverseerOptions))
4 /usr/share/xdmod/classes/ETL/EtlOverseer.php(435): ETL\EtlOverseer->_execute('xdmod.ingest-re...', Object(ETL\Ingestor\DatabaseIngestor))
5 /usr/share/xdmod/tools/etl/etl_overseer.php(608): ETL\EtlOverseer->execute(Object(ETL\Configuration\EtlConfiguration))
6 {main}
I am assuming that this is using the following json files: /etc/xdmod/resources.json [ { "resource": "Cloud", "resource_type": "Cloud", "name": "kaizen" } ]
/etc/xdmod/resource_types.json { "resource_types": { "HPC": { "description": "High-performance computing", "realms": [ "Jobs" ] }, "HTC": { "description": "High-throughput computing", "realms": [ "Jobs" ] }, "DIC": { "description": "Data-intensive computing", "realms": [ "Jobs" ] }, "Grid": { "description": "Grid of resources", "realms": [ ] }, "Cloud": { "description": "Cloud resource", "realms": [ "Cloud" ] }, "Vis": { "description": "Visualization system", "realms": [ "Jobs" ] }, "VM": { "description": "Virtual Machine system", "realms": [ "Cloud" ] }, "Tape": { "description": "Tape storage resource", "realms": [ "Storage" ] }, "Disk": { "description": "Disk storage resource", "realms": [ "Storage" ] }, "StGrid": { "description": "Storage grid", "realms": [ "Storage" ] }, "US": { "description": "User Support", "realms": [ ] }, "Gateway": { "description": "Web-based access to CI resources", "realms": [ ] } } }
/etc/xdmo/resource_specs.json [ { "resource": "Cloud", "processors": 1056, "nodes": 26, "ppn": 40 } ]
Any suggestions? Gregary Dean , said over 1 year ago Ticket: https://help.xdmod.org/support/tickets/29490
Hi Robert, There is another pipeline to run as well that I forgot to include in my last message.
/usr/share/xdmod/tools/etl/etl_overseer.php -p ingest-resource-types Sorry about that. -greg
Robert Bartlett Baron , said over 1 year ago This part worked. Thank you.
This ticket has been Closed | over 1 year ago Reply