Fairbench pipeline output is empty

[elmokhtar-mohamedmoussa-idnow]

Hello, I am following this tutorial to run the fairbench pipeline in the toolkit. However, when I complete all the steps the run output doesn't appear. I get a screen with the message : There are no runs to display.

I attached an image to show my screen after completing all the steps :

[sudhang]

I have the same issue. I found a "Results" button at the top right, which opens a tab but without any content.

Also, a few combinations of parameters leads to a blank page in the "Bias Metric" stage

[sudhang]

EDIT: I should also add, I am on fedora and use podman which is aliased to docker. I was suspecting that this might have been the source of my problem until I saw this issue. In case someone has succeeded in running the fairbench output, please let me know; I'll then try with docker instead of podman.

Also, on fedora, I had to turn off selinux as well.

[georgiosn]

Thank you for your comments we work on them! There is also an issue with the results button as you said.

We also checked the comment about selinux, it seems that if you use the toolkit under native linux you may have to add a rule to the firewall for the services in docker to be able to access the KFP.

Please check that you have installed KFP using the latest instructions.

If by going to http://kfp.local.exus.ai:8082/ you can see the KFP, you have used the latest instructions

You can try then then following command:

sudo ufw allow in to 172.0.0.0/8 proto tcp port 8082 comment 'host.docker.internal access'

[sudhang]

Thank you for the response. ufw is inactive for me, so I don't believe I need to set those rules. That KFP url does work for me though.

[georgiosn]

Hello again, a new version of the backend configuration engine of the toolkit uploaded. Please git pull the latest code and run it. During the wizard when you reach the last step after clicking on the run button if you get an error please run the following command in a terminal and send the logs here with a message.

docker logs mammoth_kk-api-1

Thanks!

[elmokhtar-mohamedmoussa-idnow]

Hello, Thanks for the help, now I have a 500 error with the following logs:

172.21.0.6:53728 - "GET /wizard/databias/metric/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:00,901 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:00,901 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:00,901 INFO sqlalchemy.engine.Engine [cached since 7298s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
2024-10-14 13:47:00,903 INFO sqlalchemy.engine.Engine ROLLBACK
{'result_id': 'STATUS_OK', 'result_message': 'OK', 'is_error': False, 'status_code': None, 'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'run_status': None, 'run_id': None, 'run_type': 'MODEL_BIAS', 'run_start_time': None, 'run_end_time': None, 'attributes': ['marital'], 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'loader_data': {'id': 'loader_csvdataset', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'metrics': None, 'url_data': '', 'url_model': '', 'run_artifacts': None, 'step': 3}
2024-10-14 13:47:00,904 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:00,904 INFO sqlalchemy.engine.Engine SELECT domain.id, domain.short_id, domain.name 
FROM domain 
WHERE domain.short_id = %(short_id_1)s ORDER BY domain.name
2024-10-14 13:47:00,905 INFO sqlalchemy.engine.Engine [cached since 8.602s ago] {'short_id_1': 'fairbench_initial'}
2024-10-14 13:47:00,906 INFO sqlalchemy.engine.Engine SELECT attributes.id AS attributes_id, attributes.attribute AS attributes_attribute, attributes.domain_id AS attributes_domain_id 
FROM attributes 
WHERE %(param_1)s = attributes.domain_id
2024-10-14 13:47:00,906 INFO sqlalchemy.engine.Engine [cached since 8.601s ago] {'param_1': 1}
2024-10-14 13:47:00,907 INFO sqlalchemy.engine.Engine ROLLBACK
['marital']
172.21.0.6:53728 - "GET /wizard/databias/attributes/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:02,606 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:02,607 INFO sqlalchemy.engine.Engine SELECT "user".id, "user".username, "user".user_auth_id, "user".email 
FROM "user" 
WHERE "user".username = %(username_1)s
2024-10-14 13:47:02,607 INFO sqlalchemy.engine.Engine [cached since 28.79s ago] {'username_1': 'test test'}
2024-10-14 13:47:02,608 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:02,608 INFO sqlalchemy.engine.Engine [cached since 7300s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
Existing json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 3, 'loader_data': {'id': 'loader_csvdataset', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'attributes': ['marital']}
Response json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'attributes': ['marital'], 'step': 3}
Updated json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 3, 'loader_data': {'id': 'loader_csvdataset', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'attributes': ['marital']}
2024-10-14 13:47:02,609 INFO sqlalchemy.engine.Engine UPDATE wizard_options SET configuration=%(configuration)s WHERE wizard_options.id = %(wizard_options_id)s
2024-10-14 13:47:02,610 INFO sqlalchemy.engine.Engine [cached since 10.35s ago] {'configuration': '{"uuid": "861ef2e8-c687-47d8-95ca-eda9b71e1a26", "name": "test", "group": "test", "loader_model": {"id": "model_onnx", "parameters_value": {"path": " ... (336 characters truncated) ... "on_bad_lines": "skip", "path": "http://host.k3d.internal:5000/bank.csv"}}, "domain": "fairbench_initial", "url_data": "", "attributes": ["marital"]}', 'wizard_options_id': 3}
2024-10-14 13:47:02,610 INFO sqlalchemy.engine.Engine COMMIT
172.21.0.6:53728 - "POST /wizard/store/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:02,627 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:02,628 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:02,628 INFO sqlalchemy.engine.Engine [cached since 7300s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
2024-10-14 13:47:02,629 INFO sqlalchemy.engine.Engine ROLLBACK
{'result_id': 'STATUS_OK', 'result_message': 'OK', 'is_error': False, 'status_code': None, 'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'run_status': None, 'run_id': None, 'run_type': 'MODEL_BIAS', 'run_start_time': None, 'run_end_time': None, 'attributes': ['marital'], 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'loader_data': {'id': 'loader_csvdataset', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'metrics': None, 'url_data': '', 'url_model': '', 'run_artifacts': None, 'step': 3}
2024-10-14 13:47:02,630 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:02,631 INFO sqlalchemy.engine.Engine SELECT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.type = %(type_1)s AND in_out_types.type = %(type_2)s ORDER BY component.name
2024-10-14 13:47:02,631 INFO sqlalchemy.engine.Engine [cached since 28.77s ago] {'type_1': 'METRIC', 'type_2': 'IN'}
<sqlalchemy.engine.result.ChunkedIteratorResult object at 0x7de274cf6090>
2024-10-14 13:47:02,633 INFO sqlalchemy.engine.Engine SELECT DISTINCT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.short_id = %(short_id_1)s ORDER BY component.name
2024-10-14 13:47:02,634 INFO sqlalchemy.engine.Engine [cached since 28.76s ago] {'short_id_1': 'loader_csvdataset'}
(<src.dbops.Component object at 0x7de2754bf050>,)
2024-10-14 13:47:02,636 INFO sqlalchemy.engine.Engine SELECT DISTINCT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.short_id = %(short_id_1)s ORDER BY component.name
2024-10-14 13:47:02,636 INFO sqlalchemy.engine.Engine [cached since 28.77s ago] {'short_id_1': 'model_onnx'}
(<src.dbops.Component object at 0x7de2754bf170>,)
2024-10-14 13:47:02,638 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:02,638 INFO sqlalchemy.engine.Engine [cached since 7300s ago] {'param_1': 10}
2024-10-14 13:47:02,641 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:02,641 INFO sqlalchemy.engine.Engine [cached since 7300s ago] {'param_1': 1}
Input types: ['ONNX', 'CSVDataset']
(<src.dbops.Component object at 0x7de2754bf4a0>,)
Fairbench Analysis
2024-10-14 13:47:02,643 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:02,643 INFO sqlalchemy.engine.Engine [cached since 7300s ago] {'param_1': 2}
[<src.dbops.InOutTypes object at 0x7de2754bfa70>, <src.dbops.InOutTypes object at 0x7de2754bfa10>]
CSVDataset
CSVDataset detected
ONNXModel
(<src.dbops.Component object at 0x7de2754bf4a0>,)
Fairbench Analysis
[<src.dbops.InOutTypes object at 0x7de2754bfa70>, <src.dbops.InOutTypes object at 0x7de2754bfa10>]
CSVDataset
CSVDataset detected
ONNXModel
(<src.dbops.Component object at 0x7de2754bf110>,)
image bias analysis
2024-10-14 13:47:02,644 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:02,644 INFO sqlalchemy.engine.Engine [cached since 7300s ago] {'param_1': 8}
[<src.dbops.InOutTypes object at 0x7de2754bfbc0>, <src.dbops.InOutTypes object at 0x7de2754bfc80>]
Image
Pytorch
(<src.dbops.Component object at 0x7de2754bf110>,)
image bias analysis
[<src.dbops.InOutTypes object at 0x7de2754bfbc0>, <src.dbops.InOutTypes object at 0x7de2754bfc80>]
Image
Pytorch
(<src.dbops.Component object at 0x7de2754bc6b0>,)
model card
2024-10-14 13:47:02,646 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:02,646 INFO sqlalchemy.engine.Engine [cached since 7300s ago] {'param_1': 12}
[<src.dbops.InOutTypes object at 0x7de2754bfdd0>, <src.dbops.InOutTypes object at 0x7de2754bfe90>]
CSV
ONNX
ONNX detected
(<src.dbops.Component object at 0x7de2754bc6b0>,)
model card
[<src.dbops.InOutTypes object at 0x7de2754bfdd0>, <src.dbops.InOutTypes object at 0x7de2754bfe90>]
CSV
ONNX
ONNX detected
2024-10-14 13:47:02,647 INFO sqlalchemy.engine.Engine ROLLBACK
172.21.0.6:53728 - "GET /wizard/databias/metric/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:06,628 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:06,630 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:06,630 INFO sqlalchemy.engine.Engine [cached since 7304s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
2024-10-14 13:47:06,631 INFO sqlalchemy.engine.Engine ROLLBACK
{'result_id': 'STATUS_OK', 'result_message': 'OK', 'is_error': False, 'status_code': None, 'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'run_status': None, 'run_id': None, 'run_type': 'MODEL_BIAS', 'run_start_time': None, 'run_end_time': None, 'attributes': ['marital'], 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'loader_data': {'id': 'loader_csvdataset', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'metrics': None, 'url_data': '', 'url_model': '', 'run_artifacts': None, 'step': 3}
2024-10-14 13:47:06,632 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:06,632 INFO sqlalchemy.engine.Engine SELECT domain.id, domain.short_id, domain.name 
FROM domain 
WHERE domain.short_id = %(short_id_1)s ORDER BY domain.name
2024-10-14 13:47:06,633 INFO sqlalchemy.engine.Engine [cached since 14.33s ago] {'short_id_1': 'fairbench_initial'}
2024-10-14 13:47:06,634 INFO sqlalchemy.engine.Engine SELECT attributes.id AS attributes_id, attributes.attribute AS attributes_attribute, attributes.domain_id AS attributes_domain_id 
FROM attributes 
WHERE %(param_1)s = attributes.domain_id
2024-10-14 13:47:06,634 INFO sqlalchemy.engine.Engine [cached since 14.33s ago] {'param_1': 1}
2024-10-14 13:47:06,635 INFO sqlalchemy.engine.Engine ROLLBACK
['marital']
172.21.0.6:53728 - "GET /wizard/databias/attributes/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:08,177 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:08,178 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:08,178 INFO sqlalchemy.engine.Engine [cached since 7305s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
2024-10-14 13:47:08,180 INFO sqlalchemy.engine.Engine ROLLBACK
RESULT_JSON: {'result_id': 'STATUS_OK', 'result_message': 'OK', 'is_error': False, 'status_code': None, 'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'run_status': None, 'run_id': None, 'run_type': 'MODEL_BIAS', 'run_start_time': None, 'run_end_time': None, 'attributes': ['marital'], 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'loader_data': {'id': 'loader_csvdataset', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'metrics': None, 'url_data': '', 'url_model': '', 'run_artifacts': None, 'step': 3}
2024-10-14 13:47:08,181 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:08,182 INFO sqlalchemy.engine.Engine SELECT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.type = %(type_1)s AND in_out_types.type = %(type_2)s ORDER BY component.name
2024-10-14 13:47:08,182 INFO sqlalchemy.engine.Engine [cached since 34.32s ago] {'type_1': 'LOADER_DATA', 'type_2': 'OUT'}
<sqlalchemy.engine.result.ChunkedIteratorResult object at 0x7de274cf6890>
2024-10-14 13:47:08,185 INFO sqlalchemy.engine.Engine SELECT DISTINCT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.short_id = %(short_id_1)s ORDER BY component.name
2024-10-14 13:47:08,186 INFO sqlalchemy.engine.Engine [cached since 34.32s ago] {'short_id_1': 'model_onnx'}
(<src.dbops.Component object at 0x7de2754bc980>,)
2024-10-14 13:47:08,189 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:08,189 INFO sqlalchemy.engine.Engine [cached since 7305s ago] {'param_1': 10}
Input types: []
2024-10-14 13:47:08,192 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:08,192 INFO sqlalchemy.engine.Engine [cached since 7305s ago] {'param_1': 1}
2024-10-14 13:47:08,194 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:08,194 INFO sqlalchemy.engine.Engine [cached since 7305s ago] {'param_1': 4}
2024-10-14 13:47:08,196 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:08,196 INFO sqlalchemy.engine.Engine [cached since 7305s ago] {'param_1': 6}
2024-10-14 13:47:08,198 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:08,198 INFO sqlalchemy.engine.Engine [cached since 7305s ago] {'param_1': 9}
2024-10-14 13:47:08,201 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:08,201 INFO sqlalchemy.engine.Engine [cached since 7305s ago] {'param_1': 13}
2024-10-14 13:47:08,203 INFO sqlalchemy.engine.Engine ROLLBACK
2024-10-14 13:47:08,204 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:08,205 INFO sqlalchemy.engine.Engine SELECT domain.id, domain.short_id, domain.name 
FROM domain ORDER BY domain.name
2024-10-14 13:47:08,205 INFO sqlalchemy.engine.Engine [cached since 34.32s ago] {}
images
fairbench_initial
finance
[Domain(id='images', name='Face recognition', attributes=[]), Domain(id='fairbench_initial', name='Fairbench Pipeline Run', attributes=[]), Domain(id='finance', name='Finance', attributes=[])]
2024-10-14 13:47:08,208 INFO sqlalchemy.engine.Engine ROLLBACK
172.21.0.6:53728 - "GET /wizard/databias/data/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:19,747 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:19,747 INFO sqlalchemy.engine.Engine SELECT "user".id, "user".username, "user".user_auth_id, "user".email 
FROM "user" 
WHERE "user".username = %(username_1)s
2024-10-14 13:47:19,748 INFO sqlalchemy.engine.Engine [cached since 45.93s ago] {'username_1': 'test test'}
2024-10-14 13:47:19,749 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:19,749 INFO sqlalchemy.engine.Engine [cached since 7317s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
Existing json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 3, 'loader_data': {'id': 'loader_csvdataset', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'attributes': ['marital']}
Response json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'step': 3}
Updated json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 3, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'attributes': ['marital']}
2024-10-14 13:47:19,750 INFO sqlalchemy.engine.Engine UPDATE wizard_options SET configuration=%(configuration)s WHERE wizard_options.id = %(wizard_options_id)s
2024-10-14 13:47:19,750 INFO sqlalchemy.engine.Engine [cached since 27.49s ago] {'configuration': '{"uuid": "861ef2e8-c687-47d8-95ca-eda9b71e1a26", "name": "test", "group": "test", "loader_model": {"id": "model_onnx", "parameters_value": {"path": " ... (327 characters truncated) ... "on_bad_lines": "skip", "path": "http://host.k3d.internal:5000/bank.csv"}}, "domain": "fairbench_initial", "url_data": "", "attributes": ["marital"]}', 'wizard_options_id': 3}
2024-10-14 13:47:19,751 INFO sqlalchemy.engine.Engine COMMIT
172.21.0.6:33538 - "POST /wizard/store/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:19,766 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:19,766 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:19,766 INFO sqlalchemy.engine.Engine [cached since 7317s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
2024-10-14 13:47:19,767 INFO sqlalchemy.engine.Engine ROLLBACK
{'result_id': 'STATUS_OK', 'result_message': 'OK', 'is_error': False, 'status_code': None, 'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'run_status': None, 'run_id': None, 'run_type': 'MODEL_BIAS', 'run_start_time': None, 'run_end_time': None, 'attributes': ['marital'], 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'metrics': None, 'url_data': '', 'url_model': '', 'run_artifacts': None, 'step': 3}
2024-10-14 13:47:19,768 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:19,768 INFO sqlalchemy.engine.Engine SELECT domain.id, domain.short_id, domain.name 
FROM domain 
WHERE domain.short_id = %(short_id_1)s ORDER BY domain.name
2024-10-14 13:47:19,769 INFO sqlalchemy.engine.Engine [cached since 27.47s ago] {'short_id_1': 'fairbench_initial'}
2024-10-14 13:47:19,770 INFO sqlalchemy.engine.Engine SELECT attributes.id AS attributes_id, attributes.attribute AS attributes_attribute, attributes.domain_id AS attributes_domain_id 
FROM attributes 
WHERE %(param_1)s = attributes.domain_id
2024-10-14 13:47:19,770 INFO sqlalchemy.engine.Engine [cached since 27.46s ago] {'param_1': 1}
2024-10-14 13:47:19,770 INFO sqlalchemy.engine.Engine ROLLBACK
['marital']
172.21.0.6:33538 - "GET /wizard/databias/attributes/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:21,807 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:21,807 INFO sqlalchemy.engine.Engine SELECT "user".id, "user".username, "user".user_auth_id, "user".email 
FROM "user" 
WHERE "user".username = %(username_1)s
2024-10-14 13:47:21,807 INFO sqlalchemy.engine.Engine [cached since 47.99s ago] {'username_1': 'test test'}
2024-10-14 13:47:21,809 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:21,809 INFO sqlalchemy.engine.Engine [cached since 7319s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
Existing json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 3, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'attributes': ['marital']}
Response json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'attributes': ['marital'], 'step': 3}
Updated json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 3, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'attributes': ['marital']}
2024-10-14 13:47:21,811 INFO sqlalchemy.engine.Engine UPDATE wizard_options SET configuration=%(configuration)s WHERE wizard_options.id = %(wizard_options_id)s
2024-10-14 13:47:21,811 INFO sqlalchemy.engine.Engine [cached since 29.55s ago] {'configuration': '{"uuid": "861ef2e8-c687-47d8-95ca-eda9b71e1a26", "name": "test", "group": "test", "loader_model": {"id": "model_onnx", "parameters_value": {"path": " ... (327 characters truncated) ... "on_bad_lines": "skip", "path": "http://host.k3d.internal:5000/bank.csv"}}, "domain": "fairbench_initial", "url_data": "", "attributes": ["marital"]}', 'wizard_options_id': 3}
2024-10-14 13:47:21,812 INFO sqlalchemy.engine.Engine COMMIT
172.21.0.6:33538 - "POST /wizard/store/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:21,830 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:21,830 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:21,830 INFO sqlalchemy.engine.Engine [cached since 7319s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
2024-10-14 13:47:21,831 INFO sqlalchemy.engine.Engine ROLLBACK
{'result_id': 'STATUS_OK', 'result_message': 'OK', 'is_error': False, 'status_code': None, 'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'run_status': None, 'run_id': None, 'run_type': 'MODEL_BIAS', 'run_start_time': None, 'run_end_time': None, 'attributes': ['marital'], 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'metrics': None, 'url_data': '', 'url_model': '', 'run_artifacts': None, 'step': 3}
2024-10-14 13:47:21,832 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:21,832 INFO sqlalchemy.engine.Engine SELECT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.type = %(type_1)s AND in_out_types.type = %(type_2)s ORDER BY component.name
2024-10-14 13:47:21,833 INFO sqlalchemy.engine.Engine [cached since 47.97s ago] {'type_1': 'METRIC', 'type_2': 'IN'}
<sqlalchemy.engine.result.ChunkedIteratorResult object at 0x7de274ceb450>
2024-10-14 13:47:21,834 INFO sqlalchemy.engine.Engine SELECT DISTINCT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.short_id = %(short_id_1)s ORDER BY component.name
2024-10-14 13:47:21,834 INFO sqlalchemy.engine.Engine [cached since 47.96s ago] {'short_id_1': 'data_csv'}
(<src.dbops.Component object at 0x7de2751c0950>,)
2024-10-14 13:47:21,836 INFO sqlalchemy.engine.Engine SELECT DISTINCT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.short_id = %(short_id_1)s ORDER BY component.name
2024-10-14 13:47:21,836 INFO sqlalchemy.engine.Engine [cached since 47.97s ago] {'short_id_1': 'model_onnx'}
(<src.dbops.Component object at 0x7de2751c0260>,)
2024-10-14 13:47:21,837 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:21,837 INFO sqlalchemy.engine.Engine [cached since 7319s ago] {'param_1': 10}
2024-10-14 13:47:21,838 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:21,838 INFO sqlalchemy.engine.Engine [cached since 7319s ago] {'param_1': 4}
Input types: ['ONNX', 'CSV']
(<src.dbops.Component object at 0x7de2754bc6e0>,)
Fairbench Analysis
2024-10-14 13:47:21,839 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:21,840 INFO sqlalchemy.engine.Engine [cached since 7319s ago] {'param_1': 2}
[<src.dbops.InOutTypes object at 0x7de2751c0c50>, <src.dbops.InOutTypes object at 0x7de2751c0bf0>]
CSVDataset
ONNXModel
(<src.dbops.Component object at 0x7de2754bc6e0>,)
Fairbench Analysis
[<src.dbops.InOutTypes object at 0x7de2751c0c50>, <src.dbops.InOutTypes object at 0x7de2751c0bf0>]
CSVDataset
ONNXModel
(<src.dbops.Component object at 0x7de2754bf500>,)
image bias analysis
2024-10-14 13:47:21,840 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:21,841 INFO sqlalchemy.engine.Engine [cached since 7319s ago] {'param_1': 8}
[<src.dbops.InOutTypes object at 0x7de2751c0dd0>, <src.dbops.InOutTypes object at 0x7de2751c0fb0>]
Image
Pytorch
(<src.dbops.Component object at 0x7de2754bf500>,)
image bias analysis
[<src.dbops.InOutTypes object at 0x7de2751c0dd0>, <src.dbops.InOutTypes object at 0x7de2751c0fb0>]
Image
Pytorch
(<src.dbops.Component object at 0x7de2754bef00>,)
model card
2024-10-14 13:47:21,842 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-14 13:47:21,842 INFO sqlalchemy.engine.Engine [cached since 7319s ago] {'param_1': 12}
[<src.dbops.InOutTypes object at 0x7de2751c0ec0>, <src.dbops.InOutTypes object at 0x7de2751c1460>]
CSV
CSV detected
ONNX
ONNX detected
model card component is compatible
(<src.dbops.Component object at 0x7de2754bef00>,)
model card
[<src.dbops.InOutTypes object at 0x7de2751c0ec0>, <src.dbops.InOutTypes object at 0x7de2751c1460>]
CSV
CSV detected
ONNX
ONNX detected
model card component is compatible
2024-10-14 13:47:21,843 INFO sqlalchemy.engine.Engine ROLLBACK
172.21.0.6:33538 - "GET /wizard/databias/metric/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:34,575 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:34,575 INFO sqlalchemy.engine.Engine SELECT "user".id, "user".username, "user".user_auth_id, "user".email 
FROM "user" 
WHERE "user".username = %(username_1)s
2024-10-14 13:47:34,576 INFO sqlalchemy.engine.Engine [cached since 60.76s ago] {'username_1': 'test test'}
2024-10-14 13:47:34,577 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:34,577 INFO sqlalchemy.engine.Engine [cached since 7332s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
Existing json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 3, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'attributes': ['marital']}
Response json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'metrics': [{'id': 'model_card', 'parameters_value': {}}], 'step': 4}
Updated json {'uuid': '861ef2e8-c687-47d8-95ca-eda9b71e1a26', 'name': 'test', 'group': 'test', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 4, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'fairbench_initial', 'url_data': '', 'attributes': ['marital'], 'metrics': [{'id': 'model_card', 'parameters_value': {}}]}
2024-10-14 13:47:34,580 INFO sqlalchemy.engine.Engine UPDATE wizard_options SET configuration=%(configuration)s WHERE wizard_options.id = %(wizard_options_id)s
2024-10-14 13:47:34,580 INFO sqlalchemy.engine.Engine [cached since 42.32s ago] {'configuration': '{"uuid": "861ef2e8-c687-47d8-95ca-eda9b71e1a26", "name": "test", "group": "test", "loader_model": {"id": "model_onnx", "parameters_value": {"path": " ... (386 characters truncated) ... 000/bank.csv"}}, "domain": "fairbench_initial", "url_data": "", "attributes": ["marital"], "metrics": [{"id": "model_card", "parameters_value": {}}]}', 'wizard_options_id': 3}
2024-10-14 13:47:34,581 INFO sqlalchemy.engine.Engine COMMIT
172.21.0.6:52440 - "POST /wizard/store/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:34,619 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:34,620 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:34,620 INFO sqlalchemy.engine.Engine [cached since 7332s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
2024-10-14 13:47:34,621 INFO sqlalchemy.engine.Engine ROLLBACK
172.21.0.6:52440 - "GET /wizard/databias/overview/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 200
2024-10-14 13:47:36,911 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:36,911 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:36,912 INFO sqlalchemy.engine.Engine [cached since 7334s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
ACTION INFO FOR PIPELINES
result_id='STATUS_OK' result_message='OK' is_error=False status_code=None uuid='861ef2e8-c687-47d8-95ca-eda9b71e1a26' name='test' group='test' run_status=None run_id=None run_type=<RunType.MODEL_BIAS: 'MODEL_BIAS'> run_start_time=None run_end_time=None attributes=['marital'] loader_model=ComponentSelection(id='model_onnx', parameters_value={'path': 'http://host.k3d.internal:5000/model.onnx'}) loader_data=ComponentSelection(id='data_csv', parameters_value={'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}) domain='fairbench_initial' metrics=[ComponentSelection(id='model_card', parameters_value={})] url_data='' url_model='' run_artifacts=None step=4
/opt/pysetup/.venv/lib/python3.12/site-packages/kfp/client/client.py:159: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
  warnings.warn(
Error: Cannot submit the pipeline for execution.
MaxRetryError("HTTPConnectionPool(host='kfp.local.exus.ai', port=8082): Max retries exceeded with url: /apis/v2beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7de2754bed50>: Failed to establish a new connection: [Errno 111] Connection refused'))")
2024-10-14 13:47:36,920 INFO sqlalchemy.engine.Engine ROLLBACK
172.21.0.6:52440 - "GET /wizard/databias/submit/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 502
2024-10-14 13:47:44,316 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:47:44,317 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:47:44,318 INFO sqlalchemy.engine.Engine [cached since 7341s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
ACTION INFO FOR PIPELINES
result_id='STATUS_OK' result_message='OK' is_error=False status_code=None uuid='861ef2e8-c687-47d8-95ca-eda9b71e1a26' name='test' group='test' run_status=None run_id=None run_type=<RunType.MODEL_BIAS: 'MODEL_BIAS'> run_start_time=None run_end_time=None attributes=['marital'] loader_model=ComponentSelection(id='model_onnx', parameters_value={'path': 'http://host.k3d.internal:5000/model.onnx'}) loader_data=ComponentSelection(id='data_csv', parameters_value={'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}) domain='fairbench_initial' metrics=[ComponentSelection(id='model_card', parameters_value={})] url_data='' url_model='' run_artifacts=None step=4
Error: Cannot submit the pipeline for execution.
MaxRetryError("HTTPConnectionPool(host='kfp.local.exus.ai', port=8082): Max retries exceeded with url: /apis/v2beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7de2754ae360>: Failed to establish a new connection: [Errno 111] Connection refused'))")
2024-10-14 13:47:44,323 INFO sqlalchemy.engine.Engine ROLLBACK
172.21.0.6:36344 - "GET /wizard/databias/submit/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 502
2024-10-14 13:48:24,606 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-14 13:48:24,606 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-14 13:48:24,606 INFO sqlalchemy.engine.Engine [cached since 7382s ago] {'conf_id_1': '861ef2e8-c687-47d8-95ca-eda9b71e1a26'}
ACTION INFO FOR PIPELINES
result_id='STATUS_OK' result_message='OK' is_error=False status_code=None uuid='861ef2e8-c687-47d8-95ca-eda9b71e1a26' name='test' group='test' run_status=None run_id=None run_type=<RunType.MODEL_BIAS: 'MODEL_BIAS'> run_start_time=None run_end_time=None attributes=['marital'] loader_model=ComponentSelection(id='model_onnx', parameters_value={'path': 'http://host.k3d.internal:5000/model.onnx'}) loader_data=ComponentSelection(id='data_csv', parameters_value={'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}) domain='fairbench_initial' metrics=[ComponentSelection(id='model_card', parameters_value={})] url_data='' url_model='' run_artifacts=None step=4
Error: Cannot submit the pipeline for execution.
MaxRetryError("HTTPConnectionPool(host='kfp.local.exus.ai', port=8082): Max retries exceeded with url: /apis/v2beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7de2752c75c0>: Failed to establish a new connection: [Errno 111] Connection refused'))")
2024-10-14 13:48:24,614 INFO sqlalchemy.engine.Engine ROLLBACK
172.21.0.6:58744 - "GET /wizard/databias/submit/861ef2e8-c687-47d8-95ca-eda9b71e1a26 HTTP/1.1" 502

[georgiosn]

Thank you for the logs @elmokhtar-mohamedmoussa-idnow

We keep that the toolkit cannot connect to KFP based on the last lines of the log:

Error: Cannot submit the pipeline for execution. MaxRetryError("HTTPConnectionPool(host='kfp.local.exus.ai', port=8082): Max retries exceeded with url: /apis/v2beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7de2752c75c0>: Failed to establish a new connection: [Errno 111] Connection refused'))")

Please check that you can access http://kfp.local.exus.ai:8082/ from your browser. Is it possible?

If not check that K3D is running with running in terminal

k3d cluster list

what you see?

[elmokhtar-mohamedmoussa-idnow]

I can't access http://kfp.local.exus.ai:8082/. when i run k3d cluster list, i get :

NAME   SERVERS   AGENTS   LOADBALANCER
kfp    0/1       0/0      true

[georgiosn]

Thanks for the feedback.

Please do

k3d cluster delete kfp

Then recreate the KFP cluster by running the command

k3d cluster create kfp --api-port 6550 -p "8082:80@loadbalancer" --agents 1  # create a K3D cluster

and then follow step 3 https://github.com/mammoth-eu/mammoth-toolkit-releases?tab=readme-ov-file#step-3-kfp

After this you should be able to open http://kfp.local.exus.ai:8082

and retry the toolkit wizard.

[elmokhtar-mohamedmoussa-idnow]

ok, step 3 (./scripts/kfp_install.sh) gives me an error:

This will deploy Kubeflow Pipelines Standalone instance v2.2.0
Please monitor the installation and provide input as required. Press enter to continue...

---- Please make sure that you have installed and active: ----
- Docker
- Kubernetes cluster, could be K3d, Minikube, or any other distribution
- kubectl

 Press enter to continue, or CTRL+C to stop the script...
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
error: error validating "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=2.2.0": error validating data: failed to download openapi: an error on the server ("unknown") has prevented the request from succeeding; if you choose to ignore these errors, turn validation off with --validate=false

[georgiosn]

A new version of KFP became available.

Please do a git pull to get the updated scripts and retry.

Thanks!

[elmokhtar-mohamedmoussa-idnow]

unfortunatly I still have the error at step 3

(base) elmokhtarmohamedmoussa@1vl9px3:~/Desktop/mammoth-toolkit-releases$ ./scripts/kfp_install.sh 
This will deploy Kubeflow Pipelines Standalone instance v2.2.0
Please monitor the installation and provide input as required. Press enter to continue...

---- Please make sure that you have installed and active: ----
- Docker
- Kubernetes cluster, could be K3d, Minikube, or any other distribution
- kubectl

 Press enter to continue, or CTRL+C to stop the script...
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
error: error validating "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=2.3.0": error validating data: failed to download openapi: an error on the server ("unknown") has prevented the request from succeeding; if you choose to ignore these errors, turn validation off with --validate=false
(base) elmokhtarmohamedmoussa@1vl9px3:~/Desktop/mammoth-toolkit-releases$ git pull
Already up-to-date.

[sudhang]

Hm, after pulling the changes and running compose again (but not detached), i get some python errors from [api]. Don't believe it is on my end. Note: I didn't reinstall anything

[krakend]           | [00] 
[api]               | 2024-10-14 17:31:23,052 INFO sqlalchemy.engine.Engine BEGIN (implicit)
[api]               | 2024-10-14 17:31:23,053 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
[api]               | FROM wizard_options 
[api]               | WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
[api]               | 2024-10-14 17:31:23,053 INFO sqlalchemy.engine.Engine [cached since 156s ago] {'conf_id_1': 'fbc588f9-5596-493a-8b96-ce05d052166a'}
[api]               | ACTION INFO FOR PIPELINES
[api]               | result_id='STATUS_OK' result_message='OK' is_error=False status_code=None uuid='fbc588f9-5596-493a-8b96-ce05d052166a' name='test' group='test' run_status=None run_id=None run_type=<RunType.MODEL_BIAS: 'MODEL_BIAS'> run_start_time=None run_end_time=None attributes=['Marital Status'] loader_model=ComponentSelection(id='model_onnx', parameters_value={'path': 'http://host.k3d.internal:5000/model.onnx'}) loader_data=ComponentSelection(id='data_csv', parameters_value={'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}) domain='finance' metrics=[ComponentSelection(id='model_card', parameters_value={})] url_data='' url_model='' run_artifacts=None step=4
[api]               | /opt/pysetup/.venv/lib/python3.12/site-packages/kfp/client/client.py:159: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
[api]               |   warnings.warn(
[api]               | File Import Contents: from typing import Dict, List
[api]               | from kfp import compiler, components, dsl
[api]               | 
[api]               | model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
[api]               | data_csv = components.load_component_from_file("/var/lib/components_yaml/data_csv.yaml")
[api]               | model_card = (components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")
[api]               | 
[api]               | 
[api]               | Component path is: /var/lib/components_yaml/model_onnx.yaml
[api]               | Loading component from path: /var/lib/components_yaml/model_onnx.yaml
[api]               | Component path is: /var/lib/components_yaml/data_csv.yaml
[api]               | Loading component from path: /var/lib/components_yaml/data_csv.yaml
[api]               | Component path is: /var/lib/components_yaml/model_card.yaml
[api]               | Loading component from path: /var/lib/components_yaml/model_card.yaml
[api]               | Components list size: 3
[api]               | File Import Contents: from typing import Dict, List
[api]               | from kfp import compiler, components, dsl
[api]               | 
[api]               | model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
[api]               | data_csv = components.load_component_from_file("/var/lib/components_yaml/data_csv.yaml")
[api]               | model_card = (components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")
[api]               | 
[api]               | 
[api]               | Pipeline function: @dsl.pipeline(
[api]               |   name='test'
[api]               | )
[api]               | def pipeline(model_onnx__params:Dict, data_csv__params:Dict, sensitive:List, model_card__params:Dict):
[api]               |   model_onnx_task = model_onnx(model_onnx__params=model_onnx__params)
[api]               |   data_csv_task = data_csv(data_csv__params=data_csv__params)
[api]               |   model_card_task = model_card(model_card__params=model_card__params, sensitive=sensitive, dataset=data_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])
[api]               | 
[api]               | Pipeline TOTAL content: from typing import Dict, List
[api]               | from kfp import compiler, components, dsl
[api]               | 
[api]               | model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
[api]               | data_csv = components.load_component_from_file("/var/lib/components_yaml/data_csv.yaml")
[api]               | model_card = (components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")
[api]               | 
[api]               | @dsl.pipeline(
[api]               |   name='test'
[api]               | )
[api]               | def pipeline(model_onnx__params:Dict, data_csv__params:Dict, sensitive:List, model_card__params:Dict):
[api]               |   model_onnx_task = model_onnx(model_onnx__params=model_onnx__params)
[api]               |   data_csv_task = data_csv(data_csv__params=data_csv__params)
[api]               |   model_card_task = model_card(model_card__params=model_card__params, sensitive=sensitive, dataset=data_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])
[api]               | 
[api]               | 
[api]               | compiler.Compiler().compile(pipeline, "/var/lib/pipelines/pipeline.yaml")
[api]               | Pipeline arguments
[api]               | {'sensitive': ['Marital Status'], 'model_onnx__params': {'path': 'http://host.k3d.internal:5000/model.onnx'}, 'data_csv__params': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}, 'model_card__params': {}}
[api]               |   File "/var/lib/pipelines/pipeline.py", line 6
[api]               |     model_card = (components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")
[api]               |                  ^
[api]               | SyntaxError: '(' was never closed
[api]               | Error running pipeline COMMAND LINE
[api]               | GENERATED ACTION INFO FOR PIPELINES
[api]               | {'sensitive': ['Marital Status'], 'data_csv__params': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}, 'model_onnx__params': {'path': 'http://host.k3d.internal:5000/model.onnx'}, 'model_card__params': {}}
[api]               | Experiment details: http://kfp.local.exus.ai:8082/#/experiments/details/f5efad8d-c29e-4ef2-bfce-31fcaf813824
[api]               | Error: Cannot submit the pipeline for execution.
[api]               | UnboundLocalError("cannot access local variable 'result' where it is not associated with a value")
[api]               | 2024-10-14 17:31:23,196 INFO sqlalchemy.engine.Engine ROLLBACK
[api]               | 10.89.2.57:40716 - "GET /wizard/databias/submit/fbc588f9-5596-493a-8b96-ce05d052166a HTTP/1.1" 502
[krakend]           | [00] 2024/10/14 17:31:23 KRAKEND ERROR: [ENDPOINT: /wizard/databias/submit/:uuid] invalid status code

[sudhang]

UPDATE:

I think it is a typo. I exece'd into the running container and the the model_card assignment has an extra paren

# cat pipeline.py
from typing import Dict, List
from kfp import compiler, components, dsl

model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
data_csv = components.load_component_from_file("/var/lib/components_yaml/data_csv.yaml")
model_card = (components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")

I also tried altering the file inside the container, but it seems to get recreated/redownloaded when I try that, and the same error arises.

I don't know if this is the only issue though. Also, for me, I only see "Finance" and "Face recognition" Domains

[georgiosn]

Thank you for this info @sudhang ! Great help we work on it and will have a new version soon for you to check.

@elmokhtar-mohamedmoussa-idnow we also check the issue you have and will come back with suggestions.

[georgiosn]

@sudhang a new version fixing the problem you mentioned uploaded. Please git pull and check using the latest version.

@elmokhtar-mohamedmoussa-idnow we managed to install the KFP with the current scripts. Is there any firewalls or network rules blocking you?

[elmokhtar-mohamedmoussa-idnow]

Thank you for the feedback @georgiosn, I will then contact IT to see.

[georgiosn]

Hello please before any testing do a git pull again as another issue resolved in sending the pipeline to KFP for execution. Thanks!

[elmokhtar-mohamedmoussa-idnow]

Thank you for the logs @elmokhtar-mohamedmoussa-idnow

We keep that the toolkit cannot connect to KFP based on the last lines of the log:

Error: Cannot submit the pipeline for execution. MaxRetryError("HTTPConnectionPool(host='kfp.local.exus.ai', port=8082): Max retries exceeded with url: /apis/v2beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7de2752c75c0>: Failed to establish a new connection: [Errno 111] Connection refused'))")

Please check that you can access http://kfp.local.exus.ai:8082/ from your browser. Is it possible?

Hi @georgiosn I can however access kfp.local.exus.ai:8080 is that enough or should it be exactly the same port ?

[georgiosn]

Hi @elmokhtar-mohamedmoussa-idnow

It is weird that you can access KFP on 8080, toolkit expects it to be accessible at 8082 so that's why is not operating properly.

You have two options or you do again the procedure as described in https://github.com/mammoth-eu/mammoth-toolkit-releases/issues/9#issuecomment-2411600939

In 8080 is the Keycloak service and not the KFP one.

The KFP looks like this:

[elmokhtar-mohamedmoussa-idnow]

OK @georgiosn I can now acces http://kfp.local.exus.ai:8082/ there was a typo in the installation script of step 2 : https://github.com/mammoth-eu/mammoth-toolkit-releases/blob/dev/scripts/kfp_install.sh#L24 Since the output was very verbose I didn't see the error the first time.

[georgiosn]

Hi @elmokhtar-mohamedmoussa-idnow I updated the previous comment, in the initial comment I wrote that you should edit the .env and change the port, this is not valid as in 8080 runs another service and not KFP that we need.

[georgiosn]

OK @georgiosn I can now acces http://kfp.local.exus.ai:8082/ there was a typo in the installation script of step 2 : https://github.com/mammoth-eu/mammoth-toolkit-releases/blob/dev/scripts/kfp_install.sh#L24 Since the output was very verbose I didn't see the error the first time.

Thanks for the reply!

[elmokhtar-mohamedmoussa-idnow]

Ok now, I still have the 500 error when run the fairbench pipeline tutorial. Here are the logs :

model card component is compatible
2024-10-24 15:10:08,019 INFO sqlalchemy.engine.Engine ROLLBACK
172.19.0.3:37950 - "GET /wizard/databias/metric/9fa3c740-3b49-4f54-a600-3df2f63a444c HTTP/1.1" 200
2024-10-24 15:10:16,759 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-24 15:10:16,760 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-24 15:10:16,760 INFO sqlalchemy.engine.Engine [cached since 109.8s ago] {'conf_id_1': '9fa3c740-3b49-4f54-a600-3df2f63a444c'}
2024-10-24 15:10:16,761 INFO sqlalchemy.engine.Engine ROLLBACK
{'result_id': 'STATUS_OK', 'result_message': 'OK', 'is_error': False, 'status_code': None, 'uuid': '9fa3c740-3b49-4f54-a600-3df2f63a444c', 'name': 'test', 'group': 'testfairbench', 'run_status': None, 'run_id': None, 'run_type': 'MODEL_BIAS', 'run_start_time': None, 'run_end_time': None, 'attributes': ['marital'], 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'finance', 'metrics': [{'id': 'model_card', 'parameters_value': {}}], 'url_data': '', 'url_model': '', 'run_artifacts': None, 'step': 4}
2024-10-24 15:10:16,762 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-24 15:10:16,763 INFO sqlalchemy.engine.Engine SELECT domain.id, domain.short_id, domain.name 
FROM domain 
WHERE domain.short_id = %(short_id_1)s ORDER BY domain.name
2024-10-24 15:10:16,763 INFO sqlalchemy.engine.Engine [cached since 42.71s ago] {'short_id_1': 'finance'}
2024-10-24 15:10:16,764 INFO sqlalchemy.engine.Engine SELECT attributes.id AS attributes_id, attributes.attribute AS attributes_attribute, attributes.domain_id AS attributes_domain_id 
FROM attributes 
WHERE %(param_1)s = attributes.domain_id
2024-10-24 15:10:16,764 INFO sqlalchemy.engine.Engine [cached since 42.71s ago] {'param_1': 1}
2024-10-24 15:10:16,765 INFO sqlalchemy.engine.Engine ROLLBACK
['Language', 'Occupation', 'Nationality', 'Marital Status', 'Gender', 'Birth Date', 'Identification Type', 'Country of Residence', 'Is Bankrupt', 'Bankrupt Date', 'marital']
172.19.0.3:46562 - "GET /wizard/databias/attributes/9fa3c740-3b49-4f54-a600-3df2f63a444c HTTP/1.1" 200
2024-10-24 15:10:18,620 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-24 15:10:18,620 INFO sqlalchemy.engine.Engine SELECT "user".id, "user".username, "user".user_auth_id, "user".email 
FROM "user" 
WHERE "user".username = %(username_1)s
2024-10-24 15:10:18,620 INFO sqlalchemy.engine.Engine [cached since 70.16s ago] {'username_1': 'test test'}
2024-10-24 15:10:18,622 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-24 15:10:18,622 INFO sqlalchemy.engine.Engine [cached since 111.7s ago] {'conf_id_1': '9fa3c740-3b49-4f54-a600-3df2f63a444c'}
Existing json {'uuid': '9fa3c740-3b49-4f54-a600-3df2f63a444c', 'name': 'test', 'group': 'testfairbench', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 4, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'finance', 'url_data': '', 'attributes': ['marital'], 'metrics': [{'id': 'model_card', 'parameters_value': {}}]}
Response json {'uuid': '9fa3c740-3b49-4f54-a600-3df2f63a444c', 'attributes': ['marital'], 'step': 4}
Updated json {'uuid': '9fa3c740-3b49-4f54-a600-3df2f63a444c', 'name': 'test', 'group': 'testfairbench', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 4, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'finance', 'url_data': '', 'attributes': ['marital'], 'metrics': [{'id': 'model_card', 'parameters_value': {}}]}
2024-10-24 15:10:18,625 INFO sqlalchemy.engine.Engine UPDATE wizard_options SET configuration=%(configuration)s WHERE wizard_options.id = %(wizard_options_id)s
2024-10-24 15:10:18,625 INFO sqlalchemy.engine.Engine [cached since 44.62s ago] {'configuration': '{"uuid": "9fa3c740-3b49-4f54-a600-3df2f63a444c", "name": "test", "group": "testfairbench", "loader_model": {"id": "model_onnx", "parameters_value": { ... (385 characters truncated) ... internal:5000/bank.csv"}}, "domain": "finance", "url_data": "", "attributes": ["marital"], "metrics": [{"id": "model_card", "parameters_value": {}}]}', 'wizard_options_id': 2}
2024-10-24 15:10:18,626 INFO sqlalchemy.engine.Engine COMMIT
172.19.0.3:46562 - "POST /wizard/store/9fa3c740-3b49-4f54-a600-3df2f63a444c HTTP/1.1" 200
2024-10-24 15:10:18,650 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-24 15:10:18,650 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-24 15:10:18,650 INFO sqlalchemy.engine.Engine [cached since 111.7s ago] {'conf_id_1': '9fa3c740-3b49-4f54-a600-3df2f63a444c'}
2024-10-24 15:10:18,652 INFO sqlalchemy.engine.Engine ROLLBACK
{'result_id': 'STATUS_OK', 'result_message': 'OK', 'is_error': False, 'status_code': None, 'uuid': '9fa3c740-3b49-4f54-a600-3df2f63a444c', 'name': 'test', 'group': 'testfairbench', 'run_status': None, 'run_id': None, 'run_type': 'MODEL_BIAS', 'run_start_time': None, 'run_end_time': None, 'attributes': ['marital'], 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'finance', 'metrics': [{'id': 'model_card', 'parameters_value': {}}], 'url_data': '', 'url_model': '', 'run_artifacts': None, 'step': 4}
2024-10-24 15:10:18,653 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-24 15:10:18,654 INFO sqlalchemy.engine.Engine SELECT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.type = %(type_1)s AND in_out_types.type = %(type_2)s ORDER BY component.name
2024-10-24 15:10:18,654 INFO sqlalchemy.engine.Engine [cached since 70.17s ago] {'type_1': 'METRIC', 'type_2': 'IN'}
<sqlalchemy.engine.result.ChunkedIteratorResult object at 0x76c2580adad0>
2024-10-24 15:10:18,655 INFO sqlalchemy.engine.Engine SELECT DISTINCT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.short_id = %(short_id_1)s ORDER BY component.name
2024-10-24 15:10:18,655 INFO sqlalchemy.engine.Engine [cached since 70.17s ago] {'short_id_1': 'data_csv'}
(<src.dbops.Component object at 0x76c2587e6030>,)
2024-10-24 15:10:18,657 INFO sqlalchemy.engine.Engine SELECT DISTINCT component.id, component.short_id, component.name, component.filename, component.description, component.parameter_info, component.parameter_default, component.type 
FROM component JOIN association_comp_in_out_table AS association_comp_in_out_table_1 ON component.id = association_comp_in_out_table_1.component_id JOIN in_out_types ON in_out_types.id = association_comp_in_out_table_1.in_out_type_id 
WHERE component.short_id = %(short_id_1)s ORDER BY component.name
2024-10-24 15:10:18,657 INFO sqlalchemy.engine.Engine [cached since 70.17s ago] {'short_id_1': 'model_onnx'}
(<src.dbops.Component object at 0x76c2587e4ef0>,)
2024-10-24 15:10:18,658 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-24 15:10:18,658 INFO sqlalchemy.engine.Engine [cached since 111.7s ago] {'param_1': 7}
2024-10-24 15:10:18,659 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-24 15:10:18,660 INFO sqlalchemy.engine.Engine [cached since 111.7s ago] {'param_1': 1}
Input types: ['ONNX', 'CSV']
(<src.dbops.Component object at 0x76c2587e30e0>,)
image bias analysis
2024-10-24 15:10:18,661 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-24 15:10:18,661 INFO sqlalchemy.engine.Engine [cached since 111.7s ago] {'param_1': 5}
[<src.dbops.InOutTypes object at 0x76c2587e4530>, <src.dbops.InOutTypes object at 0x76c2587e4380>]
Image
Pytorch
(<src.dbops.Component object at 0x76c2587e30e0>,)
image bias analysis
[<src.dbops.InOutTypes object at 0x76c2587e4530>, <src.dbops.InOutTypes object at 0x76c2587e4380>]
Image
Pytorch
(<src.dbops.Component object at 0x76c2587e3680>,)
model card
2024-10-24 15:10:18,662 INFO sqlalchemy.engine.Engine SELECT in_out_types.id AS in_out_types_id, in_out_types.name AS in_out_types_name, in_out_types.type AS in_out_types_type 
FROM in_out_types, association_comp_in_out_table 
WHERE %(param_1)s = association_comp_in_out_table.component_id AND in_out_types.id = association_comp_in_out_table.in_out_type_id
2024-10-24 15:10:18,662 INFO sqlalchemy.engine.Engine [cached since 111.7s ago] {'param_1': 9}
[<src.dbops.InOutTypes object at 0x76c2587e60c0>, <src.dbops.InOutTypes object at 0x76c2587e4740>]
CSV
CSV detected
ONNX
ONNX detected
model card component is compatible
(<src.dbops.Component object at 0x76c2587e3680>,)
model card
[<src.dbops.InOutTypes object at 0x76c2587e60c0>, <src.dbops.InOutTypes object at 0x76c2587e4740>]
CSV
CSV detected
ONNX
ONNX detected
model card component is compatible
2024-10-24 15:10:18,663 INFO sqlalchemy.engine.Engine ROLLBACK
172.19.0.3:46562 - "GET /wizard/databias/metric/9fa3c740-3b49-4f54-a600-3df2f63a444c HTTP/1.1" 200
2024-10-24 15:10:29,613 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-24 15:10:29,613 INFO sqlalchemy.engine.Engine SELECT "user".id, "user".username, "user".user_auth_id, "user".email 
FROM "user" 
WHERE "user".username = %(username_1)s
2024-10-24 15:10:29,613 INFO sqlalchemy.engine.Engine [cached since 81.15s ago] {'username_1': 'test test'}
2024-10-24 15:10:29,616 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-24 15:10:29,616 INFO sqlalchemy.engine.Engine [cached since 122.7s ago] {'conf_id_1': '9fa3c740-3b49-4f54-a600-3df2f63a444c'}
Existing json {'uuid': '9fa3c740-3b49-4f54-a600-3df2f63a444c', 'name': 'test', 'group': 'testfairbench', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 4, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'finance', 'url_data': '', 'attributes': ['marital'], 'metrics': [{'id': 'model_card', 'parameters_value': {}}]}
Response json {'uuid': '9fa3c740-3b49-4f54-a600-3df2f63a444c', 'metrics': [{'id': 'model_card', 'parameters_value': {}}], 'step': 4}
Updated json {'uuid': '9fa3c740-3b49-4f54-a600-3df2f63a444c', 'name': 'test', 'group': 'testfairbench', 'loader_model': {'id': 'model_onnx', 'parameters_value': {'path': 'http://host.k3d.internal:5000/model.onnx'}}, 'url_model': '', 'step': 4, 'loader_data': {'id': 'data_csv', 'parameters_value': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}}, 'domain': 'finance', 'url_data': '', 'attributes': ['marital'], 'metrics': [{'id': 'model_card', 'parameters_value': {}}]}
2024-10-24 15:10:29,619 INFO sqlalchemy.engine.Engine UPDATE wizard_options SET configuration=%(configuration)s WHERE wizard_options.id = %(wizard_options_id)s
2024-10-24 15:10:29,619 INFO sqlalchemy.engine.Engine [cached since 55.61s ago] {'configuration': '{"uuid": "9fa3c740-3b49-4f54-a600-3df2f63a444c", "name": "test", "group": "testfairbench", "loader_model": {"id": "model_onnx", "parameters_value": { ... (385 characters truncated) ... internal:5000/bank.csv"}}, "domain": "finance", "url_data": "", "attributes": ["marital"], "metrics": [{"id": "model_card", "parameters_value": {}}]}', 'wizard_options_id': 2}
2024-10-24 15:10:29,620 INFO sqlalchemy.engine.Engine COMMIT
172.19.0.3:45982 - "POST /wizard/store/9fa3c740-3b49-4f54-a600-3df2f63a444c HTTP/1.1" 200
2024-10-24 15:10:29,654 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-24 15:10:29,655 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-24 15:10:29,655 INFO sqlalchemy.engine.Engine [cached since 122.7s ago] {'conf_id_1': '9fa3c740-3b49-4f54-a600-3df2f63a444c'}
2024-10-24 15:10:29,657 INFO sqlalchemy.engine.Engine ROLLBACK
172.19.0.3:45982 - "GET /wizard/databias/overview/9fa3c740-3b49-4f54-a600-3df2f63a444c HTTP/1.1" 200
2024-10-24 15:10:34,098 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2024-10-24 15:10:34,098 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
FROM wizard_options 
WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
2024-10-24 15:10:34,098 INFO sqlalchemy.engine.Engine [cached since 127.2s ago] {'conf_id_1': '9fa3c740-3b49-4f54-a600-3df2f63a444c'}
ACTION INFO FOR PIPELINES
result_id='STATUS_OK' result_message='OK' is_error=False status_code=None uuid='9fa3c740-3b49-4f54-a600-3df2f63a444c' name='test' group='testfairbench' run_status=None run_id=None run_type=<RunType.MODEL_BIAS: 'MODEL_BIAS'> run_start_time=None run_end_time=None attributes=['marital'] loader_model=ComponentSelection(id='model_onnx', parameters_value={'path': 'http://host.k3d.internal:5000/model.onnx'}) loader_data=ComponentSelection(id='data_csv', parameters_value={'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}) domain='finance' metrics=[ComponentSelection(id='model_card', parameters_value={})] url_data='' url_model='' run_artifacts=None step=4
File Import Contents: from typing import Dict, List
from kfp import compiler, components, dsl

model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
data_csv = components.load_component_from_file("/var/lib/components_yaml/data_csv.yaml")
model_card = (components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")

Component path is: /var/lib/components_yaml/model_onnx.yaml
Loading component from path: /var/lib/components_yaml/model_onnx.yaml
Component path is: /var/lib/components_yaml/data_csv.yaml
Loading component from path: /var/lib/components_yaml/data_csv.yaml
Component path is: /var/lib/components_yaml/model_card.yaml
Loading component from path: /var/lib/components_yaml/model_card.yaml
Components list size: 3
File Import Contents: from typing import Dict, List
from kfp import compiler, components, dsl

model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
data_csv = components.load_component_from_file("/var/lib/components_yaml/data_csv.yaml")
model_card = (components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")

Pipeline function: @dsl.pipeline(
        name='test'
)
def pipeline(model_onnx__params:Dict, data_csv__params:Dict, model_card__params:Dict, sensitive:List):
        model_onnx_task = model_onnx(model_onnx__params=model_onnx__params)
        data_csv_task = data_csv(data_csv__params=data_csv__params)
        model_card_task = model_card(model_card__params=model_card__params, sensitive=sensitive, dataset=data_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])

Pipeline TOTAL content: from typing import Dict, List
from kfp import compiler, components, dsl

model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
data_csv = components.load_component_from_file("/var/lib/components_yaml/data_csv.yaml")
model_card = (components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")

@dsl.pipeline(
        name='test'
)
def pipeline(model_onnx__params:Dict, data_csv__params:Dict, model_card__params:Dict, sensitive:List):
        model_onnx_task = model_onnx(model_onnx__params=model_onnx__params)
        data_csv_task = data_csv(data_csv__params=data_csv__params)
        model_card_task = model_card(model_card__params=model_card__params, sensitive=sensitive, dataset=data_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])

compiler.Compiler().compile(pipeline, "/var/lib/pipelines/pipeline.yaml")
Error: Cannot submit the pipeline for execution.
PermissionError(13, 'Permission denied')
2024-10-24 15:10:34,145 INFO sqlalchemy.engine.Engine ROLLBACK
172.19.0.3:45982 - "GET /wizard/databias/submit/9fa3c740-3b49-4f54-a600-3df2f63a444c HTTP/1.1" 502
``

[sudhang]

Ok now, I still have the 500 error when run the fairbench pipeline tutorial. Here are the logs :

model card component is compatible
2024-10-24 15:10:08,019 INFO sqlalchemy.engine.Engine ROLLBACKmodel_card__params=model_card__params, sensitive=sensitive, dataset=data_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])

compiler.Compiler().compile(pipeline, "/var/lib/pipelines/pipeline.yaml")
Error: Cannot submit the pipeline for execution.
PermissionError(13, 'Permission denied')
2024-10-24 15:10:34,145 INFO sqlalchemy.engine.Engine ROLLBACK
172.19.0.3:45982 - "GET /wizard/databias/submit/9fa3c740-3b49-4f54-a600-3df2f63a444c HTTP/1.1" 502
``

Thanks for mentioning this @elmokhtar-mohamedmoussa-idnow . I get the same error, and was assuming that I had been doing something wrong

[georgiosn]

Hello both,

Do you have the latest toolkit code, please do a git pull and check that in docker-compose.yml the toolkit entry uses this image api: image: mammotheu/mt_api:v0015

[georgiosn]

Also, then you have to run at least one time docker compose up --build

[elmokhtar-mohamedmoussa-idnow]

I git pulled but when i run docker compose up --build, I have an error :

(base) elmokhtarmohamedmoussa@1vl9px3:~/Desktop/mammoth-toolkit-releases$ sudo docker compose up --build
[+] Running 5/0
 ✔ Container mammoth_kk-krakend-1     Running                                                                                                               0.0s 
 ✔ Container mammoth_kk-mammoth-fe-1  Running                                                                                                               0.0s 
 ✔ Container mammoth_kk-api-db-1      Running                                                                                                               0.0s 
 ✔ Container mammoth_kk-keycloak-1    Running                                                                                                               0.0s 
 ✔ Container mammoth_kk-adminer-1     Running                                                                                                               0.0s 
Attaching to adminer-1, api-1, api-db-1, keycloak-1, keycloak-postgres-1, krakend-1, mammoth-fe-1
keycloak-1           | Updating the configuration and installing your custom providers, if any. Please wait.
keycloak-1           | 2024-10-24 15:19:38,906 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 5072ms
keycloak-1           | 2024-10-24 15:19:40,055 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: keycloak.local.exus.ai, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
keycloak-1           | 2024-10-24 15:19:41,670 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
keycloak-1           | 2024-10-24 15:19:41,739 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:41,740 WARN  [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread) HHH000342: Could not obtain connection to query metadata: java.lang.NullPointerException: Cannot invoke "org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(java.sql.SQLException, String)" because the return value of "org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.sqlExceptionHelper()" is null
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doTheWork(JtaIsolationDelegate.java:186)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.lambda$delegateWork$1(JtaIsolationDelegate.java:75)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doInSuspendedTransaction(JtaIsolationDelegate.java:107)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.delegateWork(JtaIsolationDelegate.java:72)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentUsingJdbcMetadata(JdbcEnvironmentInitiator.java:279)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:193)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:69)
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:119)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:264)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:239)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-1           |  at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:52)
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:125)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:248)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-1           |  at org.hibernate.boot.internal.SessionFactoryOptionsBuilder.<init>(SessionFactoryOptionsBuilder.java:273)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.recording.PrevalidatedQuarkusMetadata.buildSessionFactoryOptionsBuilder(PrevalidatedQuarkusMetadata.java:70)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.boot.FastBootEntityManagerFactoryBuilder.build(FastBootEntityManagerFactoryBuilder.java:81)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.FastBootHibernatePersistenceProvider.createEntityManagerFactory(FastBootHibernatePersistenceProvider.java:74)
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:80)
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$LazyPersistenceUnit.get(JPAConfig.java:156)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$1.run(JPAConfig.java:64)
keycloak-1           |  at java.base/java.lang.Thread.run(Thread.java:840)
keycloak-1           | 
keycloak-1           | 2024-10-24 15:19:42,201 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
keycloak-1           | 2024-10-24 15:19:42,277 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
keycloak-1           | 2024-10-24 15:19:42,450 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
keycloak-1           | 2024-10-24 15:19:42,843 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:42,952 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (development) mode
keycloak-1           | 2024-10-24 15:19:42,952 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to obtain JDBC connection
keycloak-1           | 2024-10-24 15:19:42,952 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:42,952 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: keycloak-postgres
keycloak-1           | 2024-10-24 15:19:42,953 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
keycloak-1           | 2024-10-24 15:19:46,460 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: keycloak.local.exus.ai, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
keycloak-1           | 2024-10-24 15:19:48,256 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
keycloak-1           | 2024-10-24 15:19:48,373 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:48,374 WARN  [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread) HHH000342: Could not obtain connection to query metadata: java.lang.NullPointerException: Cannot invoke "org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(java.sql.SQLException, String)" because the return value of "org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.sqlExceptionHelper()" is null
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doTheWork(JtaIsolationDelegate.java:186)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.lambda$delegateWork$1(JtaIsolationDelegate.java:75)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doInSuspendedTransaction(JtaIsolationDelegate.java:107)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.delegateWork(JtaIsolationDelegate.java:72)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentUsingJdbcMetadata(JdbcEnvironmentInitiator.java:279)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:193)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:69)
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:119)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:264)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:239)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-1           |  at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:52)
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:125)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:248)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-1           |  at org.hibernate.boot.internal.SessionFactoryOptionsBuilder.<init>(SessionFactoryOptionsBuilder.java:273)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.recording.PrevalidatedQuarkusMetadata.buildSessionFactoryOptionsBuilder(PrevalidatedQuarkusMetadata.java:70)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.boot.FastBootEntityManagerFactoryBuilder.build(FastBootEntityManagerFactoryBuilder.java:81)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.FastBootHibernatePersistenceProvider.createEntityManagerFactory(FastBootHibernatePersistenceProvider.java:74)
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:80)
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$LazyPersistenceUnit.get(JPAConfig.java:156)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$1.run(JPAConfig.java:64)
keycloak-1           |  at java.base/java.lang.Thread.run(Thread.java:840)
keycloak-1           | 
keycloak-1           | 2024-10-24 15:19:48,718 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
keycloak-1           | 2024-10-24 15:19:48,781 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
keycloak-1           | 2024-10-24 15:19:48,896 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
keycloak-1           | 2024-10-24 15:19:49,368 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:49,400 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (development) mode
keycloak-1           | 2024-10-24 15:19:49,400 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to obtain JDBC connection
keycloak-1           | 2024-10-24 15:19:49,400 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:49,400 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: keycloak-postgres
keycloak-1           | 2024-10-24 15:19:49,400 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
keycloak-1           | 2024-10-24 15:19:52,407 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: keycloak.local.exus.ai, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
keycloak-1           | 2024-10-24 15:19:53,884 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
keycloak-1           | 2024-10-24 15:19:54,031 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:54,032 WARN  [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread) HHH000342: Could not obtain connection to query metadata: java.lang.NullPointerException: Cannot invoke "org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(java.sql.SQLException, String)" because the return value of "org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.sqlExceptionHelper()" is null
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doTheWork(JtaIsolationDelegate.java:186)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.lambda$delegateWork$1(JtaIsolationDelegate.java:75)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doInSuspendedTransaction(JtaIsolationDelegate.java:107)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.delegateWork(JtaIsolationDelegate.java:72)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentUsingJdbcMetadata(JdbcEnvironmentInitiator.java:279)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:193)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:69)
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:119)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:264)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:239)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-1           |  at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:52)
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:125)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:248)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-1           |  at org.hibernate.boot.internal.SessionFactoryOptionsBuilder.<init>(SessionFactoryOptionsBuilder.java:273)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.recording.PrevalidatedQuarkusMetadata.buildSessionFactoryOptionsBuilder(PrevalidatedQuarkusMetadata.java:70)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.boot.FastBootEntityManagerFactoryBuilder.build(FastBootEntityManagerFactoryBuilder.java:81)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.FastBootHibernatePersistenceProvider.createEntityManagerFactory(FastBootHibernatePersistenceProvider.java:74)
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:80)
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$LazyPersistenceUnit.get(JPAConfig.java:156)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$1.run(JPAConfig.java:64)
keycloak-1           |  at java.base/java.lang.Thread.run(Thread.java:840)
keycloak-1           | 
keycloak-1           | 2024-10-24 15:19:54,449 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
keycloak-1           | 2024-10-24 15:19:54,504 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
keycloak-1           | 2024-10-24 15:19:54,669 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
keycloak-1           | 2024-10-24 15:19:55,199 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:55,244 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (development) mode
keycloak-1           | 2024-10-24 15:19:55,244 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to obtain JDBC connection
keycloak-1           | 2024-10-24 15:19:55,244 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:55,245 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: keycloak-postgres
keycloak-1           | 2024-10-24 15:19:55,245 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
keycloak-1           | 2024-10-24 15:19:58,275 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: keycloak.local.exus.ai, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
keycloak-1           | 2024-10-24 15:19:59,740 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
keycloak-1           | 2024-10-24 15:19:59,867 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:19:59,867 WARN  [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread) HHH000342: Could not obtain connection to query metadata: java.lang.NullPointerException: Cannot invoke "org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(java.sql.SQLException, String)" because the return value of "org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.sqlExceptionHelper()" is null
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doTheWork(JtaIsolationDelegate.java:186)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.lambda$delegateWork$1(JtaIsolationDelegate.java:75)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doInSuspendedTransaction(JtaIsolationDelegate.java:107)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.delegateWork(JtaIsolationDelegate.java:72)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentUsingJdbcMetadata(JdbcEnvironmentInitiator.java:279)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:193)
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:69)
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:119)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:264)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:239)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-postgres-1  | 
keycloak-1           |  at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:52)
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:125)
keycloak-postgres-1  | 
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:248)
keycloak-postgres-1  | 2024-10-24 15:19:30.033 UTC [1] FATAL:  database files are incompatible with server
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-postgres-1  | 2024-10-24 15:19:30.033 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-1           |  at org.hibernate.boot.internal.SessionFactoryOptionsBuilder.<init>(SessionFactoryOptionsBuilder.java:273)
keycloak-postgres-1  | 
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.recording.PrevalidatedQuarkusMetadata.buildSessionFactoryOptionsBuilder(PrevalidatedQuarkusMetadata.java:70)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.boot.FastBootEntityManagerFactoryBuilder.build(FastBootEntityManagerFactoryBuilder.java:81)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.FastBootHibernatePersistenceProvider.createEntityManagerFactory(FastBootHibernatePersistenceProvider.java:74)
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:80)
keycloak-postgres-1  | 
keycloak-postgres-1  | 2024-10-24 15:19:30.688 UTC [1] FATAL:  database files are incompatible with server
keycloak-postgres-1  | 2024-10-24 15:19:30.688 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$LazyPersistenceUnit.get(JPAConfig.java:156)
keycloak-postgres-1  | 
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$1.run(JPAConfig.java:64)
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-postgres-1  | 
keycloak-postgres-1  | 2024-10-24 15:19:31.323 UTC [1] FATAL:  database files are incompatible with server
keycloak-postgres-1  | 2024-10-24 15:19:31.323 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-postgres-1  | 
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-1           |  at java.base/java.lang.Thread.run(Thread.java:840)
keycloak-postgres-1  | 
keycloak-1           | 
keycloak-postgres-1  | 2024-10-24 15:19:32.196 UTC [1] FATAL:  database files are incompatible with server
keycloak-1           | 2024-10-24 15:20:00,133 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
keycloak-postgres-1  | 2024-10-24 15:19:32.196 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-postgres-1  | 
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-postgres-1  | 
keycloak-1           | 2024-10-24 15:20:00,181 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
keycloak-1           | 2024-10-24 15:20:00,325 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
keycloak-1           | 2024-10-24 15:20:00,851 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:20:00,881 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (development) mode
keycloak-1           | 2024-10-24 15:20:00,881 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to obtain JDBC connection
keycloak-postgres-1  | 2024-10-24 15:19:33.452 UTC [1] FATAL:  database files are incompatible with server
keycloak-postgres-1  | 2024-10-24 15:19:33.452 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-postgres-1  | 
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-postgres-1  | 
keycloak-1           | 2024-10-24 15:20:00,881 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: The connection attempt failed.
keycloak-1           | 2024-10-24 15:20:00,881 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: keycloak-postgres
keycloak-1           | 2024-10-24 15:20:00,881 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
keycloak-1           | 2024-10-24 15:20:04,808 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: keycloak.local.exus.ai, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
keycloak-1           | 2024-10-24 15:20:06,599 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
keycloak-1           | 2024-10-24 15:20:06,723 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:20:06,724 WARN  [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread) HHH000342: Could not obtain connection to query metadata: java.lang.NullPointerException: Cannot invoke "org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(java.sql.SQLException, String)" because the return value of "org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.sqlExceptionHelper()" is null
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doTheWork(JtaIsolationDelegate.java:186)
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.lambda$delegateWork$1(JtaIsolationDelegate.java:75)
keycloak-postgres-1  | 2024-10-24 15:19:35.408 UTC [1] FATAL:  database files are incompatible with server
keycloak-postgres-1  | 2024-10-24 15:19:35.408 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.doInSuspendedTransaction(JtaIsolationDelegate.java:107)
keycloak-postgres-1  | 
keycloak-1           |  at org.hibernate.resource.transaction.backend.jta.internal.JtaIsolationDelegate.delegateWork(JtaIsolationDelegate.java:72)
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-postgres-1  | 
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.getJdbcEnvironmentUsingJdbcMetadata(JdbcEnvironmentInitiator.java:279)
keycloak-postgres-1  | 2024-10-24 15:19:39.069 UTC [1] FATAL:  database files are incompatible with server
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:193)
keycloak-postgres-1  | 2024-10-24 15:19:39.069 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-1           |  at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:69)
keycloak-postgres-1  | 
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:119)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:264)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:239)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-1           |  at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:52)
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-1           |  at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:125)
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:248)
keycloak-postgres-1  | 
keycloak-1           |  at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:216)
keycloak-postgres-1  | 2024-10-24 15:19:45.954 UTC [1] FATAL:  database files are incompatible with server
keycloak-1           |  at org.hibernate.boot.internal.SessionFactoryOptionsBuilder.<init>(SessionFactoryOptionsBuilder.java:273)
keycloak-postgres-1  | 2024-10-24 15:19:45.954 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.recording.PrevalidatedQuarkusMetadata.buildSessionFactoryOptionsBuilder(PrevalidatedQuarkusMetadata.java:70)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.boot.FastBootEntityManagerFactoryBuilder.build(FastBootEntityManagerFactoryBuilder.java:81)
keycloak-postgres-1  | 
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.FastBootHibernatePersistenceProvider.createEntityManagerFactory(FastBootHibernatePersistenceProvider.java:74)
keycloak-postgres-1  | PostgreSQL Database directory appears to contain a database; Skipping initialization
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:80)
keycloak-postgres-1  | 
keycloak-1           |  at jakarta.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
keycloak-postgres-1  | 2024-10-24 15:19:59.216 UTC [1] FATAL:  database files are incompatible with server
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$LazyPersistenceUnit.get(JPAConfig.java:156)
keycloak-1           |  at io.quarkus.hibernate.orm.runtime.JPAConfig$1.run(JPAConfig.java:64)
keycloak-1           |  at java.base/java.lang.Thread.run(Thread.java:840)
keycloak-1           | 
keycloak-1           | 2024-10-24 15:20:07,367 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
keycloak-1           | 2024-10-24 15:20:07,408 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
keycloak-1           | 2024-10-24 15:20:07,535 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
keycloak-postgres-1  | 2024-10-24 15:19:59.216 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 14, which is not compatible with this version 16.1 (Debian 16.1-1.pgdg120+1).
keycloak-1           | 2024-10-24 15:20:08,156 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': The connection attempt failed.
keycloak-1           | 2024-10-24 15:20:08,236 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to start server in (development) mode
keycloak-1           | 2024-10-24 15:20:08,236 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: Failed to obtain JDBC connection
keycloak-1           | 2024-10-24 15:20:08,236 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: The connection attempt failed.
keycloak-1           | 2024-10-24 15:20:08,237 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) ERROR: keycloak-postgres
keycloak-1           | 2024-10-24 15:20:08,237 ERROR [org.keycloak.quarkus.runtime.cli.ExecutionExceptionHandler] (main) For more details run the same command passing the '--verbose' option. Also you can use '--help' to see the details about the usage of the particular command.
Gracefully stopping... (press Ctrl+C again to force)
dependency failed to start: container mammoth_kk-keycloak-1 is unhealthy

[georgiosn]

Thanks for the log, for this I will need some time and I will reply.

[georgiosn]

Hello as I see it again, you have to start clean so the commands you should run are

docker compose down -v

then

docker compose up --build

[sudhang]

I had tried it with --build, yes.

But I am getting the following:

(base) sud@sudora:~/neben/mammoth-toolkit-releases$ docker compose up --build
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
>>>> Executing external compose provider "/usr/bin/podman-compose". Please see podman-compose(1) for how to disable this message. <<<<

e4183defb4a0587fccea374f3b37a11e7bfe4c7f7745530f9561f21f4fef1609
00347a4b5d708e736d023ad3f2c5bf35a208b66408f2fa8774b0f9416fd5aa8c
7315f0d5dab07c04631c2189d34bf9c86cc2fb644ccc8e0e5c865a899a31d7f8
194a082315f7bfcc42fa49a807c0592af8ec84d96627beeeaba68a42193ad3c5
a265efd2fbc90b8deb3f1c8b598dd40ee7cc8076cf2f1702be8d8dc0fd795318
e6a93ce26f66acc45cc12767d21bbeed8dcb3355c958a406c16284d3ab38cecb
1858faf2a1e2f0685e0db1450372200ed67276759f708bbdb4cf56bbb1fa8bad
0187f3f53f9bde2521ba11e9c03bc4d33ff8f8f82f1433367a4be415725a301b
[keycloak-postgres] | The files belonging to this database system will be owned by user "postgres".
[keycloak-postgres] | This user must also own the server process.
[keycloak-postgres] | 
[keycloak-postgres] | The database cluster will be initialized with locale "en_US.utf8".
[keycloak-postgres] | The default database encoding has accordingly been set to "UTF8".
[keycloak-postgres] | The default text search configuration will be set to "english".
[keycloak-postgres] | 
[keycloak-postgres] | Data page checksums are disabled.
[keycloak-postgres] | 
[keycloak-postgres] | fixing permissions on existing directory /var/lib/postgresql/data ... ok
[keycloak-postgres] | creating subdirectories ... ok
[keycloak-postgres] | selecting dynamic shared memory implementation ... posix
[keycloak-postgres] | selecting default max_connections ... 100
[keycloak-postgres] | selecting default shared_buffers ... 128MB
[keycloak-postgres] | selecting default time zone ... Etc/UTC
[keycloak-postgres] | creating configuration files ... ok
[api-db]            | cannot open `/run/user/1000/crun/194a082315f7bfcc42fa49a807c0592af8ec84d96627beeeaba68a42193ad3c5/exec.fifo`: No such file or directory
[api-db]            | Error: unable to start container 194a082315f7bfcc42fa49a807c0592af8ec84d96627beeeaba68a42193ad3c5: `/usr/bin/crun start 194a082315f7bfcc42fa49a807c0592af8ec84d96627beeeaba68a42193ad3c5` failed: exit status 1
[mammoth-fe]        | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
[mammoth-fe]        | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
[mammoth-fe]        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
[mammoth-fe]        | 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
[mammoth-fe]        | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
[mammoth-fe]        | /docker-entrypoint.sh: Sourcing /docker-entrypoint.d/15-local-resolvers.envsh
[mammoth-fe]        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
[mammoth-fe]        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
[mammoth-fe]        | /docker-entrypoint.sh: Configuration complete; ready for start up
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: using the "epoll" event method
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: nginx/1.25.5
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: built by gcc 13.2.1 20231014 (Alpine 13.2.1_git20231014) 
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: OS: Linux 6.11.3-200.fc40.x86_64
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 524288:524288
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker processes
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 24
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 25
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 26
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 27
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 28
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 29
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 30
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 31
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 32
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 33
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 34
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 35
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 36
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 37
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 38
[mammoth-fe]        | 2024/10/24 15:25:36 [notice] 1#1: start worker process 39
[keycloak-postgres] | running bootstrap script ... ok
[krakend]           | Watching changes on files /etc/krakend/
[krakend]           | Ignoring saves to file out.json
[krakend]           | [00] Starting service
[krakend]           | [00] Parsing configuration file: /etc/krakend/krakend.json
[krakend]           | [00] 2024/10/24 15:25:36 KRAKEND INFO: Starting KrakenD v2.7.2
[krakend]           | [00] 2024/10/24 15:25:36 KRAKEND INFO: Working directory is /etc/krakend
[krakend]           | [00] 2024/10/24 15:25:36 KRAKEND DEBUG: [SERVICE: Gin] Debug enabled
[krakend]           | [00] 2024/10/24 15:25:36 KRAKEND INFO: Starting the KrakenD instance
[krakend]           | [00] 2024/10/24 15:25:36 KRAKEND INFO: [SERVICE: Gin] Listening on port: 8080
[adminer]           | [Thu Oct 24 15:25:36 2024] PHP 7.4.33 Development Server (http://[::]:8080) started
[keycloak-postgres] | performing post-bootstrap initialization ... ok
[api]               | Error: unable to start container 0187f3f53f9bde2521ba11e9c03bc4d33ff8f8f82f1433367a4be415725a301b: failed to create new hosts file: unable to replace "host-gateway" of host entry "host.docker.internal:host-gateway": host containers internal IP address is empty
[keycloak-postgres] | syncing data to disk ... ok
[keycloak-postgres] | 
[keycloak-postgres] | 
[keycloak-postgres] | Success. You can now start the database server using:
[keycloak-postgres] | 
[keycloak-postgres] |     pg_ctl -D /var/lib/postgresql/data -l logfile start
[keycloak-postgres] | 
[keycloak-postgres] | initdb: warning: enabling "trust" authentication for local connections
[keycloak-postgres] | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
[keycloak-postgres] | waiting for server to start....2024-10-24 15:25:37.791 UTC [43] LOG:  starting PostgreSQL 16.1 (Debian 16.1-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
[keycloak-postgres] | 2024-10-24 15:25:37.793 UTC [43] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
[keycloak-postgres] | 2024-10-24 15:25:37.799 UTC [46] LOG:  database system was shut down at 2024-10-24 15:25:36 UTC
[keycloak-postgres] | 2024-10-24 15:25:37.807 UTC [43] LOG:  database system is ready to accept connections
[keycloak-postgres] |  done
[keycloak-postgres] | server started
[keycloak]          | Updating the configuration and installing your custom providers, if any. Please wait.
[keycloak-postgres] | CREATE DATABASE
[keycloak-postgres] | 
[keycloak-postgres] | 
[keycloak-postgres] | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
[keycloak-postgres] | 
[keycloak-postgres] | waiting for server to shut down....2024-10-24 15:25:38.031 UTC [43] LOG:  received fast shutdown request
[keycloak-postgres] | 2024-10-24 15:25:38.032 UTC [43] LOG:  aborting any active transactions
[keycloak-postgres] | 2024-10-24 15:25:38.036 UTC [43] LOG:  background worker "logical replication launcher" (PID 49) exited with exit code 1
[keycloak-postgres] | 2024-10-24 15:25:38.036 UTC [44] LOG:  shutting down
[keycloak-postgres] | 2024-10-24 15:25:38.038 UTC [44] LOG:  checkpoint starting: shutdown immediate
[keycloak-postgres] | 2024-10-24 15:25:38.429 UTC [44] LOG:  checkpoint complete: wrote 923 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.063 s, sync=0.320 s, total=0.394 s; sync files=301, longest=0.002 s, average=0.002 s; distance=4257 kB, estimate=4257 kB; lsn=0/1913080, redo lsn=0/1913080
[keycloak-postgres] | 2024-10-24 15:25:38.443 UTC [43] LOG:  database system is shut down
[keycloak-postgres] |  done
[keycloak-postgres] | server stopped
[keycloak-postgres] | 
[keycloak-postgres] | PostgreSQL init process complete; ready for start up.
[keycloak-postgres] | 
[keycloak-postgres] | 2024-10-24 15:25:38.561 UTC [1] LOG:  starting PostgreSQL 16.1 (Debian 16.1-1.pgdg120+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
[keycloak-postgres] | 2024-10-24 15:25:38.562 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
[keycloak-postgres] | 2024-10-24 15:25:38.562 UTC [1] LOG:  listening on IPv6 address "::", port 5432
[keycloak-postgres] | 2024-10-24 15:25:38.564 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
[keycloak-postgres] | 2024-10-24 15:25:38.570 UTC [59] LOG:  database system was shut down at 2024-10-24 15:25:38 UTC
[keycloak-postgres] | 2024-10-24 15:25:38.578 UTC [1] LOG:  database system is ready to accept connections
[keycloak]          | 2024-10-24 15:25:44,915 INFO  [io.quarkus.deployment.QuarkusAugmentor] (main) Quarkus augmentation completed in 5800ms
[keycloak]          | 2024-10-24 15:25:46,351 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: Base URL: <unset>, Hostname: keycloak.local.exus.ai, Strict HTTPS: false, Path: <request>, Strict BackChannel: false, Admin URL: <unset>, Admin: <request>, Port: -1, Proxied: false
[keycloak]          | 2024-10-24 15:25:48,137 WARN  [io.quarkus.agroal.runtime.DataSources] (main) Datasource <default> enables XA but transaction recovery is not enabled. Please enable transaction recovery by setting quarkus.transaction-manager.enable-recovery=true, otherwise data may be lost if the application is terminated abruptly
[keycloak]          | 2024-10-24 15:25:48,728 WARN  [org.infinispan.PERSISTENCE] (keycloak-cache-init) ISPN000554: jboss-marshalling is deprecated and planned for removal
[keycloak]          | 2024-10-24 15:25:48,787 WARN  [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000569: Unable to persist Infinispan internal caches as no global state enabled
[keycloak]          | 2024-10-24 15:25:48,887 INFO  [org.infinispan.CONTAINER] (keycloak-cache-init) ISPN000556: Starting user marshaller 'org.infinispan.jboss.marshalling.core.JBossUserMarshaller'
[keycloak]          | 2024-10-24 15:25:49,425 INFO  [org.keycloak.connections.infinispan.DefaultInfinispanConnectionProviderFactory] (main) Node name: node_411610, Site name: null
[keycloak]          | 2024-10-24 15:25:49,430 INFO  [org.keycloak.broker.provider.AbstractIdentityProviderMapper] (main) Registering class org.keycloak.broker.provider.mappersync.ConfigSyncEventListener
[keycloak-postgres] | 2024-10-24 15:25:49.443 UTC [63] ERROR:  relation "migration_model" does not exist at character 25
[keycloak-postgres] | 2024-10-24 15:25:49.443 UTC [63] STATEMENT:  SELECT ID, VERSION FROM MIGRATION_MODEL ORDER BY UPDATE_TIME DESC
[keycloak-postgres] | 2024-10-24 15:25:51.943 UTC [63] ERROR:  relation "public.databasechangelog" does not exist at character 22
[keycloak-postgres] | 2024-10-24 15:25:51.943 UTC [63] STATEMENT:  SELECT COUNT(*) FROM public.databasechangelog
[keycloak-postgres] | 2024-10-24 15:25:52.240 UTC [64] ERROR:  relation "public.databasechangeloglock" does not exist at character 22
[keycloak-postgres] | 2024-10-24 15:25:52.240 UTC [64] STATEMENT:  SELECT COUNT(*) FROM public.databasechangeloglock
[keycloak-postgres] | 2024-10-24 15:25:52.292 UTC [63] ERROR:  relation "public.databasechangelog" does not exist at character 22
[keycloak-postgres] | 2024-10-24 15:25:52.292 UTC [63] STATEMENT:  SELECT COUNT(*) FROM public.databasechangelog
[keycloak]          | 2024-10-24 15:25:52,294 INFO  [org.keycloak.quarkus.runtime.storage.legacy.liquibase.QuarkusJpaUpdaterProvider] (main) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
[keycloak]          | 
[keycloak]          | UPDATE SUMMARY
[keycloak]          | Run:                        117
[keycloak]          | Previously run:               0
[keycloak]          | Filtered out:                 0
[keycloak]          | -------------------------------
[keycloak]          | Total change sets:          117
[keycloak]          | 
[keycloak]          | 2024-10-24 15:25:56,530 INFO  [org.keycloak.services] (main) KC-SERVICES0050: Initializing master realm
[keycloak]          | 2024-10-24 15:25:57,990 INFO  [org.keycloak.exportimport.dir.DirImportProvider] (main) Importing from directory /opt/keycloak/bin/../data/import
[keycloak]          | 2024-10-24 15:25:57,990 INFO  [org.keycloak.services] (main) KC-SERVICES0030: Full model import requested. Strategy: IGNORE_EXISTING
[keycloak]          | 2024-10-24 15:25:57,990 INFO  [org.keycloak.services] (main) KC-SERVICES0032: Import finished successfully
[keycloak]          | 2024-10-24 15:25:58,272 INFO  [io.quarkus] (main) Keycloak 23.0.1 on JVM (powered by Quarkus 3.2.9.Final) started in 13.195s. Listening on: http://0.0.0.0:8080 and https://0.0.0.0:8443
[keycloak]          | 2024-10-24 15:25:58,272 INFO  [io.quarkus] (main) Profile dev activated. 
[keycloak]          | 2024-10-24 15:25:58,272 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-h2, jdbc-mariadb, jdbc-mssql, jdbc-mysql, jdbc-oracle, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
[keycloak]          | 2024-10-24 15:25:58,695 INFO  [org.keycloak.services] (main) KC-SERVICES0009: Added user 'kc_admin' to realm 'master'
[keycloak]          | 2024-10-24 15:25:58,703 WARN  [org.keycloak.quarkus.runtime.KeycloakMain] (main) Running the server in development mode. DO NOT use this configuration in production.

[keycloak-postgres] | 2024-10-24 15:30:38.638 UTC [57] LOG:  checkpoint starting: time
[keycloak-postgres] | 2024-10-24 15:31:33.048 UTC [57] LOG:  checkpoint complete: wrote 545 buffers (3.3%); 1 WAL file(s) added, 0 removed, 0 recycled; write=54.269 s, sync=0.107 s, total=54.410 s; sync files=548, longest=0.010 s, average=0.001 s; distance=3813 kB, estimate=3813 kB; lsn=0/1CCC5A0, redo lsn=0/1CCC568

I think the following part is the suspicious. I thought it was wrong on my part, but I am not sure

[keycloak-postgres] | performing post-bootstrap initialization ... ok
[api]               | Error: unable to start container 0187f3f53f9bde2521ba11e9c03bc4d33ff8f8f82f1433367a4be415725a301b: failed to create new hosts file: unable to replace "host-gateway" of host entry "host.docker.internal:host-gateway": host containers internal IP address is empty

Also the following:

[keycloak-postgres] | 2024-10-24 15:25:49.443 UTC [63] ERROR:  relation "migration_model" does not exist at character 25
[keycloak-postgres] | 2024-10-24 15:25:49.443 UTC [63] STATEMENT:  SELECT ID, VERSION FROM MIGRATION_MODEL ORDER BY UPDATE_TIME DESC
[keycloak-postgres] | 2024-10-24 15:25:51.943 UTC [63] ERROR:  relation "public.databasechangelog" does not exist at character 22
[keycloak-postgres] | 2024-10-24 15:25:51.943 UTC [63] STATEMENT:  SELECT COUNT(*) FROM public.databasechangelog
[keycloak-postgres] | 2024-10-24 15:25:52.240 UTC [64] ERROR:  relation "public.databasechangeloglock" does not exist at character 22
[keycloak-postgres] | 2024-10-24 15:25:52.240 UTC [64] STATEMENT:  SELECT COUNT(*) FROM public.databasechangeloglock
[keycloak-postgres] | 2024-10-24 15:25:52.292 UTC [63] ERROR:  relation "public.databasechangelog" does not exist at character 22
[keycloak-postgres] | 2024-10-24 15:25:52.292 UTC [63] STATEMENT:  SELECT COUNT(*) FROM public.databasechangelog
[keycloak]          | 2024-10-24 15:25:52,294 INFO  [org.keycloak.quarkus.runtime.storage.legacy.liquibase.QuarkusJpaUpdaterProvider] (main) Initializing database schema. Using changelog META-INF/jpa-changelog-master.xml
[keycloak]          | 

I am not able to get the kfp url to work now. I checked, and I am using version 15 of the image

[georgiosn]

Please login with Demo user using username: demo password: demo

[elmokhtar-mohamedmoussa-idnow]

Yes I was under master and not toolkit (sorry). Now I have a 502 error

logs:

pi-1                | 2024-10-24 15:38:33,903 INFO sqlalchemy.engine.Engine BEGIN (implicit)
api-1                | 2024-10-24 15:38:33,904 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
api-1                | FROM wizard_options 
api-1                | WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
api-1                | 2024-10-24 15:38:33,904 INFO sqlalchemy.engine.Engine [cached since 53.51s ago] {'conf_id_1': 'bb261607-6f24-44bc-82ef-b0a408ac03e6'}
api-1                | 2024-10-24 15:38:33,906 INFO sqlalchemy.engine.Engine ROLLBACK
api-1                | INFO:     172.20.0.4:51532 - "GET /wizard/databias/overview/bb261607-6f24-44bc-82ef-b0a408ac03e6 HTTP/1.1" 200 OK
krakend-1            | [00] [GIN] 2024/10/24 - 15:38:33 | 200 |   20.365795ms |      172.19.0.1 | GET      "/wizard/databias/overview/bb261607-6f24-44bc-82ef-b0a408ac03e6"
krakend-1            | [00] 2024/10/24 15:38:35 KRAKEND DEBUG: [CORS] 2024/10/24 15:38:35 Handler: Preflight request
krakend-1            | [00] ders: map[Access-Control-Allow-Origin:[http://localhost:5173] Access-Control-Expose-Headers:[Content-Length, Content-Type] Vary:[Origin]]
krakend-1            | [00]  Access-Control-Max-Age:[43200] Vary:[Origin, Access-Control-Request-Method, Access-Control-Request-Headers]]
krakend-1            | [00] uest-Headers]]
krakend-1            | [00] 
krakend-1            | [00] 2024/10/24 15:38:35 KRAKEND DEBUG: [CORS] 2024/10/24 15:38:35   Preflight response headers: map[Access-Control-Allow-Headers:[Authorization] Access-Control-Allow-Methods:[GET] Access-Control-Allow-Origin:[http://localhost:5173] Access-Control-Max-Age:[43200] Vary:[Origin, Access-Control-Request-Method, Access-Control-Request-Headers]]
krakend-1            | [00] uest-Headers]]
krakend-1            | [00] 
krakend-1            | [00] 2024/10/24 15:38:35 KRAKEND DEBUG: [CORS] 2024/10/24 15:38:35 Handler: Actual request
krakend-1            | [00] ders: map[Access-Control-Allow-Headers:[Authorization] Access-Control-Allow-Methods:[GET] Access-Control-Allow-Origin:[http://localhost:5173] Access-Control-Max-Age:[43200] Vary:[Origin, Access-Control-Request-Method, Access-Control-Request-Headers]]
krakend-1            | [00] uest-Headers]]
krakend-1            | [00] 
krakend-1            | [00] 2024/10/24 15:38:35 KRAKEND DEBUG: [CORS] 2024/10/24 15:38:35   Actual response added headers: map[Access-Control-Allow-Origin:[http://localhost:5173] Access-Control-Expose-Headers:[Content-Length, Content-Type] Vary:[Origin]]
krakend-1            | [00]  Access-Control-Max-Age:[43200] Vary:[Origin, Access-Control-Request-Method, Access-Control-Request-Headers]]
krakend-1            | [00] uest-Headers]]
krakend-1            | [00] 
api-1                | 2024-10-24 15:38:35,386 INFO sqlalchemy.engine.Engine BEGIN (implicit)
api-1                | 2024-10-24 15:38:35,387 INFO sqlalchemy.engine.Engine SELECT wizard_options.id, wizard_options.user_id, wizard_options.conf_id, wizard_options.configuration 
api-1                | FROM wizard_options 
api-1                | WHERE wizard_options.conf_id = %(conf_id_1)s::UUID
api-1                | 2024-10-24 15:38:35,387 INFO sqlalchemy.engine.Engine [cached since 54.99s ago] {'conf_id_1': 'bb261607-6f24-44bc-82ef-b0a408ac03e6'}
api-1                | ACTION INFO FOR PIPELINES
api-1                | result_id='STATUS_OK' result_message='OK' is_error=False status_code=None uuid='bb261607-6f24-44bc-82ef-b0a408ac03e6' name='test' group='test' run_status=None run_id=None run_type=<RunType.MODEL_BIAS: 'MODEL_BIAS'> run_start_time=None run_end_time=None attributes=['marital'] loader_model=ComponentSelection(id='model_onnx', parameters_value={'path': 'http://host.k3d.internal:5000/model.onnx'}) loader_data=ComponentSelection(id='data_auto_csv', parameters_value={'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}) domain='finance' metrics=[ComponentSelection(id='interactive_report', parameters_value={}), ComponentSelection(id='model_card', parameters_value={})] url_data='' url_model='' run_artifacts=None step=4
api-1                | /opt/pysetup/.venv/lib/python3.12/site-packages/kfp/client/client.py:159: FutureWarning: This client only works with Kubeflow Pipeline v2.0.0-beta.2 and later versions.
api-1                |   warnings.warn(
api-1                | File Import Contents: from typing import Dict, List
api-1                | from kfp import compiler, components, dsl
api-1                | 
api-1                | model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
api-1                | data_auto_csv = components.load_component_from_file("/var/lib/components_yaml/data_auto_csv.yaml")
api-1                | interactive_report = components.load_component_from_file("/var/lib/components_yaml/interactive_report.yaml")
api-1                | 
api-1                | model_card = components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")
api-1                | 
api-1                | 
api-1                | Component path is: /var/lib/components_yaml/model_onnx.yaml
api-1                | Loading component from path: /var/lib/components_yaml/model_onnx.yaml
api-1                | Component path is: /var/lib/components_yaml/data_auto_csv.yaml
api-1                | Loading component from path: /var/lib/components_yaml/data_auto_csv.yaml
api-1                | Component path is: /var/lib/components_yaml/interactive_report.yaml
api-1                | Loading component from path: /var/lib/components_yaml/interactive_report.yaml
api-1                | Component path is: /var/lib/components_yaml/model_card.yaml
api-1                | Loading component from path: /var/lib/components_yaml/model_card.yaml
api-1                | Components list size: 4
api-1                | File Import Contents: from typing import Dict, List
api-1                | from kfp import compiler, components, dsl
api-1                | 
api-1                | model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
api-1                | data_auto_csv = components.load_component_from_file("/var/lib/components_yaml/data_auto_csv.yaml")
api-1                | interactive_report = components.load_component_from_file("/var/lib/components_yaml/interactive_report.yaml")
api-1                | 
api-1                | model_card = components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")
api-1                | 
api-1                | 
api-1                | Pipeline function: @dsl.pipeline(
api-1                |  name='test'
api-1                | )
api-1                | def pipeline(model_onnx__params:Dict, data_auto_csv__params:Dict, sensitive:List, interactive_report__params:Dict, sensitive:List, model_card__params:Dict):
api-1                |  model_onnx_task = model_onnx(model_onnx__params=model_onnx__params)
api-1                |  data_auto_csv_task = data_auto_csv(data_auto_csv__params=data_auto_csv__params)
api-1                |  interactive_report_task = interactive_report(interactive_report__params=interactive_report__params, sensitive=sensitive, dataset=data_auto_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])
api-1                |  model_card_task = model_card(model_card__params=model_card__params, sensitive=sensitive, dataset=data_auto_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])
api-1                | 
api-1                | Pipeline TOTAL content: from typing import Dict, List
api-1                | from kfp import compiler, components, dsl
api-1                | 
api-1                | model_onnx = components.load_component_from_file("/var/lib/components_yaml/model_onnx.yaml")
api-1                | data_auto_csv = components.load_component_from_file("/var/lib/components_yaml/data_auto_csv.yaml")
api-1                | interactive_report = components.load_component_from_file("/var/lib/components_yaml/interactive_report.yaml")
api-1                | 
api-1                | model_card = components.load_component_from_file("/var/lib/components_yaml/model_card.yaml")
api-1                | 
api-1                | @dsl.pipeline(
api-1                |  name='test'
api-1                | )
api-1                | def pipeline(model_onnx__params:Dict, data_auto_csv__params:Dict, sensitive:List, interactive_report__params:Dict, sensitive:List, model_card__params:Dict):
api-1                |  model_onnx_task = model_onnx(model_onnx__params=model_onnx__params)
api-1                |  data_auto_csv_task = data_auto_csv(data_auto_csv__params=data_auto_csv__params)
api-1                |  interactive_report_task = interactive_report(interactive_report__params=interactive_report__params, sensitive=sensitive, dataset=data_auto_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])
api-1                |  model_card_task = model_card(model_card__params=model_card__params, sensitive=sensitive, dataset=data_auto_csv_task.outputs['output'], model=model_onnx_task.outputs['output'])
api-1                | 
api-1                | 
api-1                | compiler.Compiler().compile(pipeline, "/var/lib/pipelines/pipeline.yaml")
api-1                | Pipeline arguments
api-1                | {'sensitive': ['marital'], 'model_onnx__params': {'path': 'http://host.k3d.internal:5000/model.onnx'}, 'data_auto_csv__params': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}, 'interactive_report__params': {}, 'model_card__params': {}}
api-1                |   File "/var/lib/pipelines/pipeline.py", line 13
api-1                |     def pipeline(model_onnx__params:Dict, data_auto_csv__params:Dict, sensitive:List, interactive_report__params:Dict, sensitive:List, model_card__params:Dict):
api-1                |                                                                                                                        ^^^^^^^^^^^^^^
api-1                | SyntaxError: duplicate argument 'sensitive' in function definition
api-1                | Error running pipeline COMMAND LINE
api-1                | GENERATED ACTION INFO FOR PIPELINES
api-1                | {'sensitive': ['marital'], 'data_auto_csv__params': {'categorical': ['job', 'marital', 'education', 'default', 'housing', 'loan', 'contact', 'poutcome'], 'delimiter': ';', 'labels': 'y', 'numeric': ['age', 'duration', 'campaign', 'pdays', 'previous'], 'on_bad_lines': 'skip', 'path': 'http://host.k3d.internal:5000/bank.csv'}, 'model_onnx__params': {'path': 'http://host.k3d.internal:5000/model.onnx'}, 'interactive_report__params': {}, 'model_card__params': {}}
api-1                | Pipeline yaml path: /var/lib/pipelines/pipeline.yaml
api-1                | Experiment details: http://kfp.local.exus.ai:8082/#/experiments/details/f530c0d5-5262-4b2c-9bae-fea04a0ec325
api-1                | Error: Cannot submit the pipeline for execution.
api-1                | UnboundLocalError("cannot access local variable 'result' where it is not associated with a value")
api-1                | 2024-10-24 15:38:35,607 INFO sqlalchemy.engine.Engine ROLLBACK
api-1                | INFO:sqlalchemy.engine.Engine:ROLLBACK
api-1                | INFO:     172.20.0.4:51532 - "GET /wizard/databias/submit/bb261607-6f24-44bc-82ef-b0a408ac03e6 HTTP/1.1" 502 Bad Gateway
krakend-1            | [00] 2024/10/24 15:38:35 KRAKEND ERROR: [ENDPOINT: /wizard/databias/submit/:uuid] {"detail":"Error submitting pipeline: UnboundLocalError(\"cannot access local variable 'result' where it is not associated with a value\")"}
krakend-1            | [00] [GIN] 2024/10/24 - 15:38:35 | 502 |  232.687101ms |      172.19.0.1 | GET      "/wizard/databias/submit/bb261607-6f24-44bc-82ef-b0a408ac03e6"
krakend-1            | [00] Error #01: {"detail":"Error submitting pipeline: UnboundLocalError(\"cannot access local variable 'result' where it is not associated with a value\")"}

[georgiosn]

@sudhang

There is an issue starting the api as you wrote

Specific this that you mentioned

[keycloak-postgres] | performing post-bootstrap initialization ... ok [api] | Error: unable to start container 0187f3f53f9bde2521ba11e9c03bc4d33ff8f8f82f1433367a4be415725a301b: failed to create new hosts file: unable to replace "host-gateway" of host entry "host.docker.internal:host-gateway": host containers internal IP address is empty

This should be related to your OS most probably.

[georgiosn]

@elmokhtar-mohamedmoussa-idnow this needs more investigation.

[georgiosn]

@elmokhtar-mohamedmoussa-idnow thanks for your detailed log for the issue.

We found the cause and we fixed it. A new version is now available, it also includes some extra features.

Please do a git pull and docker compose up --build