search

datahub-project / datahub

The Metadata Platform for your Data and AI Stack
https://datahubproject.io
Apache License 2.0
9.89k stars 2.93k forks source link

cli not creting igestion into IU #8625

Closed AntonChernyaevArammeem closed 1 year ago

AntonChernyaevArammeem commented 1 year ago

env kubernetes 1.24/helm datahub 10.4 (cant't update to 10.5 a lot of ussiues but it's not current issue) Created igestion on sandbox all running well in production not. Only one [CLI] igestions into list into UI All others CLI ingestion through kubernetes jobs not show after apply If i remove one as currently present into UI and run again kubernetes job - will see this ingestion.

Could someone helps with this?)

ERRORS


2023-08-14 14:56:46,086 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:44 - Failed to feed bulk request. Number of events: 453 Took time ms: -1 Message: failure in bulk execution:
2023-08-14 15:30:25,031 [qtp522764626-242] ERROR c.l.m.filter.RestliLoggingFilter:38 - Rest.li error:
Caused by: java.sql.BatchUpdateException: Batch entry 0 update metadata_aspect_v2 set metadata='{"name":"DMF_every_1hour_30min_load","description":"None\n\n","externalUrl":"https://xxxxxxxxxxxxxxxxx.c6.eu-central-1.airflow.amazonaws.com/tree?dag_id=DMF_every_1hour_30min_load","customProperties":{"_access_control":"None","timezone":"Timezone(''UTC'')","is_paused_upon_creation":"None","catchup":"False","_default_view":"''grid''","fileloc":"''/usr/local/airflow/dags/dmf_generator_script.py''","tags":"[]","start_date":"None"}}', createdOn='2023-07-13 11:30:31.593+00', createdBy='urn:li:corpuser:datahub', createdFor=NULL, systemmetadata='{"registryVersion":"0.0.0.0-dev","runId":"no-run-id-provided","registryName":"unknownRegistry","lastObserved":1692027025004}' where urn='urn:li:dataFlow:(airflow,DMF_every_1hour_30min_load,prod)' and aspect='dataFlowInfo' and version=0 was aborted: ERROR: could not serialize access due to concurrent update  Call getNextException to see other errors in the batch.
Caused by: org.postgresql.util.PSQLException: ERROR: could not serialize access due to concurrent update
root@anton-book:/mnt/l/git/amr/backend-sensitive# kubectl logs -n datahub datahub-datahub-gms-5946857bc5-6lpbl  | grep ERR -A2
2023-08-14 14:56:46,086 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:44 - Failed to feed bulk request. Number of events: 453 Took time ms: -1 Message: failure in bulk execution:
[258]: index [datasetindex_v2_1690878237602], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Aredshift%2Cdev.analytics.fct_reppayment__driver_order_corrections%2CPROD%29], message [[datasetindex_v2_1690878237602/2qFyFexRSDGJbHGLffbGgw][[datasetindex_v2_1690878237602][0]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Aredshift%2Cdev.analytics.fct_reppayment__driver_order_corrections%2CPROD%29]: document missing]]]
2023-08-14 14:56:47,014 [I/O dispatcher 1] INFO  c.l.m.s.e.update.BulkListener:47 - Successfully fed bulk request. Number of events: 515 Took time ms: -1
--
2023-08-14 15:30:25,031 [qtp522764626-242] ERROR c.l.m.filter.RestliLoggingFilter:38 - Rest.li error:
com.linkedin.restli.server.RestLiServiceException: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?
        at com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)
--
Caused by: java.sql.BatchUpdateException: Batch entry 0 update metadata_aspect_v2 set metadata='{"name":"DMF_every_1hour_30min_load","description":"None\n\n","externalUrl":"https://xxxxxxxxxxxxxxx.c6.eu-central-1.airflow.amazonaws.com/tree?dag_id=DMF_every_1hour_30min_load","customProperties":{"_access_control":"None","timezone":"Timezone(''UTC'')","is_paused_upon_creation":"None","catchup":"False","_default_view":"''grid''","fileloc":"''/usr/local/airflow/dags/dmf_generator_script.py''","tags":"[]","start_date":"None"}}', createdOn='2023-07-13 11:30:31.593+00', createdBy='urn:li:corpuser:datahub', createdFor=NULL, systemmetadata='{"registryVersion":"0.0.0.0-dev","runId":"no-run-id-provided","registryName":"unknownRegistry","lastObserved":1692027025004}' where urn='urn:li:dataFlow:(airflow,DMF_every_1hour_30min_load,prod)' and aspect='dataFlowInfo' and version=0 was aborted: ERROR: could not serialize access due to concurrent update  Call getNextException to see other errors in the batch.
        at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:165)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
--
Caused by: org.postgresql.util.PSQLException: ERROR: could not serialize access due to concurrent update
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
root@anton-book:/mnt/l/git/amr/backend-sensitive# kubectl logs -n datahub datahub-datahub-gms-5946857bc5-6lpbl  | grep ERR -A7
2023-08-14 14:56:46,086 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:44 - Failed to feed bulk request. Number of events: 453 Took time ms: -1 Message: failure in bulk execution:
[258]: index [datasetindex_v2_1690878237602], type [_doc], id [urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Aredshift%2Cdev.analytics.fct_reppayment__driver_order_corrections%2CPROD%29], message [[datasetindex_v2_1690878237602/2qFyFexRSDGJbHGLffbGgw][[datasetindex_v2_1690878237602][0]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Aredshift%2Cdev.analytics.fct_reppayment__driver_order_corrections%2CPROD%29]: document missing]]]
2023-08-14 14:56:47,014 [I/O dispatcher 1] INFO  c.l.m.s.e.update.BulkListener:47 - Successfully fed bulk request. Number of events: 515 Took time ms: -1
2023-08-14 14:56:47,017 [pool-11-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 3552ms
2023-08-14 14:56:47,017 [pool-11-thread-4] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 3551ms
2023-08-14 14:56:47,017 [pool-11-thread-9] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 3557ms
2023-08-14 14:56:47,017 [pool-11-thread-6] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 3557ms
2023-08-14 14:56:47,017 [pool-11-thread-1] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 3546ms
--
2023-08-14 15:30:25,031 [qtp522764626-242] ERROR c.l.m.filter.RestliLoggingFilter:38 - Rest.li error:
com.linkedin.restli.server.RestLiServiceException: javax.persistence.PersistenceException: Error when batch flush on sql: update metadata_aspect_v2 set metadata=?, createdOn=?, createdBy=?, createdFor=?, systemmetadata=? where urn=? and aspect=? and version=?
        at com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:42)
        at com.linkedin.metadata.restli.RestliUtil.toTask(RestliUtil.java:50)
        at com.linkedin.metadata.resources.entity.AspectResource.ingestProposal(AspectResource.java:191)
        at jdk.internal.reflect.GeneratedMethodAccessor136.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
--
Caused by: java.sql.BatchUpdateException: Batch entry 0 update metadata_aspect_v2 set metadata='{"name":"DMF_every_1hour_30min_load","description":"None\n\n","externalUrl":"https://xxxxxxxxxxxxxxxxxxxxxxx.c6.eu-central-1.airflow.amazonaws.com/tree?dag_id=DMF_every_1hour_30min_load","customProperties":{"_access_control":"None","timezone":"Timezone(''UTC'')","is_paused_upon_creation":"None","catchup":"False","_default_view":"''grid''","fileloc":"''/usr/local/airflow/dags/dmf_generator_script.py''","tags":"[]","start_date":"None"}}', createdOn='2023-07-13 11:30:31.593+00', createdBy='urn:li:corpuser:datahub', createdFor=NULL, systemmetadata='{"registryVersion":"0.0.0.0-dev","runId":"no-run-id-provided","registryName":"unknownRegistry","lastObserved":1692027025004}' where urn='urn:li:dataFlow:(airflow,DMF_every_1hour_30min_load,prod)' and aspect='dataFlowInfo' and version=0 was aborted: ERROR: could not serialize access due to concurrent update  Call getNextException to see other errors in the batch.
        at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:165)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:559)
        at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:887)
        at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:910)
        at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1649)
        at io.ebean.datasource.delegate.PreparedStatementDelegator.executeBatch(PreparedStatementDelegator.java:357)
--
Caused by: org.postgresql.util.PSQLException: ERROR: could not serialize access due to concurrent update
        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2675)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2365)
        ... 115 common frames omitted
2023-08-14 15:30:25,034 [qtp522764626-249] INFO  c.l.m.r.entity.AspectResource:171 - INGEST PROPOSAL proposal: {aspectName=globalTags, entityUrn=urn:li:dataFlow:(airflow,DMF_every_1hour_30min_load,prod), entityType=dataflow, aspect={contentType=application/json, value=ByteString(length=12,bytes=7b227461...205b5d7d)}, changeType=UPSERT}
2023-08-14 15:30:25,040 [pool-11-thread-9] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 12ms
2023-08-14 15:30:25,041 [pool-11-thread-2] INFO  c.l.m.filter.RestliLoggingFilter:55 - POST /aspects?action=ingestProposal - ingestProposal - 200 - 7ms
2023-08-14 15:30:25,044 [qtp522764626-256] INFO  c.l.m.r.entity.AspectResource:171 - INGEST PROPOSAL proposal: {aspectName=ownership, entityUrn=urn:li:dataJob:(urn:li:dataFlow:(airflow,DMF_every_1hour_30min_load,prod),sleep_in_pool_2), entityType=datajob, aspect={contentType=application/json, value=ByteString(length=167,bytes=7b226f77...77227d7d)}, changeType=UPSERT}
AntonChernyaevArammeem commented 1 year ago

Hh it's really bug, if i use into recipe pipeline_name: DWH_USAGE this not works beacuse already this ingestion was into datahub (but i delete this ingestion from ui and this ingestion was created from UI) But if i use thi pipeline_name: CLI_DWH_USAGE all works and i can see this into UI. it's really looks like a bug.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

hsheth2 commented 1 year ago

Looks like the underlying error message is ERROR: could not serialize access due to concurrent update. The bug that caused that has been fixed in 0.11.0.

I'm going to close this issue for now, but let me know (1) what issues you ran into with upgrading and (2) once you upgrade to 0.11.0, if you see this issue again.