search

apache / incubator-devlake

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
https://devlake.apache.org/
Apache License 2.0
2.57k stars 514 forks source link

[Bug][Gitlab] Error during fetching from GitLab #7194

Closed antoniomuso closed 1 month ago

antoniomuso commented 6 months ago

Search before asking

What happened

Fetching from Gitlab sometimes crashes with the following error:

time="2024-03-17 00:02:45" level=info msg=" [pipeline service] [pipeline #111] [task #634] [extractApiMergeRequestsNotes] get data from _raw_gitlab_api_merge_request_notes where params={\"ConnectionId\":1,\"ProjectId\":1328} and got 3112"
time="2024-03-17 00:02:51" level=info msg=" [pipeline service] [pipeline #111] [task #634] [extractApiMergeRequestsNotes] finished records: 1"
time="2024-03-17 00:02:58" level=info msg=" [pipeline service] [pipeline #111] [task #634] [extractApiMergeRequestsNotes] finished records: 500"
time="2024-03-17 00:03:00" level=info msg=" [pipeline service] [pipeline #111] [task #634] [extractApiMergeRequestsNotes] finished records: 1000"
time="2024-03-17 00:03:03" level=info msg=" [pipeline service] [pipeline #111] [task #634] [extractApiMergeRequestsNotes] finished records: 1500"
time="2024-03-17 00:03:03" level=info msg=" [pipeline service] [pipeline #111] [task #634] [extractApiMergeRequestsNotes] finished records: 2000"
time="2024-03-17 00:03:06" level=info msg=" [pipeline service] [pipeline #111] [task #634] [extractApiMergeRequestsNotes] finished records: 3000"
time="2024-03-17 00:03:09" level=error msg=" [pipeline service] [pipeline #111] [task #634] subtask extractApiMergeRequestsNotes ended unexpectedly\n\tWraps: (2) Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction (500)\n\tWraps: (3) Error 1213 (40001): Deadlock found when trying to get lock; try restarting transaction\n\tError types: (1) *hintdetail.withDetail (2) *hintdetail.withDetail (3) *mysql.MySQLError"

What do you expect to happen

It should not crash.

How to reproduce

I don't know why it happens. Currently, I have this error only for one project.

Anything else

No response

Version

v0.20.0-beta10

Are you willing to submit PR?

Code of Conduct

klesh commented 6 months ago

Interesting, were there any other pipelines running in parallel? Maybe related to #7099

antoniomuso commented 6 months ago

Interesting, were there any other pipelines running in parallel? Maybe related to #7099

Yes, I think so because this happens during the schedule. I currently have two projects scheduled at the same time.

klesh commented 6 months ago

@antoniomuso I see, the deadlock problem is being worked on #7099, meanwhile you may set the Environment Variable to PIPELINE_MAX_PARALLEL=1 as a workaround.

antoniomuso commented 4 months ago

@antoniomuso I see, the deadlock problem is being worked on #7099, meanwhile you may set the Environment Variable to PIPELINE_MAX_PARALLEL=1 as a workaround.

Thank you; your suggestion works.

klesh commented 3 months ago

Closing this in favor of #7099

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 1 month ago

This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future.