beezlabs-org / l3-support

L3 support tracker
0 stars 0 forks source link

CRAWLER SITES ISSUE #68

Open ARUNASALAVADIVU opened 1 year ago

ARUNASALAVADIVU commented 1 year ago

Creating issue to find a RCA for crawler

Past 2 months 1st and 2nd days scheduled workflows went to received status. We Retriggered all sites one by one and all workflow has been monitored through VM bcz we don't know which execution will be stuck and which one is making problem. If one execution went to received status then automatically other executions will start as per schedule but workflow move on running and bot will be in received state. This month July also same problem happened. on 1st and 2nd july report is 48 workflow in running and bots are in received state,10 workflows in failed status. No more successful.

Sakthi1502 commented 1 year ago

Hi @ARUNASALAVADIVU, please find the RCA for the issue mentioned.

Issue: Workflow executions in running state for long time. Root Cause: The issue is identified to be from Sodiesliquor which is an enhancement. It doesn’t get failed since there is some issue with code. It is in running status for long time. Once it is failed manually, the next automation from queue is not getting triggered for the same. Mitigation: Need to stop the agent and retrigger other executions in queue. Action to be performed: 1) Stop Sodiesliquor execution for now since it is an enhancement and it is not expected to get successful until we plan for enhancement. 2) Have raised the issue of automations not getting triggered from queue in “tulip-issues” git repo which is to be worked upon by Beezlabs Product team. cc: @mbharathm @swapnamn @Kalidass2011 @anandamurugan36