search

bcgov / foi-flow

Freedom of Information modernization
Apache License 2.0
5 stars 3 forks source link

Zipper Service Error handling Task #4510

Closed abin-aot closed 1 year ago

abin-aot commented 1 year ago

Describe the bug in current situation

Its being observed by Dev Team( @sumathi-thirumani-aot , @divyav-aot .... ) that Zipper Service on TEST Marshal has all of a sudden started to restart and gone crashloopback error on OCP. Upon analysis, its informed that it can be related to the exceptions thrown from the services inside.

Purpose of this task is to analysis the code and handle error correctly with DB status change. Also, to evict the corresponding REDIS message from the REDIS STREAM.

Link bug to the User Story

Impact of this bug Describe the impact, i.e. what the impact is, and number of users impacted.

Chance of Occurring (high/medium/low/very low)

Pre Conditions: which Env, any pre-requesites or assumptions to execute steps?

Steps to Reproduce Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Actual/ observed behaviour/ results

Expected behaviour A clear and concise description of what you expected to happen. Use the gherking language.

Screenshots/ Visual Reference/ Source If applicable, add screenshots to help explain your problem. You an use screengrab.

abin-aot commented 1 year ago

As per @divyav-aot , this specific request still giving issue, https://test-marshal-foirequest.apps.silver.devops.gov.bc.ca/foi/foirequests/558/ministryrequest/558

image
abin-aot commented 1 year ago

As per @divyav-aot , this specific request still giving issue, https://test-marshal-foirequest.apps.silver.devops.gov.bc.ca/foi/foirequests/558/ministryrequest/558

image
abin-aot commented 1 year ago

Need to check Scenario#4 of #3301 as well!

abin-aot commented 1 year ago

The ZipperService restart issue is related with the resources on POD while its processing files more than the assigned resources limit Private Zenhub Image

This was the previous settings, and ZipperService was having near to 500MB in memory. Now I made like this. Private Zenhub Image

so, @divyav-aot , @sumathi-thirumani-aot - so the restart of the PODs while doing Harms stiching is unrelated to the error handling code written, rather its with POD running out of resources - happy to demo that

abin-aot commented 1 year ago

Closing as per discussions on the stand up