search

quipucords / yupana

Service app to process & upload data from quipucords and satellite to the host based inventory
GNU General Public License v3.0
4 stars 14 forks source link

Stop processors when fatal errors occur #252

Closed abaiken closed 5 years ago

abaiken commented 5 years ago

Closes #251

DB ERROR HANDLING TESTS:

  1. Bring up local dev
  2. Upload a report
  3. Stop the db docker stop yupana_db_1
  4. Once the db has stopped, upload another report (or you can wait for it to error on the next query for new reports/slices)- and watch the processors shut down
  5. Restart the db docker start yupana_db_1 and restart the server -and watch it process the first report immediately - as well as pull the second one off the queue docker start yupana_db_1

KAFKA ERROR HANDLING TESTS:

  1. Bring up local development
  2. Kill the kafka container docker stop docker_kafka_1 before the server starts and watch it shutoff processors
  3. Upload a large report (for example a 20k host report from sat) - wait until it is sending hosts to HBI - kill kafka ( This will take a while but it will eventually stop due to a kafka timeout error and trigger the processors to shut down)

The Kafka LOG errors are a bug with aiokafka --> see https://github.com/aio-libs/aiokafka/issues/496)

codecov-io commented 5 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@81ffd5c). Click here to learn what that means. The diff coverage is 69.59%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master     #252   +/-   ##
=========================================
  Coverage          ?   94.73%           
=========================================
  Files             ?       29           
  Lines             ?     2452           
  Branches          ?      286           
=========================================
  Hits              ?     2323           
  Misses            ?       84           
  Partials          ?       45
Impacted Files Coverage Δ
yupana/processor/garbage_collection.py 100% <100%> (ø)
yupana/processor/report_processor.py 94.24% <100%> (ø)
yupana/processor/report_slice_processor.py 92.12% <100%> (ø)
yupana/processor/report_consumer.py 68.42% <60.46%> (ø)
yupana/processor/abstract_processor.py 95.85% <71.42%> (ø)
yupana/processor/processor_utils.py 75.67% <75.67%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 81ffd5c...d7585bf. Read the comment docs.

myersCody commented 5 years ago

I retested the critical error using the methods used in the description and it worked for me! 👍