OpenBudget / BudgetKey

Opening the Israeli Budget!
https://next.obudget.org
49 stars 15 forks source link

define auto tests list #445

Open noamoss opened 4 years ago

noamoss commented 4 years ago

General mechanism proposal:

  1. Two kinds of checks: frequent or triggered.
  2. Two levels of indication: warning (on logs/reports), error (send an alert email).
test goal apply on trigger / frequency definition outcome Time estimation & follow up issue #
site status next.obuget.org, sociamap.org.il , sociamap.org.il/hakol-kore, next.obuget.org/govbuy, example page per content type every 4 hours error
scrapers all scrapers ongoing process when the first try fails --> warning, On the second fail in a row --> error warning, error
personal alerts: new items are added as expected example user record twice a week this test is intended to ensure we are not missing new items from the different ppageossible content types. My proposal: a. create a list of the currently relevant content types for the alert mechanism (notice it might change in the future)

b. create personal alerts for the tests user tracking "_test" on all relevant content types

c. create two mock items per content type. The title should include the convention ("_test1" and "_test2") it is a test item, not to be integrated into real users' alerts.

d. validate the new items are included in the personal alerts list.

e. run the two following tests.

f. delete all the mock items.
personal alerts: removed alerts are not included in the submitted list example user record twice a week following the previous check, remove "_test2" from the user's alerts list, and ensure the mock items are not showing in the personal alerts list error
emails delivery: ensure email submission example user record twice a week following the two previous tests: ensure mail is being submitted and working SMTP interface, submitted email should include "_test1" items from all relevant content types, and should not include "_test2" items from all the relevant content types error
noamoss commented 4 years ago

@akariv ready for review. For the tests you accept, please add your time estimation.