search

killuhwhale / amace_validator

0 stars 0 forks source link

Compare AMACE window results to Global list, Compat team's list and Android team's lists. #1

Closed killuhwhale closed 1 year ago

killuhwhale commented 1 year ago

We want to be able to know when a status is updated.

Compat team has a list that indicates when an app is not O4C. They will continue to check this list. When we detect a change, we should notify the team that the app is now O4C. They can then update their list. We will continue to notify until their list is updated.

We will do the same for Android team list.

We will also track a global list of apps and their O4C status.

Also, if we detect a regression, we will notify the proper team.

killuhwhale commented 1 year ago

We need to design a process to check this.

We will want to report all errors at once.

So we will need to collect all the mismatches and report at the end.

  1. Pull down each list into the program
  2. Each app, check the list for update/ regression/ no change
  3. If update/ regression, add to some datastructure to hold until end of run.
  4. At the end of the run, report all

Considerations:

  1. We have multiple devices checking the same app each run, possibly. We expect there to be no difference between each run. Since we are checking ARC-P devices as well, we should check on each device and report locally first to gather resutls.

DUT reports to Host device first.... At the end of the run, it can upload the results or send the email. We can take advantage of the image server and add routes to support this. 1 route to send app results to 1 route to get the current session or to trigger sending the current session in an email Store app info in current session using redis?

Overview: DUT sends app status update/ regresion to a local server on the host machine (just like images) where the result will be stored in redis using the runID with package name (and any other info needed to report)

The runID and package names are unique keys where each device can just overwrite the results since we expect them to be identical, unless there is no result if the device is ARC-P with no AMACE info.

Then at the end of the run our python script can call the local server with the current runID to trigger an email to be sent with the data stored locally in redis.

We need to check changes of current app status => global list

What should the flow look like, how do we actually compare?

Strategy 1: Load all lists into Go Program, checking each app against the list. Check 3 lists: List A:

We should store for each runID, 3 separate sections:

TL;DR Fill this structure out using http requests from each DUT to HOST device. Skip running on ARC-P. At the end of run, send request to send 2 emails and update the global list. We can provide a view of this list on the dashboard.

List Storage: Android Team => Sheets copied from Table using appscript Compat Team => ??? Global List => FireStore?

Redis: { "runID": { "TeamA": { "pkgName1": { "field": value }, "pkgName2": { "field": value }, }, "TeamB": { "pkgName1": { "field": value }, "pkgName2": { "field": value }, }, "Global": { "pkgName1": { "field": value }, "pkgName2": { "field": value }, }, } }

killuhwhale commented 1 year ago

Currently, we are maintaining a global list.

killuhwhale commented 1 year ago

Currently on hold.

Most likely, Manager will use Google Tools to query 3 sheets and check for changes. We are not going to perform this check within this automation.

We will only maintain a dataset to be used externally.