Incorrect numbers printed in "Launched <x> tests, skipped <y> tests." ending line

[hagertnl]

The final line of output in a harness run looks like:

Launched 0 tests, skipped 1 tests.

The intent is to tell the user how many tests successfully made it to the queue and how many did not. The count is currently broken, and is actually counting the number of apps launched/skipped, not the number of tests.

The region of code that does this accounting is:

        with concurrent.futures.ThreadPoolExecutor(max_workers=self.__num_workers) as executor:
            for appname in self.__app_subtests.keys():
                future = executor.submit(apptest.do_application_tasks,
                                         self.__launch_id,
                                         self.__app_subtests[appname],
                                         self.__tasks,
                                         self.__stdout_stderr,
                                         self.__separate_build_stdio)
                future_to_appname[future] = appname

            # Log when all job tasks are initiated.
            for my_future in concurrent.futures.as_completed(future_to_appname):
                appname = future_to_appname[my_future]

                # Check if an exception has been raised
                my_future_exception = my_future.exception()
                if my_future_exception:
                    message = "Application {} future exception:\n{}".format(appname, my_future_exception)
                    self.__myLogger.doCriticalLogging(message)
                    self.__skipped_tests += 1
                else:
                    message = "Application {} future is completed.".format(appname)
                    self.__myLogger.doInfoLogging(message)
                    self.__launched_tests += 1

            message = "All applications completed futures. Yahoo!!"
            self.__myLogger.doInfoLogging(message)
            # For the moment, hard-code this as a print statement.
            print(f"Launched {self.__launched_tests} tests, skipped {self.__skipped_tests} tests.")

So somehow we need to get the test data from the asynchronous do_application_tasks call in order to have accurate printing.

[hagertnl]

@ddietz89 technicality we need to decide for this.

Currently, a test will occasionally raise an exception when it encounters most failures. Raising an exception will abort the remaining tests for that application and will not return any usable value for this count. So two questions we need to decide:

• Do we want to use exceptions to raise errors? I think that this isn't always user friendly, since it ends up dumping stack traces
• Do we want a build failure (for example) to immediately abort all other tests for that application? I think I'd rather it not, but maybe it's better to make this a command-line flag? --abort-on-failure?

[hagertnl]

After talking with Veronica, I am planning to implement the following and we can discuss once we have a prototype:

• There are some exceptions that should always result in harness termination. Leave those. For example, the harness somehow got to a bad location and can't tell where it is anymore.
• For regular errors (such as the test name is mis-spelled), use doErrorLogging to log the error appropriately and return a non-zero exit code to indicate an error, but do not raise an exception.

[hagertnl]

We can look at an --abort-on-failure flag later. That might be hard, if we enable asynchronous/parallel test builds. It would not clean up very nicely, possibly.