denis99999
commented
6 years ago Hi @JeannedArk , congratulation for your work, it is very interesting and well documented, but in the other hand my general feeling is that you should gain many benefits to externalise your testing tools instead to place it inside STF, let me explain about that hereafter.
Firstly, following are some personal comments on how your testing tool should work better “inside” STF (i.e. under this assumption):
1) adb -s serial vs adb connect @IP:port
I took a closer look on your code, so except if I made a mistake I suppose you made your tests on a single STF machine ?
Nevertheless, in the general case, that is in the context of STF deployed on multiple machines (i.e. full deployment model), in particular separating at least the “Provider” units from the rest, your system should not work unfortunately.
In fact, in your implementation, I saw that the access to a device is made by a docker container running on the machine on which run a Websocket unit, using a adb -s serial command. So, using such ADB command (i.e. with serial) suppose that the considered devices are USB connected (i.e. or looking as) to the same machine on which run also the corresponding Provider unit.
In fact, in case of a local deployment in which all units are co-located on the same machine, it works although from my point of view it is not correct to bypass the Provider unit to access devices, for consistency and reliability purposes.
But in case of a full deployment in which Provider units may be distributed anywhere in the network, it can not work because devices are no more USB connected to the machine on which run the Websocket unit.
So, one solution should be to communicate with the devices using the adb connect @IP:port command, so replacing the device serial parameter at container creation level by the Remote Debug URL of the controlled device (i.e. available through RESTful API).
Moreover, with this solution, if for some reason the device is kicked during testing stage and then after controlled (or not) by another user, the considered network ADB connection should be closed normally which prevent concurrent accesses to the same device by two distinct users.
2) ADB binary
In executor.js server file, you hardcoded the location of an ADB binary for use by the testing containers:
ADB_PATH_HOST = '/opt/Android/Sdk/platform-tools/adb'
Note this implementation information is unknown from the STF deployer which have moreover until now the choice of how he wants to install ADB.
Moreover, if a couple of Provider and ADB (i.e. adbd.service) units are co-located with the executor machine, then the two ADB binaries running on this machine may lead to unpredictable behaviours.
So, you should review the way on how you manage the access to ADB binary in order to conform to native STF constraints.
3) Websocket unit hosting testing tool runtime
At server side, you placed the testing tool runtime inside the Websocket unit for which in theory multiple instances may be deployed over one or more machines.
Firstly, I think you should create you own unit type, for example a “TestExecutor” unit type, in order to be able to place instances on dedicated machines eventually, so as not to risk degrading the native STF services (i.e. screencast, api, …) due to the additional and significant processing of your tool.
Perhaps also, you should review your storage policy in order to avoid to rebuild the same image on many TestExecutor machines ? For example, we can imagine another docker repository storing the built images shared by all the TestExecutor units.
Now, in substance as said in my opening remarks, I am not sure the core STF code is the right place to host this interesting ‘TestingTool’ feature, and may be you should externalise it on a dedicated cluster communicating properly with STF using its RESTful API and the adb connect @PI:port command, as it is already made successfully with similar tools like jenkins, gitlab CI, etc.
In fact, I feel STF is made to expose devices “through the network” and to control their “network” access from the final users (i.e. humans, tools, …), so it means STF should host mainly device control & access logics, and interacts with development and testing tools (i.e. application logics) through the network only, that is mainly using the STF RESTful API and the adb connect @IP:port command which are designed for that.
Unfortunately, a common practice that sounds like your proposal, unless I make a mistake, is to install STF in local mode and testing tools together on a same machine in which devices are USB connected.
The major issue met in such configuration is a loss of reliability mainly because tools access directly to the devices using adb -s serial command, then bypassing the controlling function of STF, which may lead to concurrent controls of a same device by different users or tools.
Mixing these two distinct functions has also other drawbacks:
-
no scalability due to the limited number of devices which can be connected to the STF machine
-
poorer maintainability of the whole which are essential in production environment
-
closing to a set of tools instead to make available the devices through the network for any ADB compliant tools inside a concurrency running model
-
additional significant processing of tools on the STF machine may also degrade the native STF services
-
deploying STF on an other hardware architecture (ex: ARM) requires to have tools available on such hardware architecture and may also implies a loss a performance due to the limited system resources in such machines (i.e raspberry PI)
-
…
While separating these two functions by the network does not suffer from these drawbacks, and in case of performance issue due to network latency between the tools and STF, the solution is then to co-locate the machines inside an appropriate network (i.e. local, wired, etc.).
In conclusion, I feel this idea #38 you propose to extend is mostly useless right now since STF has published its complete RESTful API enabling any tool as Jenkins or GitLab CI to remote control a device and use it as it was locally attached to the ‘executor’ machine.
So, I feel if you would externalise your tool for a reasonable effort, besides to preserve what I believe is the STF spirit, you will gain also some precious benefits:
-
control of the architecture, deployment model, path to binaries, etc.
-
better reliability and maintainability
-
managing of one or more STF platforms instead only one
-
…
Well, this is only my personal opinion, and perhaps I did not well understand your proposal or the real STF purpose. In any case I would be honored to have your feedback and that of @sorccu and @gunta also, and once again congratulation for this work.
Thanks.
JeannedArk
Hi,
like discussed with @sorccu, I opened a Pull Request: https://github.com/openstf/stf/pull/962
for an advanced automated testing feature, which extends the idea of: https://github.com/openstf/stf/issues/38
Idea: An integration which provides the following functionalities:
Implementation: You can see a demo here: https://github.com/uds-se/stf/wiki/%5Ben%5D-Farm-Testing also a (incomplete) wiki for how to use and setup a test. Please read through it to have an understanding of how the whole system works.
The following describes the properties of the implemented extension:
Known issues: I am really not an expert in any of these technologies, so don't be too harsh :) I am aware that I sometimes don't conform to the coding guidelines. Sometimes I am just not familiar enough with the architecture, so maybe you can help.
Feel free to comment :)