can we start a appium instance for each device ?

[kotini]

When we reserver device using API . provider should start a appium instance for that device . This way we can use remote testing easily . For each device once call reserve then start appium instance . For each device once call release then stop appium instance .

[koral--]

Appium integration is out of the scope of this project. It provides you ability to adb connect to devices and it is up to you what to do with that. Do you have any idea what exactly can be added to this repo?

[kotini]

@koral-- I see your connect.js code where it start var server = adb.createTcpUsbBridge I am planning to start docker instance of appium . This solution will work only in stf docker . Idea is . if stf provider is running in a host , when we do call remote Connection . It will start a docker instance of appium . docker stf provider docker appium

If you are ok i will raise a pull request for a prototype .

[denis99999]

Hi @kotini ,

I feel @koral-- is right, STF is made to expose devices “through the network” and to control their “network” access from the final users (i.e. humans, tools, …), so it means STF should host mainly device control & access logics, and interacts with development and testing tools (i.e. application logics) through the network only, that is mainly using the STF RESTful API and the adb connect @IP:port command which are designed for that.

Unfortunately, a common practice that sounds like your proposal, is to install STF and testing tools together on a same machine in which devices are USB connected to fit specific application requirements.

In fact, in your proposal, some issues are met:

• You seems to forget that device testing is only one use case of STF while it can be used for other purposes like development, demo, etc. in which the launching of an appium instance each time a device is took control is not expected..
• Appium is just one testing tool, what about the others ?
• Additional significant processing of tools on the STF machine may degrade the native STF services with potential loss of reliability of the whole, in particular if tools access directly to the devices using adb -s serial command, then bypassing the controlling function of STF, which may lead to concurrent controls of a same device by different users or tools..
• Deploying STF on other hardware architectures (e.g. ARM) requires to have tools available on such hardware architectures and may also implies a loss a performance due to the limited system resources in such machines (i.e raspberry PI)
• ...

While separating clearly these device and tool logics by the network does not suffer from these drawbacks, and in case of performance issue due to network latency between the tools and STF, the solution is then to co-locate the machines inside an appropriate network (i.e. local, wired, etc.), but above all to not mix the whole inside a same machine.

So, in substance as said in my opening remarks being in tune I think with @koral-- , the core STF code which is device access oriented is not the right place to host your specific needs which are application testing oriented, you should gain to implement them as a software library or plugin integrated into a dedicated environment (i.e. CI/CD) communicating properly with STF using its RESTful API and the adb connect @PI:port command, as it is already made successfully with Jenkins, Travis, Gitlab CI, etc.

Of course, it is my personal opinion and perhaps I did not well understand your proposal or the real STF purpose..

[kotini]

@denis99999 and @koral-- i agree about u and your correct goal . My idea is default let make remote connection as adb . if some one set ENV VARIABLE REMOTE_CONNECTION_WITH_DOCKER true . user should pass docker compose file as mount option . and its user choice either they pass appium docker , xyz docker .abc docker . Then when REMOTE_CONNECTION_WITH_DOCKER is true . When Remote Connection start -instead adb remote start , start docker given by user . When Remote Connection stop -instead adb remote stop , stop docker given by user .

docker image name we can set to device serial . connect.js options.connectPort will expose to the docker instance . new container will run in openstf network . " --net container:openstfserver"

[denis99999]

Hi @kotini , Sorry but all is said in our previous comments, I think your proposal is simply out of scope of STF project and is a very dangerous design, and furthermore personally I don't see the point of that..

[jupe]

I’ve been using appium server in client side for each device succesfully. I created simple tool for test automation which reserve device based on test ”DUT requirements”, create adb bridge, spawn appium and eventually call actual tests using local appium server. That approach works very well and I’m happy with it - no needs any stf-appium integration.

But, things seems to be little different in iOS case which support is coming (#64). I had to spawn appium server in same host where devices are physically connected- which would be the “stf provider host”. I’m not 100% sure if this is only way to get iOS worked with appium, but little annoying that architecture is bit different in iOS and android. Does anybody have experiences with iOS with appium?

[nanoscopic]

@jupe I actually agree that appium should be an option to start automatically at the appropriate time by the provider. In the case of IOS support; I will be adding this feature to the coordinator that starts the stf-ios-provider in the near future.

I agree that it is a bit odd for the main STF project itself to start Appium, as there is nothing else in OpenSTF done that way. OpenSTF ( the main Android project ) is all NodeJS code and manages the bare minimum of external processes needed. Despite it being odd, I'll add this to the list of things to figure out in the future; as my team is internally using both Android and IOS devices with OpenSTF and Appium. Currently you can simply start up everything, then start up Appium also and leave it running.

I don't though see the purpose of only starting Appium when you reserve a device. Why not just leave Appium running always when a device is connected?? ( which is my plan and what we are already doing in a more manual fashion )

The integration I have planned for the stf_ios_support coordinator is for the coordinator process to automatically start Appium ( when configured in config.json ) after everything else is started up.

[jupe]

Why not just leave Appium running always when a device is connected?

It gives some benefits:

• we can separated maintain/update appium server and STF
  • client can decide which version of appium is running and it does not affect to any other clients. Basically client can use docker containers for everything - client uses just pure ADB connection via STF..
  • if appium crashes it does not affect to other STF clients
• client can collect all appium logs -> debugging is easier when system logs are stored along with test runs

I haven't play so heavily (yet) with appium so maybe appium is robust enough and above benefits are negligible and make no sense.. Opinions? In the past, I have learned that the hw test system should be divided into parts that cause minimal impact of even a single part of the crash. (there is always bugs somewhere in the system in long run unfortunately)..

The integration I have planned for the stf_ios_support coordinator is for the coordinator process to automatically start Appium ( when configured in config.json ) after everything else is started up.

Optional feature sounds good approach - let user decide how to deploy STF.

[denis99999]

But, things seems to be little different in iOS case which support is coming (#64). I had to spawn appium server in same host where devices are physically connected- which would be the “stf provider host”. I’m not 100% sure if this is only way to get iOS worked with appium, but little annoying that architecture is bit different in iOS and android. Does anybody have experiences with iOS with appium?

For iOS case, one way to keep the testing client (e.g. Appium, etc.) outside the iOS provider while giving the client the feeling that the device it took control is physically connected to its running machine, is to use USB/IP technology which is part of the linux main line.

So, as the Android provider proxies in a secure way the ADB connection between the client and the device, the iOS provider should do the same working with USB/IP connection, with the great benefits to not mix STF logic and application logic inside a same running machine.

Obviously, it requires some additional developments:

• device control & access through USB/IP at provider side (i.e. simple linux machine)
• make available a USB/IP client at MacOS side running the application logic, here are two main tracks:
  • make a specific development on MacOS
  • use the native USB/IP client of VirtualBox (i.e. running a MacOS virtual machine inside a MacOS machine)
  • PS: some commercial solutions also exist as VirualHere for instance to make a PoC :-)

Moreover, by locating the iOS provider on a standard linux machine (i.e. using libimobiledevice library to interface with iOS device) instead of locating it on a MacOS machine, we gain on scalability at very lower cost!

[kotini]

my suggestion is 1>default let remote connection give a TcpUsbBridge

2>user can set environment remote connection using appium . It should start a appium instance . [This is tricky because appium has a dependance of adb command ]

3> user can set environment remote connection using appium docker . when user call remote connection one appium docker instance we can run . [ This will work only openstf + docker case]

[nanoscopic]

@denis99999 The IOS provider ( specifically Xcode running WDA ) has to run on MacOS. Using USB/IP doesn't prevent this from being a requirement. I have already investigated MacOS within VirtualBox; the performance is abysmal and I wouldn't recommend trying to make everything work that route.

libimobiledevice doesn't take the place of XCUItest + WDA. It only enables a small subset of what XCUITest is doing to remotely control IOS devices.

[jupe]

For iOS case, one way to keep the testing client (e.g. Appium, etc.) outside the iOS provider while giving the client the feeling that the device it took control is physically connected to its running machine, is to use USB/IP technology which is part of the linux main line.

after all, the fact is (?) that stf provider and appium has to be running at the same host regardless do we use with or without usb/ip, right?

Any estimate how many ios devices one basic mac mini (eg) can manage properly assuming “worst” scenario where each devices are in use at the same time? I little doubt that usb/ip does not give any benefits unless its somehow “cheaper” from cpu/mem point of view than physical connected devices.. stf providers can still deploy to multiple mac hosts and share devices to same pool..

[nanoscopic]

@jupe Two things "need" to run on MacOS: WDA through Xcode/XCUITest, and whatever streams video.

Note that the phone doesn't have to be physically connected to the MacOS machine in order to run WDA; WDA can be run connecting to the phone over wifi/network. ( essentially remote network debugging through Xcode )

Video streaming; the way it is done with stf_ios_support, uses AVFoundation / Quicktime streaming via USB ( lockdown protocol ); it requires the phone to be connected to the MacOS machine via USB. There is a project to stream video directly through lockdown protocol that supports Linux. In that sense, video streaming could be done on a Linux machine once that project is integrated. ( work yet to be done )

So... in the coming future a MacOS machine will only be required to run WDA; and the IOS devices themselves can stream video via Linux machines ( possible raspberry pis )

Video streaming via the stf_ios_support method ( which uses ffmpeg ), can/does use half of the CPU on a mac mini. ( depending on model ) As a result, it is unlikely more than two IOS devices streaming video simultaneously will work right now. ( nevermind that multiple video streams don't work currently at once through AVFoundation )

Once video is streamed directly using lockdown protocol instead of through AVFoundation, and thus able to be done through a cheaper machine than a Mac, then 15+ IOS Devices can be run off a single Mac. ( needing some number of small/cheap Linux devices, or cloud, to do the video streaming )

[denis99999]

@nanoscopic , @jupe, I pursue that discussion at #1142 because it concerns IOS support in STF.

[Jitu1888]

I have done the Appium integration with stf and its running awesome.

[jupe]

@Jitu1888 I created one python lib for tests that allow to allocate and start adb for appium usage; https://pypi.org/project/stf-appium-client/

perhaps could be useful :)