iOS support

[koral--]

Continuation of https://github.com/openstf/stf/issues/1142

[viktorgogulenko]

@koral-- what's a plan regarding iOS support in DeviceFarmer? We are really waiting when at last it will be a part of STF (DeviceFarmer)!

[koral--]

@nanoscopic could you comment on that?

[naureen-34]

Any update on this?

[vdelendik]

@koral-- we already have iOS support for OpenSTF implementation as part of MCloud solution (oriented mostly for automation engineers but with ability of remote access) repositories: https://github.com/zebrunner/mcloud https://github.com/zebrunner/stf

let me know if you interested in cooperation

[nanoscopic]

The initial plan for IOS support is to merge in ( by way of a proper PR ) the TMobile STF IOS support from these repos:

• https://github.com/nanoscopic/stf ( this is a fork of DeviceFarmer/stf and a PR will be made after some additional cleanup )
• https://github.com/tmobile/stf_ios_support ( this repo will be moved into the DeviceFarmer namespace )
• https://github.com/nanoscopic/stf-ios-provider ( also to move )
• https://github.com/nanoscopic/WebDriverAgent ( also to move )

Disclaimer: I am the primary author of all of these repos.

There are a variety of other repos/projects cloned down and built by stf_ios_support. Those should all also move into the DeviceFarmer namespace.

Looking through the zebrunner STF "fork" ( why is it not a proper fork?? ), it does appear to have a full parallel implementation of IOS support within its "device IOS unit", that is entirely different but similar than the STF fork planning to be integrated currently. The nanoscopic/stf and nanoscopic/stf-ios-provider repos/projects are both based loosely off the mrx fork of STF. ( https://github.com/mrx1203/stf )

I think a variety of things have been done better/cleaner in the zebrunner fork, and ultimately the best versions across all forks should be integrated into the main project. I would like to see full cooperation and acceptance of improvements to STF regardless of where they originate.

Despite being a part of the core project now, I do not wish for any of the code that I have written to take priority over or stand in the way of other contributions or potentially better implementations. The plan remains to integrate the code I have written into STF, but I don't see that preventing integration of any/all of the quality bits of code within the zebrunner fork. Despite the differences, it should be possible to replace most of the device-ios unit with the zebrunner contents. I'll need to do a more thorough review of it to determine how to proceed in that direction.

From a quick initial review, I would like to begin pulling in pieces of zebrunner into the code I already wrote, to replace and update parts that aren't as good.

Ultimately merging the two codebases will be a longer process but is one I am open to. The rest of the core team has expressed that we want to be fully open and accepting of contributions, and I agree with such an open and collaborative view.

The challenge will be how to best cooperate to advance the project for all of us, and create a single maintainable core that includes IOS support.

--

Other architectural / overall comments and questions I have:

• I'm not clear exactly on how WDA is launched via mcloud. stf_ios_support/coordinator handles launching all the various processes needed to stand up a "IOS provider". I am curious as to how this is done with mcloud/zebrunner and if it is publicly available as open source. ( Perhaps "Zebrunner Starter" ? )
• Video in zebrunner/stf is done via the MJPEG source within WDA. It is planned to add support for that video path to stf_ios_support as well as an option, but video currently in stf_ios_support is done via: https://github.com/nanoscopic/ios_avf_pull This is done because the AVFoundation video source has a much higher/smoother framerate. It is also designed to skip duplicate frames.
• On brief review I see long tap and doubletap/click support in zebrunner/stf device ios unit. I would like to port/copy/merge that in very soon as there is no support for that within nanoscopic/stf device ios unit currently.
• I notice that tapping/swiping is done in zebrunner/stf in the same way as nanoscopic/stf used to do so. There is a better/faster method now. This commit: https://github.com/nanoscopic/WebDriverAgent/commit/4aa3818956c5229883f9e77b65eaaebcb0789295 Specifically tapat and dragat are much faster than the WDA actions used by zebrunner/stf currently.
• I see this line of text in the zebrunner/stf "fork": "Current project must be deployed only as part of Zebrunner Starter or MCloud solutions!" The project, and this fork, are MIT license. Hopefully this claim is not attempting to have some control over the usage of the MIT code beyond the norm?? As an active open source developer I certainly don't care about this particular statement. I'm assuming it is not intended this way, and merely there to head off users from trying to utilize the code and having no idea how to start it up due to lack of additional instructions.
• It looks like the "inspector" has been integrated, along with ability to "click" nodes within the inspect tree. That is nice and I'd like to see that in the main project also.
• I don't see "IOS provider" support at all in the zebrunner/stf fork. Can someone tell me where this is?
• This page is blank: https://zebrunner.github.io/documentation/components_overview/ I'd like to see an overall explanation of the components of zebrunner and how it is started up. Can it be done fully open source?

--

In closing, the current roadmap of getting IOS support into DeviceFarmer is:

• Clean up the nanoscopic/stf fork a bit more, and create a PR against devicefarmer/stf
• Move many of the above mentioned repos into the devicefarmer namespace
• Update STF code to run on Node 12+
• Begin integrating bits of zebrunner/stf wherever it is reasonable
• Longer term: Move towards one cohesive STF with IOS support that contains all the features and best parts available, so as to all collaborate on a single project.

[nanoscopic]

Preliminary support for IOS in DeviceFarmer is available via:

• Server support via this docker-compose config: https://github.com/tmobile/stf_ios_support/tree/master/server
• IOS provider pointed at an instance of that: https://github.com/tmobile/stf_ios_support

Essential 10,000 ft overview instructions:

• Setup an STF server instance using the docker-compose config
• Clone, build, and run coordinator from stf_ios_support on a MacOS machine to provide an IOS device to it

More detailed instructions are available within the stf_ios_support repo itself. As part of the integration in DeviceFarmer/stf instructions and information about this will be updated on the main project.

[nanoscopic]

Update on this: For the short term development of stf_ios_support has moved to become ios_remote_provider, providing devices to ControlFloor instead of DeviceFarmer/stf.

I will, though, likely, in the near future, be writing a "bridge" so that ios_remote_provider can be used with DeviceFarmer/stf. Will update again when that occurs.

[ababushkin-bscmsc]

@koral-- @nanoscopic Do you have any information on iOS integration? I saw it here https://github.com/EasilyTest/stf, but I can see the device, but when I start using it, the screen is blank

[nanoscopic]

@ababushkin-bscmsc This is the first I've seen of the EasilyTest/stf stuff.

Looking at it briefly, I notice the following things:

1. It uses a modified fork of WebDriverAgent ( without being an actual fork... ) The modifications are for faster clicking and swiping by way of avoiding sessions. It is still not as direct as the modification I made, but better than the default still.
2. It uses the MJPEG server within WDA for video.
3. It is using the mrx repo more directly, as opposed to my more heavily modified fork of the mrx repo. This is alright, but it doesn't have the changes I made to make it work with Node 12, so I am guessing from briefly looking that it still requires an older version of node.
4. It retains all of the madness/confusion of how the custom device unit within mrx repo works as a result of 3. As a result I am definitely not interested in supporting this fork.

One is, of course, free to document how it works and recommend it as a great solution for iOS support in STF.

There is no "please only do it this way". Currently the semi-official "stf_ios_support" repo is out of date and doesn't work with the current STF build due to changes that have not been integrated into stf_ios_support. I am not currently supporting/working on that repo, and the number of issues there is growing and not being addressed by anyone.

Essentially the current state of iOS support with STF is "wild west". If anyone wants to step up and document the different approaches / options and their current state of functionality that would be appreciated by all STF users.

[KostyaSha]

How differs tmobile/device-ios and zebrunner/device-ios?

[nanoscopic]

@KostyaSha tmobile/device-ios? What is that? Are you referring to this? https://github.com/DeviceFarmer/stf-ios-provider/tree/master/lib/units/device-ios

And this? https://github.com/zebrunner/stf/tree/develop/lib/units/ios-device

Looking at it it is unclear what it is based on when it was initially committed in 2018. You can look at the zebrunner commit history. It certainly has lots of changes. I never really used or looked at the zebrunner code. My / the-tmobile-version was created by heavily modifying the mrx ios-device code.

The T-Mobile code is very out of date at this point. I'd say that using the zebrunner version would be better currently.

I still of course don't believe much in supporting DeviceFarmer/STF at all, and think it should be replaced entirely with ControlFloor, but that cannot be done until ControlFloor fully supports Android devices. The reason is because there are underlying architectural problems with the design of STF and it is tremendously painful to build a full working set of components. There are also too many modified versions of STF floating around that have not contributed ( or even tracked against ) upstream.

The issue remains that my company ( Dry Ark LLC ) is very small, and I still do not have any clients supporting the effort to port over Android support from DeviceFarmer into ControlFloor. I want to do that work, but nobody is asking for it.

The other idea I had was writing some integration from ControlFloor into STF itself, so that ControlFloor can pretend to be a normal STF Provider and send video ( frames ) to STF, and also accept commands ( clicks, keyboard, etc ) from STF. That is certainly doable, but the ZeroMQ integration is messy. I briefly discussed this with a few potential clients, but none of them seemed to want to pay what it would take to get that done. At this point that is further muddied by the fact that the open source ControlFloor code for iOS is itself now outdated.

This leads to discussion of the overall thing holding back quality iOS support: Free open source code that works well with iOS. I took my own projects related to this closed source as I just wasn't getting adequate funding through open source.

As I discuss in my YouTube overview of creating an iOS device farm, there are a few things that matter:

1. How to get video
2. How to control the device
3. How to fetch additional device info

Video ultimately should be done using an Upload Broadcast Extension, but there is no freely available implementation of this that works smoothly / doesn't have tons of bugs. The last open source version I released is the closest you'll get to a free solution here, and nobody seems to want to touch it.

Control of the device to work best ( assuming you don't want extra hardware to emulate bluetooth devices ) should be done with XCTest. All of the people I know of working on this have based their work on and/or are directly using WebDriverAgent. I rewrote WebDriverAgent completely to work better and be simpler, but it is closed source. The last open source version I released was a heavily modified WDA. After that version I started over from scratch to avoid being tainted by anything within WDA. The closed source version Dry Ark LLC has made is a full rewrite from the ground up.

In order to run XCTests ( WDA ), something that speaks lockdown protocol is needed. The best current solution here is go-ios, which is why I was the primary sponsor for that project for a while and continue to be friends with Daniel Paulus. I also wrote my complete thing here, iosif, but I don't know that anyone is using it, and Dry Ark LLC does't use iosif in ControlFloor. We use go-ios. I use iosif for testing / diagnostics / research. Zebrunner uses go-ios now also.

I personally don't see anyone revamping WDA as we ( Dry Ark LLC ) have done, so there are going to remain deficiencies there until, probably, I released my rewrite for free.

I also don't see anyone bothering to use Upload Broadcast Extension properly in a free way. The paid device farms are doing this, but none of them ( besides myself ) released their version. Initially LambdaTest used my version, but they decided to write their own Upload Broadcast Extension using h264. They did not release it. It is relatively easy to duplicate what they did, but I couldn't rightly explain what that is as it is proprietary information. I also don't think using h264 is a good solution due to increased latency and problems with low/fluctuation bandwidth.

If you want a concrete path for adding iOS support into upstream:

1. Port out the ios-device unit from zebrunner and get it working with the current DeviceFarmer HEAD.
2. Further modify my last released Upload Broadcast Extension to move it to straight TCP instead of the buggy ZMQ, and fix the memory leaks.
3. Integrate the updated video extension into ios-device.
4. Monkey around with WDA to get it to be able to start an upload broadcast extension. You'd think this would be easy but it is extremely difficult.

I won't be explaining in detail about these things ( although even what I am saying is somewhat detailed ), as it is the entire selling point of my company product, ControlFloor. I still though believe open source should continue and am happy to point anyone listening in the right direction. I don't believe anyone is listening, but I'll still keep pointing.

I also still want to release everything myself / my company have done as free open source. I'm just not going to do that until I am sure I can pay my bills with all the bits free.

[KostyaSha]

I built and run your "vidstream" and "vistream.ext" ;) (yeap uploaded manually to device that is fine) Also patched stf_ios_support to find correct start broadcast elements, but later switched to avfoundation. I spent hours before found that running video tools from ssh can't grab screen without any errors. As soon as i run it from macos UI it asked perms for iterm to camera capture and everything run. So i was able to patch all your tools and run ios_stf_support on stf 3.4 version (right now figthing with scaling clicks that are bad also for android 11). imho the problem is that you own all upstream repos and can't process changes because of payed work conflict. I tried to rebase your CFA with the latest WDA, but they are moved/renamed half of files. Without ability to use fresh XCode using CFA doesn't make sense ( i can try again of course).

I also still want to release everything myself / my company have done as free open source. Unfortunately i can't sponsor you, but i can try to get my payed time to refresh staff. Moving forward step by step we can then get ability to swap pieces of puzzle.

So if you look it from FOSS eyes, the stf_ios_support is the only latest solution that works more or less obvious: the number of tools that do their things + messy communication of everything to everything. zebrunner is their's company product at any point they repeat OpenSTF situation, that' why i would like concentrate on devicefarm namespace, but again it's using repos pointing your personal namespace.

PS do you know who is making this https://www.youtube.com/watch?v=DA2AvuxEOxU ?

[nanoscopic]

Will comment on video first:

• Landing page: Looks sort of like CF
• page/device.html: What? Is this really a static page using JS only? That's a little weird.
• "iPhone is getting ready": For how many eternities?
• Device video: Looks very smooth and high resolution. This would be bad over a slow connection. I'm guessing full resolution h264?
• Clicking latency: The audio is just this fake crap, so we have no way to tell when the actual mouse was clicked versus when the click occurs.
• Video also: No red bar. Time is 9:41. It is using avfoundation or quicktime_video_hack. Possibly a modification of the TMobile code overall.
• No device logs: Meh. Nothing to see here imo whatever it is.
• Mouse control / WDA: I don't see WDA installed on the device but it may be disguised or just not flipping to the screen of icons that has it. I'd guess that this is using bluetooth mouse control, but it looks to be using absolute coordinates. The demo doesn't show drawing anything freehand so it can't be told. This demo is pretty bad.
• Summary: Meh.

Back to discussion:

The issue with the video extension, as you are noticing, is that it is very finicky to get it to work seamlessly starting it up and staying up. In the version of the system you are using this is particularly finicky overall. In the current closed version we've fixed those issues so it properly starts up every single time, and also stays up indefinitely without going down. Additionally one can start it up manually if you desire and it will properly detect that and show the video. There is also fallback to CFA based video so you can still see the screen even when the video extension is not running.

The permissions to do stuff on MacOS is annoying and problematic. There is no documented way to automate setting those permissions. I spent a while and figured out how to hackishly inject the permissions with adequate permission levels... ( root ), but even then it doesn't really work running as a system service which is one of the reasons I gave up on that method.

The scaling of clicks is one of the parts of STF that I hate. It sends over click coordinates 0-1, so frontend scales them to that, then they are scaled back to screen resolution in the backend. The code in both front and backend that does that is extremely messy and insane. It was 100x easier in CF to just forget all that crap and send exact pixel coordinates from the start. One still needs to know what the screen pixel size is to scale where you click to that before making the call, but doing that is much cleaner and less confusing than what STF/DF does.

I contributed the key things that mattered, I though, all over to the DeviceFarmer organization space. If there is anything missing that is needed there that needs to be altered to maintain the open source code, please feel free to have anyone in the DeviceFarmer organization clone my repo and point the other ones to that new repo. I shouldn't have to be involved for the code to live on. I think the only thing missing is the actual STF code, as I sent it in as a PR to upstream. Nobody was willing to accept my gigantic PR and they quickly diverged away from it.

I looked into updating my code to work with latest upstream changes, but too many things were altered in annoying ways and I threw in the towel. After spending SO much work updating STF to Node 12 and having my massive PR get ignored by the upstream community I decided I don't actually care about DeviceFarmer besides referencing bits of it that may be useful as anyone discovers anything. As far as I'm concerned the TMobile STF/DF code I wrote, stf_ios_support, and everything associated with it, is dead in the water. I'd highly advise anyone trying to use it not to bother. It's just a tremendous waste of time.

In regard to CFA, they didn't move the files, I did. I changed the crud out of it and vastly altered how it works. The open source CFA you see is really just a modified WDA, but it works very differently. They aren't compatible. You can't swap one with the other without using it in a totally different way.

As for compiling CFA for latest XCode, as mentioned in the ticket about this there is a relatively trivial way to do that but I'm not sharing it. I didn't even come up with it. The method is mentioned in the ticket on WDA having that problem also. Someone suggested an easy solution that works there. Go dig for it and you will find it. I also didn't find it there myself; my coworker Brian found it. It isn't just that I'm not sharing because I'm guarding the silly secret. It's because I don't remember how to find the information myself and I didn't make the change to CFA implementing it. It also doesn't matter. You can copy over device support files to older XCode and they work fine and you can still build it the old way...

I'd disagree strongly on stf_ios_support being sensible. It worked alright but it is vastly inferior compared to the last version of ControlFloor I released. The last version is still buggy, but it would still be a better starting point open source wise than stf_ios_support.

Once again, you have my support and blessing to fork any of the repos in my namespace that you need into the DeviceFarmer namespace and continue their development without me. I have no objections to that. I do not wish to gatekeep my code in any way besides having set the license and insisting the license be followed. I agree the license is bizarre, but I feel strongly about it and it must stay.

In regard to what is going on with me, and why there is no easy way to use the latest ControlFloor yet, it is because I am still finalizing legal issues with a troublesome client. Negotiations are essentially complete and all IP is being signed properly back over to me, but the actual agreement hasn't been signed yet. I believe I'll get it in docusign and it will be agreed on by everyone within the next few days. Once that is complete I can begin to give and sell CF again. This has taken far too long and has been preventing me from offering the product without excessive legal risk.

Once everything is signed, we'll be doing the following:

1. Adding per device restrictions into CF. Right now the system is not limited for how many devices can be used as it was sold directly to large clients and not to individuals or small companies.

2. Offering a free ( binary only ) version for users only using it with up to 3 devices. I would appreciate a donation of some sort from such users, but I won't require it, as I want people to be able to use it freely for simple reasons. The people I mainly want to pay are larger companies saving money from using our product instead of alternatives.

3. Selling licenses to the overall product very cheaply. My speculative idea so far is $100 per device per year. This is vastly cheaper than any polished alternative, and additionally the latest CF works better, in my opinion, than any of the alternatives I've tried ( and I've tried all the major ones... )

Assuming that these 3 things go as planned, I think that most people with a need for an iOS farm ( non hosted ) will either use the new free CF, or pay for the licenses. If that goes as expected, that will be adequate for me to both continue maintaining the product, and to slowly release parts of CF as open source as new useful bits are created to sufficiently convince users to continue licensing the overall system.

At any point if I believe that I can actual pay my bills indefinitely by just offering support ( eg: have adequate contracts to be confident in it ), then I'll be putting things back into open source. The obvious problem here is not the fault of the community, but my own. That fault is that I haven't adequately advertised or demoed any of what I've created. If I had provided better demos and videos on the product this whole time, more people would have been willing to purchase support. This is a learning experience for me in that I am a long time software engineer, not a sales person.

In summary, look forward to being able to try CF yourself and license it cheaply and easily. Once it is easy to try it out everything I'm saying about it being a lot better than stf_ios_support will make a lot more sense. Without that proof a lot of what I say will just seem like hopeful fiction.

[KostyaSha]

• Summary: Meh.

Well, the fact is that people trying to make one more tool (we remember xkcd standards.

After spending SO much work updating STF to Node 12 and having my massive PR get ignored by the upstream community I decided I don't actually care about DeviceFarmer

That's pretty bad i also see primitive issues and PRs that are not merged. Right now both stf_ios_support and stf runs fine with latest node versions.

The scaling of clicks is one of the parts of STF that I hate. It sends over click coordinates 0-1,

yeap, stf_ios_support didn't update rotation after landscape app is run. If i correctly understand zebrunner do post-call and added wire proto to sand back to browser an actual orientation https://github.com/zebrunner/stf/blob/c1a87cd65308e75a6ff47a4000d7a619bafa6639/lib/units/ios-device/plugins/wda/WdaClient.js#L520-L522

I wrote, stf_ios_support, and everything associated with it, is dead in the water.

You got experience with it, not worse to have it working bad to demonstrate why some other variant is better. The other point is that there is no better android far and digging into stf_ios_support helps understand how to deal with android things. (Note i personally hate all this js and it's "architecture" would be happy to join rewrite of everything in go). I would also like to update docs with known issue and they way i run it (i see that most of people didn't even get into what i get).

If i correctly understand according to licenses i can pick parts of zebrunner device-ios to stf_ios_support (though it seems they patched STF server side).

Once everything is signed, we'll be doing the following:

I fully understand and you can choose any monetisation model like share or not your knowledge. And i think that fine if FOSS implementation will be worst than your because there is nobody more experienced to make better. But for those who preffer foss and would like to work with code there is enough information right now to get something working. Right? For stf_ios_support:

• some issues with locks or startup flow. That should be not so difficult to fix it in go code.
• fix ios rotation(orientation), add more routings for plugins (buttons, keys, clipboard)
• avfoundation - right now it just works, probably switch to direct qvh.
• probably switch to go-ios

As of CF i would try it of course (i know that i can downgrade xcode, use whitelisted names, etc), but right after "just run" there is step with support it. That means that i would need to support STF and CF where both not documenting functions/code...

[nanoscopic]

I don't think that implementation on YouTube is anything new. It is just a tweaked version of the existing stuff floating around from what it looks. It isn't really a new standard of any sort in that fashion. I can't say for sure without a better demo or information about the source it is using, but it doesn't accomplish anything new from what I'm seeing there, nor does it even work particularly well.

STF has been updated to run on the latest Node now? That is news to me. I'm super curious who did that as I did the same and I don't think my changes were ever used. I wonder if someone did manage to merge my changes for that eventually. Good to know. I'll at least stop repeating that STF only works on an ancient version of Node. Are you sure about that??

Some apps don't support a different rotation, so the rotation of the device can change on the fly. The TMobile code never handled that properly. I don't think the last open source CF supported that either. Latest closed CF handles that properly.

That there is no better open source Android device farm is rather amazing to me. I agree it is true it's still somewhat dumbfounding. I don't actually have any objection to the core server portion of CF being open source, as that is not where most of the research and effort went. Hence I believe CF is the future rather than maintaining the aging STF code eternally. A lot of the STF bits can be reused and connected to the Go CF code without too much difficult imo.

And yes, as I was saying, the difficulty in combining bits of people's forks of STF is that they have to change pieces throughout the stack to get things working, so the forks are incompatible in messy ways that takes a lot of work just to understand what was done and why. It's made worse by how absurdly difficult it is to setup a working STF stack. That is part of what plagues stf_ios_support. I did my best to automate the build process pulling all the needed repos etc, but the result of even that was very messy. I learned from that and we vastly reduced dependencies and multiple repos in latest CF.

There are certainly enough ways to make a less featured version that functions as open source. In so much as that can be done and people will support it for their own reasons I agree it should happen and continue to advance at the pace people are willing to put in the effort at.

You are correct that a lot of the event based aspects of how a device farm needs to work is a lot easier to handle in Go. Even then it can be messy. My initial attempts were to make everything able to handle any state of the system and determine what needs to be done to get bits that are down back up and running. My coworker didn't like that and was confused by it and rewrote it instead to just do things in a specific order and if that fails exit and run the whole sequence again. His method is actually more stable, but it is also a bunch slower in specific conditions.

You are correct also that stf_ios_support startup flow is buggy. Sometimes ( all too often then is desirable ) the process bugs out. You can kill it and run it again and it will work, but it's bad that sometimes it just doesn't. Ultimately this was fixed by completely rewriting how that process works. I wrote it by just hacking more and more stuff on. After I left TMobile I rewrote that whole process from the ground up to make CF, partly in an effort to clean up that process as it was just a huge mess. This is one of the many reasons I'd recommend the CF process and hooking that up to STF over trying to make stf_ios_support work, as even the open source CF process is far better than stf_ios_support.

Rotation is easy enough to fix, but in STF it is a pain to route the events around due to the funky ZMQ process.

We never really implemented clipboard in iOS as Apple does their damnedest to try and prevent it. What was implemented in CF was paste in the form of the system typing out content that is pasted. It's useful for pasting in passwords for example.

Keyboard automation we've been through many attempts at. There are a number of ways:

1. Type out keys using WDA, which in turn uses the typeKeys XCTest call. This is total shit and never fucking works properly or well. Avoid this at all costs as it is a complete waste of time.

2. Use WDA to enter keys by calling the physical device entry... this works but only for lowercase letters. I never figured out how to make the call work for things that require shift. You can wrap it in the proper shift wrapping call, and it just doesn't do it. If someone can figure that out it would be great. It would require decompiling the supporting Apple code on the phone side to understand what this is doing.

3. Use a virtual bluetooth keyboard. This is well known how to do and there is lots of open source code floating around for this. Problematically most of the examples require a special hardware dongle. Additionally I think this could have issues with a lot of devices in close proximity. I still think this would be the best solution for open source.

4. Automate the on screen Apple keyboard by clicking on it. This is what we did in closed CF. It took a huge amount of effort to do this well. We made a small app and go through an initialization routine where we map out the entire keyboard including all keys available under each key. Once mapped out, we know where to click the keyboard to type a key without having to query screen elements. Querying screen elements is dirt slow and makes for a horrible experience. This solution actually works great but it required an unreasonable amount of development to build the solution. The amount of work is why I'm not giving the solution away. It was expensive to build. The idea of it is easy enough to understand, it just takes a whole bunch of fiddling to make it work well.

In regard to video, I'd recommend against qvh once again. The reason why is the activation of the alternate lightning/usb-mode for it is buggy. It doesn't always work. Additionally the underlying processes on the phone that support is sometimes freeze and have to be forcibly restarted. I've even seen cases where the whole phone has to be rebooted because restarting the phone processes isn't even sufficient. When it is working it is smooth.

The other problem with both qvh and avfoundation is that the phone processes themselves only hand out an h264 stream. This is potentially fine if you have the bandwidth to stream this to your destination, but it is bad if you need to resample it. Decoding h264 has a high cost. You really want to avoid this if you can. The other issue is that you can't really adjust the h264 stream. You get a relatively high resolution and no ability to adjust the data rate of it. Once again, if you have the bandwidth, and are willing to tolerate the horribly buggy nature of the on-phone Apple supporting services, it works great... when it works. Until it glitches out, then it's a pain to reset everything.

All that is why I gave up on avfoundation and qvh years ago. It just wasn't a solution I could give guarantees to any client on. Too many issues that just can't be solved because the Apple supporting code is just too buggy. If you don't want to have endless pain, use an Upload Broadcast Extension.

go-ios is just one piece of the puzzle. It should be used for fetching basic device info and for starting xctests, because starting tests with Xcode is just horrible.

I think you mean CFA in the last paragraph? That is the only bit that won't compile on latest Xcode. Agreeably you need that to run the last non-closed CF.

The beauty of CF is that it is extremely minimal in complexity compared to STF. I agree what I did can still be difficult to follow. That is why my coworker Brian rewrote a lot of what I did in latest CF to make more sense to him. The non-closed CF isn't terrible though imo ( I wrote it so I'm fond of it ). Despite that it does have bugs. I can't really say how to fix them, because the way they were fixed is by Brian spending 8 months totally revamping what I did. There is no bit I can just point at and go "fix this". I believe the number of problems is small and could be fixed without a complete overhaul such as we did though.

The elephant in the room regarding the last non-closed CF is the weird license I think. I think the license is why nobody is touching or trying to make that code work better. To me it shouldn't be an issue as it is essentially MIT minus specific companies, but open source folk are very picky about what licenses they will allow. Additionally many companies won't let their employees touch an open source project unless it uses one of a few licenses they deem acceptable. I still won't relicense it though even if nobody is willing to touch it due to my license. The reason why is if I released it under almost any of the various popular licenses one of the bigco is just going to pick up the project, fix the issues, and then sell it without contributing back. I could perhaps be convinced to relicense it as AGPL. If there are at least a handful of people willing to use and maintain open source CF if it is changed to AGPL we should all talk.

Even if it was AGPL it would not prevent either a bigco or one of the major device farm vendors from using CFA though as-is to improve their solutions, and I have a lot of issues with the companies on the license ban list. That said, I'm still okay with the last released version of CFA going open source, since it still didn't work as well as I'd like. It is an improvement over WDA, but it still had a bunch of mess, due to being based on WDA. The CFA that I am against releasing in a way that bigco can use it is the new full rewrite. Even that I may change my mind on over time though.

I'll even go so far as to say what WDA should be replaced with ( and essentially what the new CFA is ): It is just a thin wrapper around Xctest API. The thinner the wrapper and more directly the underlying calls are exposed the better. The problem is WDA tried to do way too much fancy stuff and just introduced a ton of bugs and slowness in the process. If you stick to just exposing the underlying Xctest calls as directly as possible so that they can be scripted, it works equally as well as the original calls, which is far better than the end result you get using Appium and WDA.

It isn't particularly difficult to expose Xctest calls so they can be called dynamically. The difficult part is understanding what calls there are exactly, how they can be used, and how to use them to do what you want/need. The reason is because the majority of the useful calls aren't documented by Apple. You can of course just stick to the documented API, but there are much more powerful things in Xctest that are very useful beyond the documented API.

[KostyaSha]

Are you sure about that??

right now run in nvm v17 stf and stf_ios_support (patched bit and send PR)

but the result of even that was very messy. I learned from that and we vastly reduced dependencies and multiple repos in latest CF.

not so bad, but better to use git submodules to have fixed stack of versions :) (like any package manager do lock of dependencies).

do things in a specific order and if that fails exit and run the whole sequence again

that what i wanter to fix in your coordinator as soon as i understand what fails. Or add bigger locking (i saw in commits you already had issues with concurrency of logic). Better it will be bit slower, but more stable.

All that is why I gave up on avfoundation and qvh years ago.

It much better than 2fps screenshots ;) From what i see in amazon farm videos it also faster. Yeap, bandwidth is an issue, but more or less it works, it should be possible even to add some button to kick/reset staff. And of course any time this could be swapped to use other solution. I was able to run vidstream, but didn't get stream to dst IP. That's why temporary step back with this variant.

the way they were fixed is by Brian spending 8 months totally revamping what I did

while on STF i already have working android and ios (routing rotation WIP to get landscape). That's why i want firstly to refresh stf_ios_support. I already spent one week to run all this stuff and would like to contribute back my findings (fixes/document corner case/debug flow).

. It is an improvement over WDA, but it still had a bunch of mess, due to being based on WDA.

i would say that it would be better to write a fresh app with clear communication and route. I think nobody will touch ancient "stable" project with broking backward compatibility to provide a few faster calls...

So, i'll try to fix rotation and back with a PR...

[nanoscopic]

I absolutely refuse to use git submodules ever. They are evil and horrendous.

The way I wanted to fix the ordering / concurrency issues was by using metaprogramming with a DSL configured state diagram so that it could be easily understood, updated, etc. Unfortunately that isn't what was done... but that would still be the best way imo.

qvh, when it is working, is better. As I said, the problem is that randomly the underlying Apple service(s) freeze requiring restart of them or reboot. Yes, one of the solutions we did was add a "reset" button, but it is somewhat crappy to have to tell your users "click the reset button if it isn't working".

Despite my thinking iOS support in STF is dead, I'm happy someone is updating it to work again with the current version, if for no other reason than people will stop asking me "how do I make this work?" By reading the code and fixing the bugs is, of course, the answer.

If you think the released CFA is just "a few faster calls" I see you spending months additional reading WDA code. WDA is pretty evil. Feel free to try and get Appium to accept a PR injecting those "few faster calls" into upstream. You'll find that it isn't that trivial.

A random example of WDA evil is that it has switch/if-then statements to make it backwards compatible with ancient Xcode versions that should never be used. The differences in Xctest API are more significant than can be dealt with properly with a few "if then" statements between versions, and it should never have been done that way.

[KostyaSha]

I absolutely refuse to use git submodules ever. They are evil and horrendous.

In theory yes, but as soon as repo organised in such way it's the best solution. Modules could still be updated, but they will be in a working state. git modules supported by IDE so instead of opening each repos folder in separate IDE i can see blame and read commit messages.

Feel free to try and get Appium to accept a PR injecting those "few faster calls" into upstream.

i saw thread, it started with features and end with licenses and politics... For PoC of ability to have ios as a web device i would like also to have a live swipe/drag like in android (Even if i need to press everything slowly...) Didn't understood what is the problem to add it, but it's a different thread.

[nanoscopic]

Git Submodules

This isn't really the place for this discussion, but since you started it, you will now receive the reasoning for why Git submodules are terrible.

The reason is because they don't allow you to reasonably / sanely use a different collection of repos / commit versions to build out a system.

Git submodules cause you to have each thing dependent on a specific version of some other thing.

What happens if you update a submodules? You have to remember to update all the submodules connections that you are aware of, and if you didn't even create the connection? Well in that case you'll just have to hope people using submodules have sense enough to pay attention to when the submodules are updated.

Lets not forget that when you git clone something, Git also idiotically doesn't bother to tell you that there are un-cloned submodules. You just have to magically know that.

What if you want all your git cloned things to live in a ~/git folder? Sorry. Git submodules don't work that way.

What if you have a circular dependency in your git submodules. Well... don't do that?...

What should be done, which is what I partially did in the project and the future versions and have been optimizing how it works, is that a JSON configuration file should specify what other repos are needed and what versions are acceptable.

Git submodules are horrible *** garbage and I will never **** touch them with a 10 ft pole.

IDE Magic

Don't even get me started on "well my IDE has magical features and I like adding random **** to the project that can only really be utilized using my favorite IDE".

Live Swipe

"Live swipe" is not possible without using an emulated USB or bluetooth mouse. In both cases it is relative only and you have to rely on visual feedback of the device mouse cursor to do things accurately. I agree that bluetooth mouse emulation is the common solution being used by commercial device farms, but it requires specific hardware as I've mentioned.

The only other way, which is what we have done in the latest CF, is to record the swiping path and then play it back once the mouse button is released.

[KostyaSha]

You have to remember to update all the submodules connections that you are aware of, and if you didn't even create the connection?

There is a recurse option. You may not describe how it works, because i have experience with using gitsubmodules in a huge team and cross-departments.

Lets not forget that when you git clone something, Git also idiotically doesn't bother to tell you that there are un-cloned submodules. You just have to magically know that.

Probably you tried long time ago, there are a lot of options to control them. I do pushes with changes directly from submodules. That's pretty useful when you need to touch multiple repos. Gitsubmodules works fine.

Git submodules are horrible *** garbage and I will never **** touch them with a 10 ft pole.

What if you have a circular dependency in your git submodules. Well... don't do that?...

Seriously, it's unrelated to real cases. what if a brick falls on your head? World is not ideal.

Don't even get me started on "well my IDE has magical features and I like adding random **** to the project that can only really be utilized using my favorite IDE".

Nobody enforce you to use IDE.

. I agree that bluetooth mouse emulation is the common solution being used by commercial device farms, but it requires specific hardware as I've mentioned.

Not a problem, i have experience with a HW development. BT to usb is very cheap and easy thing. Will try later if PoC will be acceptable in a slow usage variant. Thanks for direction.

(hope it's not rude comment for you) Everything looks like you are not happy that somebody trying to touch repo and code that you owned. Looks like everybody should work only like you see it. With a license you are limiting usage, with a comments you are trying to rule a development, that's a very weird FOSS. I very appreciate your technical suggestions (though i would like to see comments and small docs under methods and in code) and people may follow them and may not, right? Any steps that solve current not working state of the project i think are useful.

JSON configuration file should specify what other repos are needed and what versions are acceptable.

Well here i would like to say that json is shit for configs, yaml allows to have comments and looks pretty clean. While with json you forced to add additional tools that strip comments and glue things. Most parsers support both.

[nanoscopic]

Git Submodules

Yes, you can go and adapt your mentality to how they work. They remain problematic and there are better solutions. All you've done is say "you can change what you do to make do with them." Sure. That doesn't really explain why I would want to do that.

Push from submodules

You've failed to address my point that the submodule is cloned into a subdirectory instead of being able to share one submodule to two different repos depending on it. This is an issue. You've ignored my point about that.

Brick falling on head

There is a reason you shouldn't walk under construction projects where a brick could fall through and kill you. Same thing with walking under ladders. Sane people take precautions. There is a reason for OSHA and helmets etc. World is not ideal, hence you use your brain to do things that don't invite trouble. Using Git submodules is inviting trouble.

Being made to use IDE

If you use features that are annoying without an IDE supporting it in some fancy way, you are causing trouble for those who don't want to use any of the IDE that do that.

BT Virtual Mouse

I agree that BT mouse is easy enough and is a useful solution to have for live mousing. There are many issues with this though:

• Many clients/users won't want to get any additional hardware besides USB cables and provider machine regardless of how cheap. I've had this issue with clients. I agree they are being unreasonable, but people will still object to it as a requirement. As an option it is fine.
• How this is done requires different stuff depending on the hardware used to do it. That is, you cannot write the software once and have it work with all the different options here.
• You need to consider automatically pairing the bluetooth device. This is non-trivial with WDA.
• Wireless interference. In data centers with 100 iOS devices in a rack it could be a problem to want them all to be connecting to bluetooth dongles.

Touching my stuff

Don't use my stuff if you have any issues with it or me. I don't give a shit. I am though one of the world experts on this stuff. You can ignore me or think I'm nuts if you like. That isn't my problem. What you can't do is deny my expertise on knowledge. I don't owe it to anyone to provide open source in any way. This is why my projects aren't "open source", and are release under a custom license. It is really easy to say "It seems like you don't like a lot of things". Duh? It took you how long after reading my license that restricts 30+ companies and anyone related to them to realize this?

What is your point here exactly? I'm not embracing the FOSS way? I'm doing so more than most of the people in this very specific slice of specialty.

Code is lacking comments and docs

True. Don't use it if you don't like it. Nobody is making you. I do recall saying like 10 times not to use stf_ios_support as it is basically deprecated junk.

YAML

Maybe don't talk to me any more. Saying YAML is great... I can't even. Just stop while you are ahead.

[KostyaSha]

I am though one of the world experts on this stuff.

Can i clarify only one thing, did you ban Apple in your license?

@koral-- who are the owners of the DeviceFarm GH org? What is the position of stf_ios_support repository?

[nanoscopic]

@KostyaSha Please cease discussing things unrelated to iOS support in this issue.

Please also create issues about the different aspects rather than lumping everything here.

Finally, please do not raise political issues that are unimportant to product development or start arguments here. This is not the place for it.

I mentioned the license because it is a relevant different from the main STF project and it is specific to the iOS support.

I mentioned not using submodules because I've intentionally avoided them and I'm the author of the stf_ios_support repo entirely.

I mentioned JSON being used as it is core to how the stf_ios_support is setup to be configured and build.

Your attitude and questioning is not appreciated and is not useful to moving the project forward.

I am not a "leader" of the DeviceFarmer org, but I did come up with the name, and I'm the owner of DeviceFarmer.com I also own the Copyright for stf_ios_support and all its components, so I'm the only person who could modify the license. Therefore I do essentially control the related bits under my custom license regardless of what namespace they are in.

I do welcome anyone to fork the repos and modify them as you wish while staying within the license I've provided.

My participation here in the DeviceFarmer repo is purely as the original source of the repos before they were contributed to this namespace. I intend in no way to restrict or impede whatever other participants want to do with the repos.

I would recommend against starting a fight with me. It's just a waste of time that would be better spent improving the project.

[KostyaSha]

Please also create issues about the different aspects rather than lumping everything here.

The situation is pretty clear right now. There is a repo that author didn't support, community can continue modifying it, but author is generating long posts with his advices that end in 0 code. In fact agressive blocking any attempts to do it.

@nanoscopic even if you didn't want or like it, devicefarmer orgs can continue modifying repo in any way. If you are one of the org owners and do this, then DeviceFarmer is one more OpenSTF. Please think yourself how your presence here helps devicefarmer?

I would recommend against starting a fight with me.

Now i don't take your agressive comments into account. It's pretty clear right now why you got zero contributions in stf_ios_support and in CF. And why people like zebrunner do forks. You are not doing FOSS for a 3 years. You are happy to work with your closed source with decisions that you like. Good luck.

Anyone who would like to work on IOS support or already do it please reply (or PM).

[nanoscopic]

I have clearly stated what is going on with myself and my company the entire time. No one was misled.

Readable source has been released for everything I've done in regard to iOS farms up until a year ago. At that time I took everything closed source and have only been developing closed source since then.

I've also stated many times that I will be releasing things back into readable source once I am able to continue with the work full time while doing so. I cannot right now. It is necessary for me to keep the latest developments closed source in order to be able to continue working on it full time as I have been.

My presence here in the org and in purpose in responding to this ticket is to share what information I am still acceptable in doing to help guide the development so that it is done in a sensible way. I have no obligation to do this. I just want to be as helpful as I can do without damaging my own company and business.

Since I personally wrote a lot of the code being discussed, it also makes sense for me to comment to some degree on why things are as they are. Anyone in the org is free to diverge from my design and intentions, I am explaining what I am for and against. You don't have to agree if you don't want to, but arguing about it is pointless. If you don't like something, just go change it and submit a PR and someone else will possibly merge it. I don't merge things. I am not actively maintaining any of this stuff. I've been clear on that.

Your complaining about me being here isn't helpful.

I am not blocking anything. I in no way exercise whatever authority I may even have in this org. I will not close issues here. I will not block anyone and delete comments in the org. All I do is comment. As far as I'm concerned whatever rights I have beyond a normal user in this org space is purely honorary for having contributed a pile of code to the org.

I agree completely that the org is free to do with the code I have contributed however is desired within the licenses that the components are within.

I don't think it is legit to say I am an "org owner". I honestly haven't paid attention to what rights I've been granted within this org. If I'm an org admin, it is only honorary, and I would not stand in the way of any decisions made by the rest of the members. You are free to disagree with and make rude comments to me. It is kind of pointless and I think is distracting from the cause though.

I literally don't care what happens to stf_ios_support for over a year now. I share information only in order to attempt to be helpful where my expertise is relevant.

I told you it would be a good idea to use the zebrunner fork as it is maintained and sees active development.

I'm not interest in "open source" as in the official definition. I regularly create readable source under my custom license, as I have ever reasonable right to do so. There is no reason I must do anything the "OSS way". You are wasting time trying to trash me as I have made no claim to being some great open source person. I fail to see what that has to do with getting advice on how an iOS device farm should work though.

Please go piss into the wind elsewhere or stay on topic with technical things. The only relevance to any of this I am is as an information source. Whether I am friendly or mean, OSS embracing or not, or I like covering myself in jello and running down the street naked is none of your concern.

[KostyaSha]

Just for everybody who would like to contribute or deal with devicefarmer (?), @nanoscopic is making personal attacks against me (while i do things in my personal free time) and requested to block me here.

Just to clarify, i spent time on digging into source code, fixed things and would like to contribute them back. How all this end you can see here. Feel free to judge who is doing illegal actions. I don't want to comment or deal with such project anymore.

[nanoscopic]

@KostyaSha If you want to start a conversation about how you refuse to behave when asked to do so, please use a different ticket. This nonsense doesn't belong on this issue.

If you must persist in this, contain your drama to this new issue please: https://github.com/DeviceFarmer/stf/issues/691

[koral--]

@nanoscopic @KostyaSha I've removed the #691, marked your recent comments in this thread as off-topic and locked the conversation here. Don't use the public issue tracker for quarrels or teasing. Next time you will be blocked on this organisation.