Best practices, general improvements and announcement

I wrote a list of the new headless chrome tools. Also updated this list to the awesome test automation list and recommended that list in TagUI readme for developers looking for test automation tools. I believe there is value looking through the various tools, their features, and the issues users raised. Probably will be able to pick up and implement some best practices which are relevant to TagUI's goal.

Now is also a good time to make general improvements to TagUI, checking various areas and functionality for improvements (such as frame, popup, loops, blocks, live-mode, consistency across browsers etc). Jotting down some thoughts below, some points will end up in readme next time.

TagUI is basically a hack that is meant for lazy people who wants to get things done asap without much effort (like me). It takes in super-basic natural-language-like scripts and spits out working nested CasperJS JavaScript code to drive automation. And along with that, broad range of features in TagUI framework which are from my experience important and super-useful to prototype / deploy / maintain web automation.

It probably won't attract contributors partly because it is so specific and already developed in its goal that it will only be relevant for a small number of users (but for those who do, will find it amazingly useful), also partly the coding style is basically to maximize screen real estate when developing using vim while on a nomadic lifestyle. Also, where possible I try not to modularize. Keeping everything in one place is easiest to maintain without having to grab info from multiple sources when it is time to debug.

I also try to reduce dependency as much as possible so that it can be easily deployable by non-developers (unzip and run without the need to install any other tools/environment). Same for Chrome integration, it is done from the ground up directly to talk to Chrome using Chrome Debugging Protocol through a websocket connection, without dependency on other Chrome frameworks/tools. In order to make all these happen, you'll see it implemented in JavaScript, PHP, bash-shell, Windows batch scripting, and Python (for Sikuli visual automation).

The part that I enjoy when making TagUI an open-source project, is it helps me distribute to a larger group of people, who can bounce ideas on how it can be better, or point out something which is in my blindspot. Since 3-4 weeks ago, I have been trying to divert traffic away from this Github page (all the recommendations to other Python tools for datascraping, test automation list for testing etc). I think TagUI has a meaningful user-base now that I'm comfortable supporting. I hope that won't change soon, I would rather spend more quality time working with early users than spreading time thinly to address issues from large number of users.

As some of you already know, almost a month back I said I'm moving away from working full-time on TagUI project. I still have a few ideas to get them implement (and can't wait for that to happen because I want to use those features myself), but it'll be done part-time instead of full-time since I made the first commit in Dec'16. What I found was the initial skeleton took just a few days to write (parsing natural-language to CasperJS JavaScript code). But as features are added on, and each feature gets improved, it takes increasingly larger amounts of time to make marginal improvements.

About a month ago, with TagUI v2.0 release having built-in visible / headless Chrome integration, I think the bulk of the work is done. I would now really want to make a CLI helper which can help to run scripts. The TagUI scripts are already in natural-language-like syntax to convert to JS code. I want to have a helper that can convert to calling the right scripts with right parameters. Instead of typing tagui bank_balance on command prompt, I think it is more intuitive to support typing erina get my bank balance or something like that directly on command prompt. And then the magic happens behind the scenes to return the results. Having this is also a pre-cursor to voice-control possibility later on. Another part I really want to integrate into TagUI is some form of machine learning. Maybe the Yandex's CatBoost project? (low overhead and can be run well on single laptop). Otherwise decision-making is restricted to if statements and so on. Not very sophisticated cognitive ability. Some form of matrix or fuzzy logic ability is simply needed to make TagUI be able to do more in automation.

Studied the API and issues of the various headless Chrome tools (namely Chromy, Chrominator, Navalia, Lambdium, GhostJS, AutoGCD, Chromeless). Potential ideas broadly fall into 3 categories, actions to be taken on a webpage, deployment style, and making an automated browser behave like a real user. Listed below my thoughts and which are actionable ones for TagUI's goal.

ACTIONS - a couple of useful ideas to dig further and implement

~~pdf output~~ - there's support for pdf generation for PhantomJS and Chrome, but this sounds like a configuration nightmare, not something I'll want to make as part of main steps for now. The snap step captures image to have some form of visual record of the automation. When running in default mode (PhantomJS), the step snap page to abc.pdf will create a pdf. During runtime, casper.page.paperSize = {width: '11.7in', height: '8.3in', orientation: 'portrait', border: '0.4in'}; can be used to set the document layout. For chrome, it is a lot of trouble to even capture image outside of browser view (dev team also has to use workaround to do something like that). Probably can make it official into step snap in future, so that capturing as pdf works for different browser mode. UPDATE 19th Aug - tried implementing but Page.printToPDF seems to work only in headless mode but not visible mode. Better to just implement than having no ability to capture for both visible and headless Chrome. Skip page options and putting to readme for now, as the behavior varies btw Chrome headless (pdf ok) and visible (pdf errors out with PrintToPDF is not implemented). Also don't implement options for now as PhantomJS default can generate a one-page PDF versus forcing it to split into multi-paged pdf to be consistent with headless Chrome. For Chrome pdf options see Page.printToPDF domain.method at DevTools Protocol Viewer page.
~~scroll method~~ - haven't come across a need myself to scroll somewhere. if needed using casper.scrollTo() or casper.scrollToBottom() can do it. Or if using chrome, use dom step to call window.scrollTo(x, y) or element.scrollIntoView(). Unless something is critical, would not want to add as apart of steps.
~~visible check~~ - this is tricky to implement as there are many combinations to decide whether a web element is rendered as visible. drop for now. refer to below comment on waiting for element.
[x] wait for element - as there is already auto-waiting of element to appear (to check whether the element exists), it can generally be handled that way and auto-error out if something hasn't appear after time-out. if needed casper.waitForSelector() can be used or for Chrome, dom step can be used to check in DOM. what is useful is making it easy to adjust the timeout. At the moment casper.options.waitTimeout = 10000; can be set to adjust timeout. A timeout step can be useful to make it easier and more readable. UPDATE - added timeout step to make it easy to set at runtime.
[x] upload file - CasperJS inherits from PhantomJS this.page.uploadFile('input[type="file"]',fileName) but it doesn't support XPath. For Chrome, DOM.setFileInputFiles can be used. This is a possible new step to add to let it work for Chrome besides PhantomJS. UPDATE - added upload step. some notes and details here on the implementation wrt PhantomJS and Chrome.

DEPLOYMENT - a parallel execution controller deserves some thoughts

[x] parallel execution - parallel runs are good to speed up something, provided that process is not tied with a single user account that is meant for single-login at any one time. TagUI already has web-service for Linux/macOS to allow triggering via URL call. To deploy parallel runs, each machine needs to be a server node and a separate controller script can send the execution requirements to the servers accordingly. Not sure if adding a Windows webservice handler is useful. I think mac is a better machine but the sentiments seem to be changing. to chew on whether a parallel run controller is useful to add into TagUI (distribute flows, execute, monitor status, collect results). UPDATE - created https://github.com/tebelorg/TagUI/issues/43 to track idea, worth exploring automation farm and not only silos execution.
~~aws and docker~~ - cloud, serverless and containers. I'm not sure if these deployment channels are useful when users can run the automation locally on their laptops in the background using their own spare infrastructure and computing power. also, for regular large testing AWS/Lambda can be costly. My preference is a decentralised execution model, something like Mastodon with localised servers, instead of having a centralized execution. This is not mainstream but just can't see why to use cloud when there are ample computing infra to be utilized. can explore again when run into some use case which can't be handled using decentralised model.
~~webdriver api~~ - it is interesting that Google decides to open up headless Chrome, at a time when Selenium webdriver API becomes part of W3C official browser control protocol. I would have thought W3C's move would further solidify webdriver or WebdriverIO leadership position in browser automation, specifically test automation. Ideally, it will be great for TagUI to even remove dependency on CasperJS or PhantomJS, as PhantomJS's future seems doomed as of now. With a stable CasperJS v1.1.4 and PhantomJS v2.1.1, TagUI can still automate chrome / headless chrome / firefox. In the long term, the underlying API might have to shift to webdriver or something else. But that also brings in Node.js or other dependencies. Headache... It's still early, and the dust has not yet been settled for the Node.js based tools to commit to one or WebdriverIO. If possible, it's probably better to reduce dependency on CasperJS/PhantomJS and have some form of adapter/config based approach. So that whatever tools are appropriate, TagUI still retain a large part of the flexibility of making prototyping/deployment/maintenance easy.

REAL USER - generally not applicable or ends up in destructive cycle

~~useragent~~ - this setting controls what the target webserver will see as the supposed type of browser and OS. it is normally used to mask identity as an automated browser. not inclined to make this part easier, would recommend not going into a non-constructive race to access websites which don't want to serve headless / automated browsers. Also, the setting can be configured in tagui_config.txt or using Network.setUserAgentOverride for chrome.
~~custom header~~ - same rationale as above. also, the setting can configured in tagui_config.txt or using casper.page.customHeaders. for chrome, call chrome_step() to invoke Network.setExtraHTTPHeaders method. Syntax is easy chrome_step(method,params);
~~cookie management~~ - PhantomJS supports cookie management. Previous attempts at Firefox through SlimerJS wasn't good, still need manual clicks to select profile. For Chrome, if the automation is meant to be used on a user's browser to automate some process, this shouldn't be an issue. do not see value in exploring this now, for the purpose of automating processes.
~~cache management~~ - similar thoughts as above.
~~script injection~~ - CasperJS/PhantomJS/Chrome allows script injection. But for the purpose of automating some manual actions this isn't relevant. A normal user won't be injecting JS file into browser to do something that is needed in a manual process. Also the dom step allows running code in the browser DOM. That should work for running simple scripts / functions.

Above comment lists some ideas from other new automation tools. This comment tracks some general areas to check for improvement. As TagUI itself is a hack of converting natural-language syntax into nested working CasperJS code, the areas which most hacks are done are those below, which deals with a lot of nested blocks and handling the context appropriately. These are the areas being revisited often to try improving and make the language translation work for more and more edge cases.

[x] blocks / loops - improve code block and loop handling. to process { and } into casper blocks if the lines start with { or }. otherwise will cause many valid javascript code to throw error. such as when using chrome_step or setting api_config. with this change, the js step is not needed to avoid those errors. add more error checking for loops to show error at parsing if { is written on the same line. this will guide users to type { on the next line to be auto-parsed into casperjs blocks.
[x] frame / popup - issue found with having frame/popup step within loops. fixed bug that can mess up complicated flows with multiple loop calls. frame and popup previously don’t work in for loop. now it can be used in for loop. however, no code blocks for frame / popup. just call multiple times in the for loop. can become very messy and harder for user to debug issues.
[x] live-mode - updated details for steps that are not relevant/supported in live mode, add support for live mode navigate to url when running chrome browser. for phantomjs, set window.location.href or using casper.thenOpen will not work when event loop is not yet passed back to main 'thread'. also checked that recent changes to steps have been mirrored to live mode.
[x] cross-browser - by design chrome does not allow downloads headlessly for security reasons. using download step will give an error message with the chromium issue number 696481 for this feature. issue looks to be moving along and may have basic download ability with v62. followed-up with SlimerJS issue on reading input, so that live mode can work when using Firefox.

For various ideas from other tools, a timeout step is added, an upload step is added, a new issue to track controller for parallel execution is created. Other ideas not relevant. See above for more details. Outstanding for this issue is to review the areas in above comments for possible improvements.

All points from the review addressed, and general improvements committed to master. Closing issue.

Ken; Thanks for your great work on TagUI. The readme file is very good for TagUI. And the examples are also helpful to see the actual use of the natural language scripts. I use CasperJS heavily and am familiar with automation testing using CasperJS. I have been trying TagUI's Chrome extension for recording automation flow steps with some successes and some failures. What I notice on a number of sites (Twitter, Facebook, Zillow) that the X-path and CSS selectors for an element I need to interact with are dynamic (probably set based on a session variable or something else that changes with each test session) and cannot be relied on to be repeated. So an automation flow created once with TagUI recording (Chrome extension) fails when the next automation test is started (because the element ID cannot be found). For example "click yui_3_18_1_1_1502240389295_1616" is an example of a dynamic element reference generated by TagUI recording that changes each time the automation flow is run. I get a similar result when I manually inspect the element rather than use the recording tool.

Does TagUI have the ability (or is their a technique) to interact with dynamic element references? Is it possible for CasperJS to successfully use only part of an element reference ( such as ignore the dynamic part of an element ID )? How can I get TagUI work with dynamic element references?

Thanks @TMM21042 for your feedback and asking this interesting question! Let me re-post as a new issue, I think other users can benefit from the scenario you described. :smile:

@TMM21042 actually posting here is better as this thread is about best practices. I reply here instead.

Yes what you described is a scenario that any automation guy will encounter once they worked with enough scenarios. Many frameworks and websites uses dynamically formed ids which are really bad for doing automation. Yes there is a way to address it.

The first way is checking the css selector to see if there is a commonly occurring format which points to that same element regardless of the randomly generated id. Although the id is different every time, the location of the element relative to the rest of the webpage elements and its parent is normally the same.

The second way is using XPath, usually I use XPath as my default way of identifying elements as it is more expressive. You can even use or, and, contains and many other ways to form expressive selection of elements. In your above example, I assume that yui element is a child of some parent, so the XPath will be something like that //*[@id="parent_id"]//*[contains(@id,"yui")]. You can even wrap XPath with (XPath)[n] to access the nth occurrence of a match.

What I described above is applicable to both TagUI and CasperJS. CSS selector may be easier to use and a closer representation of DOM, but XPath really is the power-tool for identifying web elements.

@kensoh thanks for the response. Your advice on identifying dynamic elements was very helpful. Since this discussion is about best practices, I would like to hear your advice on your top 5 resource recommendations for learning more about headless automation and testing. I feel as though I have much to learn, and using the PhantomJS and CasperJS documentation online is not always easy (nor is trial and error). Are there other resources (books, tutorials, videos) that would be helpful to an intermediate level automation guy to improve his/her skill?

Also, I saw your post about Chromeless, its popularity, and V60 of Chrome, but I do not fully understand the implications of what it can do, how this will work with TagUI (if at all), or if it is an alternative or replacement of some type? Is there a comparison of the features or strengths and weaknesses of each?

Thanks.

1/2 - Hi @TMM21042 actually the test automation industry is very very fragmented. You have all the few big market leaders, but there are so many smaller players but still important in serving specific use cases. Because of this, I personally find it really hard to have a one-stop resource covering various tooling.

For example, I can't even find an adequate online guide on using XPath. The tutorial at w3school is great piece of work but it covers so many details that are not that relevant to test automation. Someone learning XPath for test automation might be more confused after reading it page by page.

Joe Colantonio website is cool and already covers broad range of topics. But it is still limited to mainstream commercial or open-source tools. Another website that I like is TEST Magazine, but it is more for general keeping up to date than hands-on guides on new tools.

I personally like this Test Automation List, it compiles tools for various languages and platforms. I don't think it can capture everything that exists in this deeply fragmented market, but I found it to be the best reference I can find. It hosted as a project on GitHub, so people can submit pull requests to update it. I can't think of anything else, relevant resources mostly exists in a fragmented one-off way through platforms such as medium.com or individual bloggers that can be found through googling a topic.

2/2 - The upside for Chromeless is obvious (serverless parallel runs + headless Chrome), so I comment on the downside. For folks familiar with Nightmare / PhantomJS / CasperJS, Chromeless is launching by over-promising too many things. For example there is this line on readme Do pretty much everything you've used PhantomJS, NightmareJS or Selenium for before.

But looking closing, their API is basic compared to those mature tools which tool years to have great coverage of methods and edge cases. Also, it highlights on the pros of serverless testing using AWS Lambda, but I believe that is costly for any regular huge volume test automation use cases.

In any case, most users won't know that and probably don't really care about these details. Since it is the maintainers intention to attract users and traffic, I wrote that article to send them a bit more traffic. There are over 100 issues raised in less than a month with 100 still open. The risk that Chromeless faces is it is actually easier for existing mature tools to be updated to support headless Chrome + AWS Lambda, than for it to catch up in maturity. For eg, NightmareJS already has a pull request that overhauls the mature tool to headless Chrome. The advantage is with the spotlight and traffic, contributors and maintainers can shape it into a great tool if enough efforts are dedicated to it.

TagUI can run in headless Chrome or visible Chrome, but do directly using websocket. I guess they are really for different purpose. You can think of TagUI as a more or less developed framework covering many areas related to automation, but with many design choices already made on behalf of user in order to do so. There are still ways to configure by editing config or source code directly, but for a large part I think it is a tool which either you love it or you hate it. Because it is so specific. Things like talking to Chrome directly without using chrome-remote-interface just to avoid Node.js dependency and to have unzip-and-run function, I don't think most developers will agree with this choice. But that is really important for me, because it can be easily deployed on any OS by curling / unzipping an URL. Without the need to have any dev environment or tooling set up.

Chromeless is more like a tool which developers themselves can shape it and add on various features according to what they need for their specific projects. So it is something that users can build on to extend or shape into something totally different compared to the base functionality. I will strongly recommend that any user or developer to check out tools such as Chromeless, Chromy and those in that article over TagUI. I also recommend this way at the developers section of readme.

Neat! Puppeteer just launched - https://github.com/GoogleChrome/puppeteer. Strongly recommend developers to check out this project. I think this project is gonna be a blessing for the community.

Hi Ken, TagUI is very useful to me to do automation but i can run it in back end in windows,How to Run In any browser ,like for example to open a website and login ..Thanks

Copying @Aussiroth to follow-up as AI Singapore and @Aussiroth is now maintaining TagUI. Thanks for your feedback. I assume you mean running in background mode. To do so, use the headless option or run without the chrome option. type tagui without any parameters to see the list of run-time options.

TagUI can only work with Chrome and PhantomJS browsers. It can also work with Firefox but pending fix on SlimerJS project that TagUI uses to interact with Firefox - https://github.com/laurentj/slimerjs/issues/694 (the newer versions of Firefox has breaking changes that affect SlimerJS)

On Mon, 12 Nov 2018 06:43:48 -0800, Ken Soh wrote:

Copying @Aussiroth [1] to follow-up as AI Singapore and @Aussiroth [2] is now maintaining TagUI. Thanks for your feedback. I assume you mean running in background mode. To do so, use the headless option or run without the chrome option. type tagui without any parameters to see the list of run-time options.

TagUI can only work with Chrome and PhantomJS browsers. It can also work with Firefox but pending fix on SlimerJS project that TagUI uses to interact with Firefox - laurentj/slimerjs#694 [3] (the newer versions of Firefox has breaking changes that affect SlimerJS)

-- You are receiving this because you commented. Reply to this email directly, view it on GitHub [4], or mute the thread [5].

Hi Kel,

Thank you for yours reply ,i will check and try it ,i am going work on TagUI so in future if i struck anywhere i need yours help to move.

Thanks & Regards,

Kiran.

Links:

[1] https://github.com/Aussiroth [2] https://github.com/Aussiroth [3] https://github.com/laurentj/slimerjs/issues/694 [4] https://github.com/kelaberetiv/TagUI/issues/42#issuecomment-437906451 [5] https://github.com/notifications/unsubscribe-auth/AdoD9xEF6sESCfGJuPA8HBBb0a11MuCqks5uuYkkgaJpZM4OuddH

aisingapore / TagUI

Best practices, general improvements and announcement #42

Links: