bcpeinhardt / schnauzerUI

A human readable scripting language for automated UI tests.
MIT License
9 stars 0 forks source link

Upcoming features needed #1

Closed bcpeinhardt closed 1 year ago

bcpeinhardt commented 1 year ago

Creating this issue as an easy way to take notes for myself while I'm still rapidly adding things, but if you're not me feel free to comment on this issue and suggest a feature.

  1. Testing needs to be a lot more ergonomic. Writing scripts as block strings and running them to test the output is a pain, as is running all the tests concurrently against a selenium server with weird settings. I want to explore a way to write sui scripts as their own files and easily load them into test processes.
  2. It's decision making time for the public interface. Hoping to make serious progress on a gui soon, and I don't want to have to keep refactoring the cli to handle the lib changes.
  3. Logging bug where after the script errors and quits it logs the remaining parsed commands. Should be a quick fix.
  4. Some more commands. To start with, under, drag-to, upload.
  5. Inline "command". I've decided not to have a ref command for referring to other scripts like TagUI. Rather, I'd like the repl to support inlining an existing script. We are optimizing for every script to be a straightforward, standalone description of an automated web process. This way scripts can easily be passed around in Jira, Teams, Slack, etc.
  6. A TUI for the repl. Stuff like autocomplete suggestions and pretty output will make schnauzer UI seem a lot more professional, and comfort non-engineers forced to use the terminal.
  7. An easy downoad and webdriver startup story. Running the program needs to be as easy as sui -i or opening the gui and pressing play.
  8. HTML test output. The logs are not nearly nice enough.

And this is just a start. I'm gonna try to get this done, then I think it's definitely GUI time.

bcpeinhardt commented 1 year ago

Okay fixed #3

bcpeinhardt commented 1 year ago

Okay we also probably want a create docs mode. You should be able to pass --create-docs and create a pdf or html with screenshots taken at each command. Comments could be automatically formatted as the body text of the document, and commands can form sections.

bcpeinhardt commented 1 year ago

Added a select command with some flexibility to it. Gonna work on the commands from 4. now.

bcpeinhardt commented 1 year ago

Added drag-to command

bcpeinhardt commented 1 year ago

Okay. Using webdriver_install and Command, sui now downloads, starts, and stops it's own webdrivers! Need to refactor so that clap parse right away, because it exiting threatens to leave drivers running. I've replaced the rust integration tests with just a folder of "test scripts". I'll need to implement rust tests eventually (at the library level), but for now this is more productive. I can do more research on testing cli applications.

bcpeinhardt commented 1 year ago

Okay. Using webdriver_install and Command, sui now downloads, starts, and stops it's own webdrivers! Need to refactor so that clap parse right away, because it exiting threatens to leave drivers running. I've replaced the rust integration tests with just a folder of "test scripts". I'll need to implement rust tests eventually (at the library level), but for now this is more productive. I can do more research on testing cli applications.

CLI parsing moved out front

bcpeinhardt commented 1 year ago

Just added "start from" functionality to the REPL in place of an "inline" command. I'm hesitant to add an inline command because it's counter to the principle of "write, run, throw away". Ooh, I should add that to the README.

bcpeinhardt commented 1 year ago

7 and 8 are done. SchnauzerUI handles chromedriver and geckodriver on it's own, so starting the repl is a simple sui -i. Rather than a log, the output is a json test report and an html test report.

bcpeinhardt commented 1 year ago

So the installation story is much improved. Users no longer have to add anything to their PATH except the sui binary. It's either a direct binary download or a single cargo install for rust users. For testing I have a main test script which produces a SchnauzerUI test report. It executes against a github pages site set up specifically for testing. If I add solid unit tests, it might be enough for the integration testing.

bcpeinhardt commented 1 year ago

Just had an idea for a feature. Add a --first-case argument to the CLI. Makes it quicker to debug scripts which pull from a CSV data table because you don't have to run the whole suite!

bcpeinhardt commented 1 year ago

There's a common validation flow we need to have a syntax for, which doesn't fit well with the single located element paradigm. Locate element and save as variable > Perform some other action that causes the element to change > Validate changes occurred on element. The solution may be to introduce saving elements as variables, requiring users to look at html to uniquely locate an element whose text changes throughout the test, or something more clever.

bcpeinhardt commented 1 year ago

4 is complete, under drag-to and upload are all implemented and documented.

bcpeinhardt commented 1 year ago

Re-add support for inputs directly following labels, even if they aren't correctly tied together.

dmgolembiowski commented 1 year ago

Hey @bcpeinhardt thanks for the email! I love this project and have worked on similar things in private side projects.

What are your thoughts on the following cases with respect to SchnauzerUI?

bcpeinhardt commented 1 year ago

Hey @dmgolembiowski ! SchnauzerUI is aimed empowering non-developers to do the 90% of Web Automation Testing, so each of my answers (which are thoughts off the top of my head that I'm 100% okay with being convinced otherwise about) will be colored by that goal.

  1. Branching, retries, and bug reports: The goal with branching and retries is to strike a balance where it's good enough for most test cases but not so complicated it looks/feels like programming. Right now there's an "if statement" that takes a command as a predicate and executes if the command doesn't produce an error. It's meant for things like popups. There's also a "catch-error" statement, but the control flow is a little clunky in my opinion, could definitely use a rework. 2 & 3. Capturing network connections and copying them as API calls & Caching static responses to disk to save time and server load: Both of these fall under the performance category I think, and would be awesome, but performance isn't my main goal atm. I think there will have to be some access to doing API work for setup and teardown to keep test times/ resource usage down, but I'd like to apply it in as user friendly a way as possible. It'd be really nice to integrate with an existing tool like Postman, so the barrier to entry stays low.
  2. Handling multiple tabs: A must implement, which I was planning on adding probably this week or next week. Don't really have any blockers on that, just need to settle on a syntax.
  3. Handling login states: Maybe! The approach right now for me is to have a login script for each system under test, and when I launch the repl I tell it to load from there. If we wanted to incorporate like a credential manager or secrets manager into the CLI I'd be up for it. But since the scripts are meant to be completely standalone we can't just add a config file. But maybe a global config file? Lots to think about there.
  4. Passing off state to a local process: I could use some clarification on this point. What's the end goal?
  5. Custom DSL aliases for specific commands: Not custom as in the user defines them. I purposely didn't include any concept of a function or a procedure, b/c it's not supposed to be like programming. The idea is that reading a script gives a straightforward description of of a series of steps to take on a browser. I do plan on changing up the syntax to match common sense shorthand, like click "submit" being the same as locate "submit" and click etc. I also probably want to support interaction with a programming language, most likely python or javascript. That could take the form of bindings to use SchnauzerUI in those languages and/or dedicated commands in SchnauzerUI for executing arbitrary code. An RPA tool that does both of these things is TagUI and it's really handy. 8.Background daemon: Yeah absolutely. The infrastructure for this is kind of in place, because SchnauzerUI scripts execute against running webdriver server compliant processes. So right now you can pass the byod flag to turn off driver management and then specify a port for a running Selenium server. I would like to make a setup like that a lot simpler to set up and maintain though, so I could see including maybe subcommands to start and stop background services rather than doing it for every script run or something.

To add on to all this, here are some other things rattling around in my brain.

  1. Should there be a GUI? Or maybe IDE support, or a really high quality TUI?
  2. I'd REALLY like to settle on a core service and implement a plugin API. I was experimenting with a project called extism to base it around webassembly, but of course tokio and webassembly don't fit together. Still, even though I love Rust, I don't necessarily want the people writing plugins to be forced to use it.
  3. I need to make a lot of changes to get this project to "professional" status, including a versioning/release strategy, a good story around installation, and a much more robust testing setup.
bcpeinhardt commented 1 year ago

The numbering got all fucked up but it'll be alright xD

bcpeinhardt commented 1 year ago

So we now have a project board going for SchnauzerUI, so I'm gonna close this ticket :)