search

IBM / xmlservice

XML-based interface for accessing IBM i resources
https://xmlservice.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
35 stars 18 forks source link

Convert tests to not require external toolkits #20

Open kadler opened 5 years ago

kadler commented 5 years ago

There are various tests already, but they are written using the PHP or Python toolkits. These should be converted to not require these toolkits and call XMLSERVICE directly.

Steps involved:

I imagine the test framework being being basically a set of input XML files and corresponding expected output XML files, along with some sort of test runner.

clarkphp commented 5 years ago

This makes so much sense. But then, you knew that, already. I may be able to help with this, or #21 .

clarkphp commented 5 years ago

Test Infrastructure

Do you envision an xUnit-style framework for running tests, or a test infrastructure that is rather pared down in comparison? If one thinks in terms of:

the tool can get complicated quickly.

What internal or external dependencies or programmming language(s) are you willing to carry along for the ride, in order to make contributing tests easier for the community? You explicitly desire to remove Python and PHP as dependencies, which makes sense. What about Node.js, recalling that barryCI has been mentioned as a potentially useful tool elsewhere, or bash scripts?

Simplifying Assumptions (or, Non-simplifying Questions)

Regarding the testing infrastructure, it might be useful to make simplifying assumptions. For example, ignoring the concept of test suites, and having the runner execute all test cases found in a given directory, means you don't have to write and test infrastructure responsible for determining whether or not a test case is in a relevant group, and thus should or shouldn't be run.

The runner could either cease test execution upon the first failure, or report all failures, and continue running all tests.

It sounds like you plan for the runner to merely compare actual output XML against expected output XML, and maybe not concern itself with warnings vs. errors, or perhaps the effect of an action initiated on the target IBM i system.

Ignoring levels of error would be a simplifying assumption. Not checking the result of an action initiated by the XMLSERVICE could be debatable. If input XML requests a change to the contents of a Data Area, should a corresponding test case verify the Data Area appropriately changed from a known beginning state to the expected ending state?

Clearly, actual vs. expected output XML is the necessary first step. Does the above mean the ordering of test case execution becomes important? Use one test case as the setup for a subsequent test case, or provide for test fixtures in each test case? Instead of test fixtures, is it simpler to ensure test cases are named in such a way that they execute in a sequence that allows one test case to set up a known state for a later test case (at the cost of losing more full test case isolation)?

If I'm overthinking the process of simplification, or missing the obvious, it wouldn't be the first time. Just let me know!

kadler commented 5 years ago

I'd love to have some sort of unit tests, but I have no idea how to do them in RPG. For this issue, I just want to take the existing tests and remove any downstream dependency (ie. PHP toolkit). These would be complete functional integration tests and focus on the XMLSERVICE interface, not on the actual XMLSERVICE actions (ie. it won't catch bugs changing a data area to the correct value).

I don't mean to ban either PHP or Python as a dependency, just that these are XMLSERVICE tests, not PHP toolkit or Python toolkit tests, so they should not depend on those toolkits to run.

As far as ordering of tests, my belief is that in an ideal world all tests should be completely self-sufficient and can run in whatever order they want and ideally all run in parallel. We don't live in an ideal world, though so there's always gonna be some limitations to that. However, I have a thorough distaste for "tests" which merely exist to do setup and are not actually tests at all. I much prefer test fixtures be used instead.

brandonp42 commented 1 year ago

@kadler would it be acceptable to do tests with PHP or NodeJS just directly using the ODBC driver and no other toolkits? On the other issues I've been looking at this last week I've just been using ACS Run SQL Scripts but it'd be "relatively" easy to automate things if we could use PHP/NodeJS as a conduit for the ODBC driver.

kadler commented 1 year ago

@brandonp42 That's definitely something I've thought about, though I'd prefer to have as few layers involved as possible.

My original idea was to have Python 3 use ctypes to call the XMLSERVICE APIs directly through PASE _ILECALL. However, thinking more recently, I don't know if that's wise since then XMLSERVICE job will always have PASE started when it's called.

Instead, I was thinking instead of calling XMLSERVICE directly, we build a new ILE test runner program which gets spawned by the ILE spawn() API (still called through ctypes from Python, though). This way, we can pass file descriptors from PASE to the ILE program to do I/O, but the program called through spawn() will not have PASE activated.

However, thinking even more about this, there are definitely benefits to testing through the database:

So I think I may be convinced to use database access to do the testing. As far as language to build the test framework in, I'd prefer Python since I'm most familiar with it and it's got ctypes in case we do want to do some "direct" testing still. In addition, it comes with great built-in XML libraries and pytest is great. The problem with Python is you become dependent on system packages for the database access (or you need compilers installed to build pyodbc). With node.js, we can depend on idb-connector/node-odbc and the pre-built binaries get installed without any extra work and I'm sure there's plenty of XML packages available we could use, but it does make calling directly much more difficult unless there's an equivalent to ctypes out there or we build a bespoke module.

kadler commented 1 year ago

I do greatly appreciate the recent PRs you've made but I admit I have not done much with them because I have little experience writing RPG code (and the XMLSERVICE RPG code is so cryptic to have been written by aliens) and we have no real way to test for regressions.

Having this issue resolved with a working (and hopefully comprehensive) test suite would make it a lot easier to handle these PRs (and tackle bigger things like #24), though it's definitely not a requirement to merging those PRs.

brandonp42 commented 1 year ago

I think I understand where you are coming from, and to be honest the test situation makes me reluctant to do more than I already have. Some of the code is pretty intimidating and there's a very real risk of breaking something while working on it. The prudent thing would definitely be to get better testing in place before accepting any non-trivial changes.

I'm most familiar with PHP first and NodeJS second, a little but not much Python... so realistically for me to work on this ODBC via NodeJS is probably the most achievable option right now given pre-built binary considerations. It's not ideal but maybe long term some tests via a mix of Python and NodeJS wouldn't be so bad?

I dunno, I would like to see this project be able to move forward more effectively and honestly some of the issues are causing me a bit of heartburn so I'd like to see them resolved. But I also don't want to be the guy everybody points at and says I broke it LOL.

brandonp42 commented 1 year ago

Also please see https://github.com/php/pecl-database-ibm_db2/issues/46 for some commentary on needing a disposable test environment to minimize security risks and so on.