utPLSQL / utPLSQL-cli

Command line client for invoking utPLSQL
Apache License 2.0
40 stars 15 forks source link

Excessive reporters sessions #120

Open jgebal opened 5 years ago

jgebal commented 5 years ago

When I run utPLSQL-cli with multiple reporters and outputs like:

time utPLSQL-cli/bin/utplsql run ${UT3_TESTER}/${UT3_TESTER_PASSWORD}@${CONNECTION_STR} \
-source_path=source -owner=ut3 \
-test_path=test -c \
-f=ut_sonar_test_reporter     -o=test_results.xml \
-f=ut_junit_reporter          -o=junit_test_results.xml \
-f=ut_tfs_junit_reporter      -o=tfs_test_results.xml \
-f=ut_sonar_test_reporter     -o=test_results1.xml \
-f=ut_junit_reporter          -o=junit_test_results1.xml \
-f=ut_tfs_junit_reporter      -o=tfs_test_results1.xml \
-f=ut_sonar_test_reporter     -o=test_results2.xml \
-f=ut_junit_reporter          -o=junit_test_results2.xml \
-f=ut_tfs_junit_reporter      -o=tfs_test_results2.xml \
-f=ut_sonar_test_reporter     -o=test_results3.xml \
-f=ut_junit_reporter          -o=junit_test_results3.xml \
-f=ut_tfs_junit_reporter      -o=tfs_test_results3.xml \
-f=ut_sonar_test_reporter     -o=test_results4.xml \
-f=ut_junit_reporter          -o=junit_test_results4.xml \
-f=ut_tfs_junit_reporter      -o=tfs_test_results4.xml \
-f=ut_documentation_reporter  -o=test_results.log -s

Ever reporter allocates separate session. This can be challenging for environments, where users have limit of concurrent sessions.

It seems that API/cli could use only 2 sessions and still be successful. session 1 - main session for running tests and collecting report outputs. session 2 (asynchronous) for collecting real-time report info and putting it into screen.

Alternatively, maybe there should be a parameter for cli to set maximum number of concurrent sessions. This could be minimum of 2 - like in the example above and max could be as many as needed. If set to 0, there is no limit, if set to 1 all is synchronous. default = 2?

pesse commented 5 years ago

I see the potential problem, but I ask myself: Is this a real issue at the moment? Do we have real-life scenarios currently like that? I'd even wonder why anyone would use so many reporters (if not for testing purpose).

I'd also not introduce another parameter for something like that. I'm not sure we won't run into serious troubles if we use only one single connection and if we change the internal behavior of connection usage to 2, this is nothing I want a user to decide (I also see no benefit in passing that decision to the user).

jgebal commented 5 years ago

Well... It's probably not just hypothetical issue. I actually have higly shared DB, where developer accounts have concurrent session limit of 4. So if I want 4 reports I neet to run utPLSQL twice as 4 reports require 5 sessions.

Flexibility of max sessions as parameter would be great but isn't a must have. Realistic request:

pesse commented 5 years ago

Ok, so we have a real-life need to reduce the concurrent sessions.

Can you explain what's the benefit you see in adding increased complexity by adding a parameter for "max sessions"?

jgebal commented 5 years ago

Potentially faster execution time if reading reporters output takes more time. This is however only a hypothesis so probably no real benefit until proven otherwise.

pesse commented 5 years ago

Thought about this topic again. What we could do: 1 Session to run the tests 1 Session to (live-)consume the results of the reporter that goes to screen or the first mentioned reporter 1 Session to consume all the other reporters after the run itself is finished

That way we would always use at most 3 sessions. What do you think @jgebal ?

pesse commented 5 years ago

Well, actually we could reuse the session of the live-consuming reporter so it would be 2 sessions at all. At the cost of the secondary reporters only being written after the complete test-run.