search

kazurayam / VisualTestingInKatalonStudio

This is a Katalon Studio project which performs 2 ways of screenshot-comparison testing for WebUI in full automation. Twins mode compares 2 host names at a time. Chronos mode compares the current view of a AUT to the previous one. You can make your own Katalon Studio project capable of the same screenshot-comparison. approach. Instruction provided.
https://forum.katalon.com/t/visual-testing-in-katalon-studio/13361
GNU General Public License v3.0
22 stars 3 forks source link

Cannot run Visualtest with vt.version 1.22.0 #32

Closed rieota238 closed 4 years ago

rieota238 commented 4 years ago

Moved from Katalon community, sorry about that.

I use your Visualtest feature and I updated your vt.version to 1.22.0 in my custom Visualtest. And I did following :

Then I got following error when I run my tests. Should I change something in my test more?

Test Cases/Exam_capture/backupCurrentTSuiteResult FAILED.
Reason:
groovy.lang.GroovyRuntimeException: Could not find matching constructor for: com.kazurayam.materials.TSuiteResultId(com.kazurayam.materials.TSuiteName, com.kazurayam.materials.TSuiteTimestamp)
    at backupCurrentTSuiteResult.run(backupCurrentTSuiteResult:21)
    at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
    at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
    at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:339)
    at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:330)
    at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:309)
    at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:301)
    at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:235)
    at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:191)
    at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:141)
    at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:90)
    at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
    at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
    at TempTestSuite1595983615557.run(TempTestSuite1595983615557.groovy:39)
07-29-2020 09:55:06 AM Test Cases/Exam_capture/backupCurrentTSuiteResult

Elapsed time: 2.147s

Test Cases/Exam_capture/backupCurrentTSuiteResult FAILED.
Reason:
groovy.lang.GroovyRuntimeException: Could not find matching constructor for: com.kazurayam.materials.TSuiteResultId(com.kazurayam.materials.TSuiteName, com.kazurayam.materials.TSuiteTimestamp)
    at backupCurrentTSuiteResult.run(backupCurrentTSuiteResult:21)
    at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
    at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
    at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:339)
    at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:330)
    at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:309)
    at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:301)
    at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:235)
    at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:191)
    at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:141)
    at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:90)
    at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
    at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
    at TempTestSuite1595983615557.run(TempTestSuite1595983615557.groovy:39)

RestorePreviousScreenshots.log.zip

FYI : I found a typo in "restorePreviousScreenshots" ... 3. user just pushed CACEL button without selecting any yyyyMMdd_hhmmss dir. CACEL -> CANCEL

kazurayam commented 4 years ago

Test Cases/Exam_capture/backupCurrentTSuiteResult

I think you created this test case. Yes, you need to change it.

Have a look at https://github.com/kazurayam/VisualTestingInKatalonStudio/blob/master/Scripts/VT/backupCurrentTSuiteResult/Script1550220558510.groovy and mimic it.

The method signature of TSuiteResult class was changed. But why? I did this change in order to solve the following problem: https://github.com/kazurayam/VisualTestingInKatalonStudio/issues/11

kazurayam commented 4 years ago

@rieota238

I have done a lot of changes to this project.

One of the changes may confuse you. https://github.com/kazurayam/VisualTestingInKatalonStudio/issues/18

Have a look at the post and prepare yourself to see a JFileChooser dialog.

rieota238 commented 4 years ago

I forgot to change Test Cases/Exam_capture/backupCurrentTSuiteResult, thanks. I changed the contents of my tests according to : https://github.com/kazurayam/VisualTestingInKatalonStudio/issues/18#issuecomment-653697984

But I still got some error at chronos exam. I read #18, but I'm sorry if I still misunderstand or I'm wrong something.

RestorePreviousScreenshots.log.zip

and

07-29-2020 11:19:32 AM Test Cases/Exam_capture/ImageDiff_chronos

Elapsed time: 2.149s

Test Cases/Exam_capture/ImageDiff_chronos FAILED.
Reason:
groovy.lang.MissingMethodException: No signature of method: com.kazurayam.visualtesting.ImageDiffer.runChronos() is applicable for argument types: (java.lang.String, com.kazurayam.visualtesting.ImageCollectionDifferDriver$ChronosOptions) values: [VisualTest/chronos_capture, {"shiftCriteriaPercentageBy":2.0,"filterDataLessThan":2.0}]
Possible solutions: runChronos(java.lang.String, java.lang.String, com.kazurayam.visualtesting.ImageCollectionDifferDriver$ChronosOptions)
    at ImageDiff_chronos.run(ImageDiff_chronos:28)
    at com.kms.katalon.core.main.ScriptEngine.run(ScriptEngine.java:194)
    at com.kms.katalon.core.main.ScriptEngine.runScriptAsRawText(ScriptEngine.java:119)
    at com.kms.katalon.core.main.TestCaseExecutor.runScript(TestCaseExecutor.java:339)
    at com.kms.katalon.core.main.TestCaseExecutor.doExecute(TestCaseExecutor.java:330)
    at com.kms.katalon.core.main.TestCaseExecutor.processExecutionPhase(TestCaseExecutor.java:309)
    at com.kms.katalon.core.main.TestCaseExecutor.accessMainPhase(TestCaseExecutor.java:301)
    at com.kms.katalon.core.main.TestCaseExecutor.execute(TestCaseExecutor.java:235)
    at com.kms.katalon.core.main.TestSuiteExecutor.accessTestCaseMainPhase(TestSuiteExecutor.java:191)
    at com.kms.katalon.core.main.TestSuiteExecutor.accessTestSuiteMainPhase(TestSuiteExecutor.java:141)
    at com.kms.katalon.core.main.TestSuiteExecutor.execute(TestSuiteExecutor.java:90)
    at com.kms.katalon.core.main.TestCaseMain.startTestSuite(TestCaseMain.java:157)
    at com.kms.katalon.core.main.TestCaseMain$startTestSuite$0.call(Unknown Source)
    at TempTestSuite1595988849302.run(TempTestSuite1595988849302.groovy:39)
kazurayam commented 4 years ago

Test Cases/Exam_capture/ImageDiff_chronos

I think you created this test case. Yes, you need to change it.

Have a look at https://github.com/kazurayam/VisualTestingInKatalonStudio/blob/master/Scripts/CURA/ImageDiff_chronos/Script1554799216780.groovy and mimic it. Especially you should look at the following part:

boolean result = imageDiffer.runChronos(TESTSUITE_ID, EXECUTION_PROFILE, options)
                                                      ^^^^^^^^^^^^^^^^^^
rieota238 commented 4 years ago

sorry to bother you, but would you check following log attached previous comment too? It's big and cannot attach directly. I had changed the ImageDiff_chronos file, but the log still occur after choosing previous screenshots folder.

RestorePreviousScreenshots.log.zip

Test Cases/Exam_capture/ImageDiff_chronos

Thanks, I removed whole CURA folder without checking..., so I replace this file to latest one... sorry.

rieota238 commented 4 years ago

to make sure, latest log (after fixing ImageDiff_chronos)

log.log

kazurayam commented 4 years ago

I guess, you have Storage　directory　containing　folders/files　created　by　the　older　version　of　VisualTestingInKatalonStudio framework.

Please delete the Storage directory and retry.

The upgraded version has changed the folder tree structure under the Materials dir and the Storage dir, so the upgraded version can not process the file/folders in the Storage directory created by the previous version.

kazurayam commented 4 years ago

@rieota238

I removed whole CURA folder without checking..., so I replace this file to latest one... sorry.

By the way, if you execute the following command in Terminal of Mac, all the CURA-related folders/files will be forcibly restored.

> cd $youVisualTestingProject
> gradle enableVisualTesting
rieota238 commented 4 years ago

Thank you for your kind explanation, now I could run my test by deleting Storage folder. Sorry to bother you.

kazurayam commented 4 years ago

Glad to hear that.

kazurayam commented 4 years ago

You may notice that the new version performs far better than the old one.

The old version had a Memory-leak issue: https://github.com/kazurayam/VisualTestingInKatalonStudio/issues/13

This problem has been completely resolved.

@rieota238

You would not need -Xmx=8G nor 10G for the new version. Please try -Xmx=2G. It should work now.

kazurayam commented 4 years ago

We also experienced slowness:

https://github.com/kazurayam/Materials/issues/36

This problem has been fixed already.

rieota238 commented 4 years ago

Sorry to bother you again, I could run my test in GUI, but it seems to get stuck in console mode at restorePreviousScreenshots tc. It got stuck over 3 hours in the test and had been canceled automatically.

Precondition:

I run my test that could be run locally by choosing folder in azure DevOps Pipeline - and it got stuck at restorePreviousScreenshots tc. We could not get any useful log because it didn't sent any log to Katalon DevOps Center due to stuck. And as mentioned before, we cannot run it locally in console mode...

What are the possible causes?

My understanding... If I run the test in console mode, restorePreviousScreenshots should call Test Cases/VT/restorePreviousTSuiteResult and choose last (initial setting)screenshots automatically , right?

kazurayam commented 4 years ago

Sorry for being not responsive for months.

but it seems to get stuck ... What are the possible causes?

I have very little idea.

I suppose that your problem is not caused by bugs in my application.

I guess, there could be some problem in the communication layer between the testing process and the network storage where you saved Screenshots.

Does the communication work on SSL? When the communication for SSL Certification has problem, it will appear that the application process hangs and not exiting. I can list a reference though it may not be relevant to you: