search

opengeospatial / ets-sta10

Repository for the Executable Test Suite for OGC Sensor Things API
Other
6 stars 8 forks source link

Error received when testing java.lang.AssertionError: Reading relation of the entity failed #43

Closed mariakrommyda closed 5 years ago

mariakrommyda commented 5 years ago

Hello everyone,

I tried to run the online test suit for the Core but there are two failed tests.

check Resource Paths and read Property Of Entity And Check Response return java.lang.AssertionError: Reading relation of the entity failed:
[Things] expected
[200] but found
[404] and java.lang.AssertionError: Reading property value of "name" of the exitixting THING with id 2 failed. expected
[200] but found
[404] Is there a way to find the http call that failed? Or get more information about the exact error? If I am interpreting the error correctly, it implies that the test cannot retrieve the Things, Things(2) and Thing(2)/name but the below resources are available. http://147.102.5.93:8090/SensorThing/v1.0/Things http://147.102.5.93:8090/SensorThing/v1.0/Things(2) http://147.102.5.93:8090/SensorThing/v1.0/Things(2)/name

Would installing the test suit locally provide me with more logs and information?

Thank you for you time, Maria.

taniakhalafbeigi commented 5 years ago

Hi Maria,

The first error relates to reading related entities for Things. I will check further to find the exact test that fails and get back to you, but in the meantime please check http://147.102.5.93:8090/SensorThing/v1.0/HistoricalLocations(149)/Thing and http://147.102.5.93:8090/SensorThing/v1.0/Things(7) But you are right, in this case we should return a better error and we will try our best to improve it, thanks for finding it.

The second issue is for property value which means /$value. (http://147.102.5.93:8090/SensorThing/v1.0/Things(2)/name/$value)

Please let me know if you have any other issue.

hylkevds commented 5 years ago

When debugging an error like this it's easiest to run the suite locally. It will generate an XML file with the test report, that will tell you which test failed. You can then run the suite in a debugger, to see why exactly the test fails.

mariakrommyda commented 5 years ago

@taniakhalafbeigi Thank you for your response. I was not testing with these data when I did the test, these were some test data inserted yesterday, I forgot to roll back the database, sorry for that. I did this now, and the error persists, even though I cannot find the offending link. I resolved the second issue thank you for pointing me to the right direction.

@hylkevds I think that I can download the same files from the online test, not sure if they are exactly the same. Can you point me to the right file that I should check in similar cases?

hylkevds commented 5 years ago

  1. First, clone the project: git clone https://github.com/opengeospatial/ets-sta10.git

  2. edit the file src/main/resources/test-run-props.xml and set your server url and required conformance level.

  3. Build the project with mvn install (currently this fails on java 11)

  4. Run the project with java -jar target/ets-sta10-1.3-SNAPSHOT-aio.jar src/main/resources/test-run-props.xml

  5. On Linux, you should now find a folder called testng in your home directory, with a folder containing the test results xml file. I don't know where this folder is created on windows.

To run the tests in your IDE (Netbeans, Eclipse, IntelliJ) you import/open the project in your IDE, set the main class to be run to org.opengis.cite.sta10.TestNGController with one parameter: target/classes/test-run-props.xml

Once you have the test results file, you should see a line at the top that reads something like: <testng-results total="??" passed="??" failed="?" skipped="?"> If the number for failed is not zero, you can search the file for "FAIL". This should lead you to the test that actually failed. It looks something like:

        <test-method signature="testMethodName()[pri:0, instance:org.opengis.cite.sta10.packagename.TestClassName@GitId]" started-at="????-??-??T??:??:??Z" name="testMethodName" description="Test description" groups="level-?" finished-at="????-??-??T??:??:??Z" duration-ms="???" status="FAIL">
          <exception class="java.lang.AssertionError">
            <message>
              <![CDATA[ The actual error message is here... ]]>
            </message>
          </exception> <!-- java.lang.AssertionError -->
          <reporter-output>
          </reporter-output>
        </test-method> <!-- testMethodName -->

In your IDE you can now find the Class that the test is in (the class name is in the signature attribute) and the test method name (in the name attribute). Once you have found the test method, you can set a breakpoint on it, re-run the test, and step through the process to see what exactly it does and expects.

mariakrommyda commented 5 years ago

@hylkevds Thank you for your response, you can access these files after the online test too. The error that I get there though is the same that I see at the test summary so it is not offering any more help.

I was hoping to somehow see the specific http request that fails, to help me identify the cause of the problem. To my understanding this is caused by a broken iot.navigationLink.

hylkevds commented 5 years ago

Unfortunately, the only way to currently find that out is to use a debugger and step through the test method.

taniakhalafbeigi commented 5 years ago

@mariakrommyda I debugged your case locally and I think the problem is for GETting the association link using $ref: http://147.102.5.93:8090/SensorThing/v1.0/Things/$ref

I also added the issue for making the error messages more detailed with the URL to the failed request. (#44 )

Please let me know if you have any issues later that you cannot find the cause and I will help you with that. In the meantime we will work on improving the error messages to be more informative.

mariakrommyda commented 5 years ago

@hylkevds Thank you for the clarification, I was set up the debugger and it is way easier to identify any problems.

@taniakhalafbeigi Thank you for that, yes that was the issue. It would be indeed nice to get the URL that failed in an easier way.

keshavnangare commented 5 years ago

@taniakhalafbeigi

Currently I am working on this issue #44 and trying to reproduce this issue but got the following error:

org.json.JSONException: JSONObject["url"] not found.
    at org.json.JSONObject.get(JSONObject.java:498)
    at org.opengis.cite.sta10.SuiteFixtureListener.checkServiceRootUri(SuiteFixtureListener.java:170)
    at org.opengis.cite.sta10.SuiteFixtureListener.processSuiteParameters(SuiteFixtureListener.java:68)
    at org.opengis.cite.sta10.SuiteFixtureListener.onStart(SuiteFixtureListener.java:36)
    at org.testng.SuiteRunner.invokeListeners(SuiteRunner.java:199)
    at org.testng.SuiteRunner.run(SuiteRunner.java:252)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    at org.testng.TestNG.run(TestNG.java:1057)
    at com.occamlab.te.spi.executors.testng.TestNGExecutor.execute(TestNGExecutor.java:112)
    at org.opengis.cite.sta10.TestNGController.doTestRun(TestNGController.java:116)
    at org.opengis.cite.sta10.TestNGController.main(TestNGController.java:61)
Exception in thread "main" java.lang.IllegalArgumentException: The service response for the root URI "http://147.102.5.93:8090/SensorThing/v1.0/Things" is not JSON.
    at org.opengis.cite.sta10.SuiteFixtureListener.processSuiteParameters(SuiteFixtureListener.java:70)
    at org.opengis.cite.sta10.SuiteFixtureListener.onStart(SuiteFixtureListener.java:36)
    at org.testng.SuiteRunner.invokeListeners(SuiteRunner.java:199)
    at org.testng.SuiteRunner.run(SuiteRunner.java:252)
    at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
    at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
    at org.testng.TestNG.runSuitesSequentially(TestNG.java:1224)
    at org.testng.TestNG.runSuitesLocally(TestNG.java:1149)
    at org.testng.TestNG.run(TestNG.java:1057)
    at com.occamlab.te.spi.executors.testng.TestNGExecutor.execute(TestNGExecutor.java:112)
    at org.opengis.cite.sta10.TestNGController.doTestRun(TestNGController.java:116)
    at org.opengis.cite.sta10.TestNGController.main(TestNGController.java:61)

This is failed at the line: https://github.com/opengeospatial/ets-sta10/blob/48607e50b5b08e3a1b5fb139f4fbf40398603667/src/main/java/org/opengis/cite/sta10/SuiteFixtureListener.java#L170

[1] http://147.102.5.93:8090/SensorThing/v1.0/Things

According to exception the entity object is not able to find 'url' from the service[1] response. Could you please point me to the correct service?

hylkevds commented 5 years ago

Since the exception happens in the checkServiceRootUri() method, It seems you are passing the url http://147.102.5.93:8090/SensorThing/v1.0/Things as a service root url. However, this url is not a service root url, it is the url to the Things collection. You probably meant to set http://147.102.5.93:8090/SensorThing/v1.0 as the service root url

You did find a bug there. Instead of if (entity.get("url") == null) {, the check should probably be if (!entity.has("url")) {, since entity.get will throw an exception if the requested entity does not exist, and thus the error that is meant to be returned is never returned.

taniakhalafbeigi commented 5 years ago

Hi @keshav-nangare. Thanks a lot for helping out with #44. Hylke is right. I think you passed the wrong URL to the test suite. Main SensorThings url that is the input parameter for the test suite should end wight after /v1.0. Things at the end is the resource entity and the test suite would check that url through testing, and there is no need to put that in the input url. So please repeat your procedure with http://147.102.5.93:8090/SensorThing/v1.0. If you encounter another error please let me know.

keshavnangare commented 5 years ago

Thanks @hylkevds @taniakhalafbeigi .

I am able run the test suite but the method checkResourcePaths from class org/opengis/cite/sta10/sensingCore/Capability1Tests is failed and got the error "org.json.JSONException: JSONArray[0] not found." at the line https://github.com/opengeospatial/ets-sta10/blob/48607e50b5b08e3a1b5fb139f4fbf40398603667/src/main/java/org/opengis/cite/sta10/sensingCore/Capability1Tests.java#L290

The request returned empty response with {"value":[]}, then it tries to access value from the json object which is empty.

I have added the null check. What should be the behaviour of the test if response is empty and not able to get the id?

The resource URL= http://147.102.5.93:8090/SensorThing/v1.0/Things(15)/HistoricalLocations(152)/Locations(150)/Things

taniakhalafbeigi commented 5 years ago

@keshav-nangare I can refer to Hylkes answer on #45.

The test suite assumes the entities on the server are the ones described in the Preconditions on the page: http://opengeospatial.github.io/ets-sta10/

The test suite wants to test many possible relations. It can only do that if those relations actually exist. Technically, an empty server is completely standards conform, but the suite can not actually do any tests on an empty server. One could argue that the suite itself should create the entities it needs to test, but then the read tests would depend on a correct implementation of the write functionality, as explained in #3

Basically, in the standard {"value":[]} is perfectly okay as the response of the request for v1.0/Things(15)/HistoricalLocations(152)/Locations(150)/Things. But in order for the test suite to check out the relations, we expect and rely on certain relations to exist. One reason could be in this case, we don't know whether the response array is empty because there is no Things for that Locations or because the service does not return correct answers.

keshavnangare commented 5 years ago

@taniakhalafbeigi

Thanks for your inputs.

If we get the response {"value":[]}: Should we throw exception and stop the further execution? OR Can we add return statement if the object is null and not able to find the id(line 290) and it will continue execution?

https://github.com/opengeospatial/ets-sta10/blob/48607e50b5b08e3a1b5fb139f4fbf40398603667/src/main/java/org/opengis/cite/sta10/sensingCore/Capability1Tests.java#L278-L314

hylkevds commented 5 years ago

In this case, you should fail an Assert with a message like: "Expecting a non-empty list for request [insert request here]. Please ensure the service is initialised with the entity set listed in http://opengeospatial.github.io/ets-sta10/"

taniakhalafbeigi commented 5 years ago

@keshav-nangare I agree with Hylke and the Assert should fail. If we just return without failing the request, there is a chance that we are not able to check any relations because the server is not initialized correctly. In that case we don't want the test to pass if we don't check the relations.

keshavnangare commented 5 years ago

Thank you for help! @hylkevds & @taniakhalafbeigi