search

telefonicaid / fiware-orion

Context Broker and CEF building block for context data management, providing NGSI interfaces.
https://github.com/telefonicaid/fiware-orion/blob/master/doc/manuals/orion-api.md
GNU Affero General Public License v3.0
210 stars 265 forks source link

[Question] About Orion functional test on aarch64 #3623

Closed fisuda closed 4 years ago

fisuda commented 4 years ago

Dear @fgalan,

I'm trying to run some FIWARE GEs on Raspberry Pi. I have built the Orion 2.3.0 binary for aarch64 and run the Orion Functional Test with it. I have a result in which 1 error and 5 warnings are. It would be great if you could give me an advice.

Orion Functional Test Log File:
================================================================================
Orion Functional tests starting
Sun Feb  9 04:22:37 UTC 2020

-----  Notifications of different sizes  -----
(0000_large_requests/notification_different_sizes.test) output not as expected
VALIDATION ERROR: input line:
   Sending message 1 to HTTP server: sending message of 19485 bytes to HTTP server
does not match ref line:
   Sending message 1 to HTTP server: sending message of REGEX((19490|19491|19492|19493|19494|19495|19496)) bytes to HTTP server

================================================================================

----------- Failing tests ------------------
  o  0000_large_requests/notification_different_sizes.test

WARNING: 5 test cases disabled:
  o  2: 0000_bad_requests/exit.test
  o  30: 0000_ipv6_support/ipv4_only.test
  o  31: 0000_ipv6_support/ipv6_only.test
  o  390: 0917_queryContext_behaves_differently/query_with_and_without_forwarding.test
  o  624: 1310_suspect_200OK/suspect_200OK.test
make: *** [functional_test] Error 1

Please see the whole test log here.

  1. Error in notification_different_sizes.test
VALIDATION ERROR: input line:
   Sending message 1 to HTTP server: sending message of 19485 bytes to HTTP server
does not match ref line:
   Sending message 1 to HTTP server: sending message of REGEX((19490|19491|19492|19493|19494|19495|19496)) bytes to HTTP server

It seemds that the size of the sending message has been 5byte less than the expected minimum size. The result of this testcase is here and the messages sent to an accumulator server is here.

And I also ran the testcase on x86_64. But the result was same as on aarch64. The details of the result is here.

Is my testing procedure wrong? What kind of things do you think could be the cause?

  1. Warnings

Is there anything that I need to do about these warnings?

  1. Test environment

The Orion and functional test script are run in a docker container.

Thanks

fgalan commented 4 years ago

I think that you can ignore the error you get. The 0000_large_requests/notification_different_sizes.test is somehow weak (note all the options already included in the REGEX, which means that it was adjusted several times in the past) and if the rest of ~1100 test are working, then it's fine.

However, if you have been able to run the test in a different environment in which it works, an interesting analysis would be to compare the messages in that case (success) with the ones you got at https://github.com/fisuda/report/tree/master/orion/20200209_orion/aarch64_notification_different_sizes/sending_message (failing) in order to know where the 5 bytes difference comes from.

Thanks for the feedback!

fisuda commented 4 years ago

Thank you for your advice.

I also ran the test on x86_64. https://github.com/fisuda/report/tree/master/orion/20200209_orion/x86_64_notification_different_sizes I compared the messages, but I wasn't able to find difference.

$ diff aarch64_notification_different_sizes/se
nding_message/Sending_message_1.txt x86_64_notification_different_sizes/sending_message/Sending_message_1_x86_6
4.txt
8c8
< Fiware-Correlator: e30bed9a-4b06-11ea-bceb-0242ac110002
---
> Fiware-Correlator: f012e480-4be2-11ea-b642-000c29ed72bc
32,33c32,33
<     "subscriptionId": "5e3fa911ddaebf848df8cc9e"
< }
---
>     "subscriptionId": "5e411a41d2842ce1633ca5f0"
> }
\ No newline at end of file

I ran the test with Orion 2.3.0, not Orion 2.3.0-next. Is there a possibility that the 5 bytes difference is '-next'?

fgalan commented 4 years ago

I ran the test with Orion 2.3.0, not Orion 2.3.0-next. Is there a possibility that the 5 bytes difference is '-next'?

I haven't thought of that possibility, but it may fully sense! :)

The Orion version is used in the User-Agent header in the notification HTTP request sent by Orion, and the headers also count for the Sending message 1 to HTTP server count.

It would be great if you could confirm this theory comparing the headers in the notifications in both cases.

In this theory gets confirmed, maybe we should simplify the line to REGEX((19485|19490)) and leave it a comment in the .test for the future (e.g. when we step from Orion 2.9.0 to Orion 2.10.0 it would be adjusted to REGEX((19486|19491))).

Thanks!

fisuda commented 4 years ago

Thank you for your quick reply. I will run this testcase with Orion 2.3.0-next on x86_64 and aarch64 and report on the result.

fisuda commented 4 years ago

I have run the test on x86_64 and aarch64. The Orion version used in the User-Agent header was orion/2.3.0-next. The Content-Length was same size. I think that the 5 bytes difference is -next.

$ LANG=C diff 20200209_orion/x86_64_notification_different_sizes/sending_message/Sending_message_1_x86_64.txt 20200212_orion/x86_64_notification_different_sizes-next/sending_message-next/Sending_message_1_x86_64.txt 
4c4
< User-Agent: orion/2.3.0 libcurl/7.29.0
---
> User-Agent: orion/2.3.0-next libcurl/7.29.0
8c8
< Fiware-Correlator: f012e480-4be2-11ea-b642-000c29ed72bc
---
> Fiware-Correlator: 1d743a98-4d80-11ea-a213-000c29ed72bc
32,33c32,33
<     "subscriptionId": "5e411a41d2842ce1633ca5f0"
< }
\ No newline at end of file
---
>     "subscriptionId": "5e43cf736ab581a3f0a6d7e4"
> }

The updating of REGEX value is not only needed for Sending message 1, but also for Sending message 2 and 3.

Sending message Orion 2.3.0 Orion 2.3.0-next Content-Length
1 19485 19490 19252
2 21485 21490 21252
3 8100487 8100492 8100252

(bytes)

The REGEX values are in the following table.

REGEX Orion d.d.d Orion d.d.d-next Orion d.dd.d Orion d.dd.d-next
1 19485 19490 19486 19491
2 21485 21490 21486 21491
3 8100487 8100492 8100488 8100493

I have confirmed the same result on aarch64.

The detail of the result is the following link. https://github.com/fisuda/report/tree/master/orion/20200212_orion

Thank you!

fgalan commented 4 years ago

Having merged PR #3625, maybe this issue should be closed? Or is anything pending?

Thanks!

fisuda commented 4 years ago

This issue has been solved. We can close it. I will send a PR about building Orion for aarch64 at a later date.

Thank you!