test updates around 0337

iCollin commented 2 years ago

ATF Test Scripts to check https://github.com/smartdevicelink/sdl_core/issues/3715

This PR is not ready for review.

Summary

Add tests for 0337 functionality Update tests that were doing PTUs when Core wasn't expecting a PTU

ATF version

latest

Changelog

Tasks Remaining:

[x] Ext Prop tests

CLA

[x] I have signed the CLA

ShobhitAd commented 2 years ago

These tests seem to be failing with an EXTERNAL_PROPRIETARY build on my end

0976: FAILED ./test_scripts/./Policies/App_Permissions/005_ATF_DISALLOWED_app_Id_policies_And_RequestType_Validation.lua

[04:58:26,836] TestStep_SDL_Allow_SystemRequest_Of_PROPRIETARY_Type                                  [FAIL] (21989 ms)
HMI call BasicCommunication.SystemRequest: Timeout: HMI call BasicCommunication.SystemRequest: Timeout expired
Response to 4: ValidIf:

0979: FAILED ./test_scripts/./Policies/App_Permissions/008_ATF_DISALLOWED_default_policies_And_RequestType_Validation.lua

[04:58:26,131] TestStep_SDL_Allow_SystemRequest_Of_PROPRIETARY_Type                                  [FAIL] (21988 ms)
Response to 3: ValidIf:
payload.resultCode: expected: SUCCESS, actual value: REJECTED                                                                                                
HMI call BasicCommunication.SystemRequest: Timeout: HMI call BasicCommunication.SystemRequest: Timeout expired

0400: FAILED ./test_scripts/./API/VehicleData/SubscribeVehicleData/014_SubscribeVD_Resumption_of_subscription_for_2_apps_Ignition_Cycle_different_data.lua

[04:52:55,776] Register_App1                                                                         [FAIL] (12404 ms)
OnSystemRequest notification: Timeout: OnSystemRequest notification: Timeout expired

0416: FAILED ./test_scripts/./API/VehicleData/UnsubscribeVehicleData/012_UnsubscribeVD_allowed_and_disallowed_after_PTU.lua

[04:52:55,762] Register_App                                                                          [FAIL] (12469 ms)
OnSystemRequest notification: Timeout: OnSystemRequest notification: Timeout expired
PTU start event: Timeout: PTU start event: Timeout expired
HMI call BasicCommunication.PolicyUpdate: Timeout: HMI call BasicCommunication.PolicyUpdate: Timeout expired

1311: FAILED ./test_scripts/./Policies/Validation_of_PolicyTables/225_ATF_pt_update_validation_rules_request_type_array_has_one_invalid_value.lua
```
[05:03:13,343] CheckPTUinLocalPT                                                                     [FAIL] (2 ms)
AutoFail: 1: SDL has wrong values in LocalPT
```
Could you confirm whether these test pass/fail on your end and make the required changes for the failing tests

iCollin commented 2 years ago

@ShobhitAd I did see the App_Permissions tests failing and fixed those issues in https://github.com/smartdevicelink/sdl_atf_test_scripts/pull/2631/commits/5ef5c17f92f090a74ed9d51762a40529f115034d

However, I see the other three tests passing. I did have to run some of the longer ones twice, and specifically with regard to 225_ATF_pt_update_validation_rules_request_type_array_has_one_invalid_value.lua, I did see the 223 test from the same folder fail once with a similar error, but re running the test it passed.

ShobhitAd commented 2 years ago

@ShobhitAd I did see the App_Permissions tests failing and fixed those issues in 5ef5c17

However, I see the other three tests passing. I did have to run some of the longer ones twice, and specifically with regard to 225_ATF_pt_update_validation_rules_request_type_array_has_one_invalid_value.lua, I did see the 223 test from the same folder fail once with a similar error, but re running the test it passed.

You're right re-running the tests seems to fix the issue for the other three tests.

VjKlepikov commented 2 years ago

@iCollin Could you please add test scripts test_scripts/Policies/Policy_Table_Update/173_PTU_when_not_UPDATING.lua, test_scripts/Policies/Policy_Table_Update/174_PTU_from_wrong_app.lua to appropriate test_sets

iCollin commented 2 years ago

@VjKlepikov yes thank you, I have created a PR

smartdevicelink / sdl_atf_test_scripts