Test Failed When ALIGNED_MEMORY_MODEL Is Enabled

[hsinghaowang]

WE have tried to enable ALIGNED_MEMORY_MODEL to test KVS.

However, we still see segmentation fault in kvspic_test which you can refer to attached kvspic_test.log. It failed at ThreadFunctionalityTest.ThreadCreateAndReleaseSimpleCheck

We also changed heap size as the followings when we test this

define HEAP_PERF_TEST_VIEW_ITEM_COUNT 150

define HEAP_PERF_TEST_MULTI_VIEW_ITEM_COUNT 150 * 100

define HEAP_PERF_TEST_MIN_ALLOCATION 200

define HEAP_PERF_TEST_MULTI_VIEW_MIN_ALLOCATION 64

define HEAP_PERF_TEST_ALLOCATION_DELTA 50

define HEAP_PERF_TEST_ITERATION_COUNT 10000

define HEAP_PERF_TEST_MULTI_VIEW_ITERATION_COUNT 100000

define HEAP_PERF_TEST_SIZE 32 1024 1024

kvspic_test.log

[MushMal]

@hsinghaowang you are receiving STATUS_THREAD_NOT_ENOUGH_RESOURCES when attempting to create a thread. This is caused by your pthread library returning EAGAIN as a result for pthread_create API which is likely caused by stack size limitation on your platform.

The crash itself comes from the unit test simply continuing without the thread creation and access violating (expected).

Please revert all your changes - those are not needed.

Set CONSTRAINED_DEVICE compiler define

https://github.com/awslabs/amazon-kinesis-video-streams-pic/blob/master/src/utils/src/Thread.c#L146

Run the tests again.

[hsinghaowang]

I revert all my changes. and set CONSTRAINED_DEVICE It failed at

[----------] 3 tests from HeapPerfTest
[ RUN      ] HeapPerfTest.randomAllocFreeWindowedPerf

[0x0000000076fb2000] [level 5] AIVHeap aivHeapInit(): Failed to allocate heap with limit size 134217728
[0x0000000076fb2000] [level 5] Heap heapInitialize(): Failed to initialize native heap.../../src/heap/tst/HeapPerfTest.cpp:82: Failure
Expected equality of these values:
  ((UINT32) 0x00000000)
    Which is: 0
  heapInitialize(128 * 1024 * 1024, 20, FLAGS_USE_AIV_HEAP, &pHeap)
    Which is: 4
Segmentation fault

Here is the entire log: kvspic_test_03-18.log

[MushMal]

It's getting STATUS_NOT_ENOUGH_MEMORY trying to allocate a large heap to run the perf tests. The perf tests should be run with enough memory. You can disable them just like the tavis.yml file is doing:

https://github.com/awslabs/amazon-kinesis-video-streams-pic/blob/master/.travis.yml#L40

Please close this issue once you get a clean run with the aligned memory model

[hsinghaowang]

Disabe perf tests:

AWS_KVS_LOG_LEVEL=7 ./kvspic_test --gtest_break_on_failure --gtest_filter="-imerQueueFunctionalityTest.*:HeapPerfTest.*"

Failed at:

[0x0000000076f45000] [level 3] platform-utils TearDown(): Final remaining allocation size is 0
[       OK ] SemaphoreFunctionalityTest.releaseBasicTest (220 ms)
[ RUN      ] SemaphoreFunctionalityTest.acquireAwaitingTest

[0x0000000076f45000] [level 3] platform-utils SetUp(): 
Setting up test: SemaphoreFunctionalityTest
../../src/utils/tst/Semaphore.cpp:165: Failure
Expected equality of these values:
  0
  globalAtomicLoad(&threadCount)
    Which is: 65
Segmentation fault

entire log: kvspic_test_03-18_2.log

[MushMal]

This is likely caused by a large number of threads being created in the tests. Try to change SEMAPHORE_TEST_THREAD_COUNT to a lower value.

https://github.com/awslabs/amazon-kinesis-video-streams-pic/blob/master/src/utils/tst/Semaphore.cpp#L121

There are other instances too in that file.

Sent from my iPhone

On Mar 17, 2020, at 10:30 PM, hsinghaowang notifications@github.com wrote:



Disabe perf tests:

AWS_KVS_LOG_LEVEL=7 ./kvspic_test --gtest_break_on_failure --gtest_filter="-imerQueueFunctionalityTest.:HeapPerfTest."

Failed at:

[0x0000000076f45000] [level 3] platform-utils TearDown(): Final remaining allocation size is 0 [ OK ] SemaphoreFunctionalityTest.releaseBasicTest (220 ms) [ RUN ] SemaphoreFunctionalityTest.acquireAwaitingTest

[0x0000000076f45000] [level 3] platform-utils SetUp(): Setting up test: SemaphoreFunctionalityTest ../../src/utils/tst/Semaphore.cpp:165: Failure Expected equality of these values: 0 globalAtomicLoad(&threadCount) Which is: 65 Segmentation fault

entire log: kvspic_test_03-18_2.loghttps://github.com/awslabs/amazon-kinesis-video-streams-pic/files/4347101/kvspic_test_03-18_2.log

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/awslabs/amazon-kinesis-video-streams-pic/issues/25#issuecomment-600432534, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIA4AHPQYRRL3P34W2V2OG3RIBL6RANCNFSM4LOCWMRA.

[hsinghaowang]

change SEMAPHORE_TEST_THREAD_COUNT to 50

Failed at:

[0x0000000076f6b000] [level 3] platform-utils TearDown(): 
Tearing down test: ViewApiTest
[       OK ] ViewApiTest.contentViewGetWindowAllocationSize_NullPointer (0 ms)
[----------] 21 tests from ViewApiTest (40 ms total)

[----------] 64 tests from PermutatedStreamInfo/AcksFunctionalityTest
[ RUN      ] PermutatedStreamInfo/AcksFunctionalityTest.CheckRollbackFromErrorAckTime/0

[0x0000000076f6b000] [level 3] platform-utils SetUp(): 
Setting up test: PermutatedStreamInfo/AcksFunctionalityTest

Segmentation fault

entire log: kvspic_test_03-18_3.log

[MushMal]

Looking at the log it seems that it he last entry in the semaphore test freeWithoutReleaseAllThreadsTest LN 275 is failing. Seems that we have a timing issue in the test. Could you please add THREAD_SLEEP(300 * HUNDREDS_OF_NANOS_IN_A_MILLISECOND); in line 274 for the test to time the thread tear down?

Sent from my iPhone

On Mar 17, 2020, at 11:59 PM, hsinghaowang notifications@github.com wrote:



change SEMAPHORE_TEST_THREAD_COUNT to 50

Failed at:

[0x0000000076f6b000] [level 3] platform-utils TearDown(): Tearing down test: ViewApiTest [ OK ] ViewApiTest.contentViewGetWindowAllocationSize_NullPointer (0 ms) [----------] 21 tests from ViewApiTest (40 ms total)

[----------] 64 tests from PermutatedStreamInfo/AcksFunctionalityTest [ RUN ] PermutatedStreamInfo/AcksFunctionalityTest.CheckRollbackFromErrorAckTime/0

[0x0000000076f6b000] [level 3] platform-utils SetUp(): Setting up test: PermutatedStreamInfo/AcksFunctionalityTest

Segmentation fault

entire log: kvspic_test_03-18_3.loghttps://github.com/awslabs/amazon-kinesis-video-streams-pic/files/4347449/kvspic_test_03-18_3.log

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://github.com/awslabs/amazon-kinesis-video-streams-pic/issues/25#issuecomment-600456387, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIA4AHJS377NUKHYTPOUKMDRIBWKNANCNFSM4LOCWMRA.

[hsinghaowang]

Update:

268     // Wait for a little
269     THREAD_SLEEP(100 * HUNDREDS_OF_NANOS_IN_A_MILLISECOND);
270 
271     EXPECT_EQ(SEMAPHORE_TEST_THREAD_COUNT, (UINT32) ATOMIC_LOAD(&threadCount));
272 
273     EXPECT_EQ(STATUS_SUCCESS, semaphoreFree(&handle));
274 
275     THREAD_SLEEP(300 * HUNDREDS_OF_NANOS_IN_A_MILLISECOND);
276 
277     EXPECT_EQ(0, (UINT32) ATOMIC_LOAD(&threadCount));

Failed at:

[0x0000000076fa3000] [level 3] platform-utils SetUp(): 
Setting up test: ViewApiTest

[0x0000000076fa3000] [level 3] platform-utils TearDown(): 
Tearing down test: ViewApiTest
[       OK ] ViewApiTest.contentViewGetWindowAllocationSize_NullPointer (0 ms)
[----------] 21 tests from ViewApiTest (10 ms total)

[----------] 64 tests from PermutatedStreamInfo/AcksFunctionalityTest
[ RUN      ] PermutatedStreamInfo/AcksFunctionalityTest.CheckRollbackFromErrorAckTime/0

entire log: kvspic_test_03-18_4.log

[MushMal]

@hsinghaowang the log doesn’t have any information about what’s causing a crash and what kind of a crash this is. These are pretty heavy C++ Involved tests and I am not sure whether it’s a memory overcommit issue or the test fixture is causing a crash. Could you capture the crash dump or somehow get the crash back trace. You could run it under the debugger.

[hsinghaowang]

I changed to DEBUG build:

[----------] 2 tests from ThreadFunctionalityTest
[ RUN      ] ThreadFunctionalityTest.ThreadCreateAndReleaseSimpleCheck
[       OK ] ThreadFunctionalityTest.ThreadCreateAndReleaseSimpleCheck (42867 ms)
[ RUN      ] ThreadFunctionalityTest.ThreadCreateAndCancel
../../src/utils/tst/Thread.cpp:87: Failure
Expected equality of these values:
  ((UINT32) 0x00000000)
    Which is: 0
  globalCancelThread(threads[index])
    Which is: 22

(gdb) bt
#0  0x003b681c in testing::UnitTest::AddTestPartResult(testing::TestPartResult::Type, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
#1  0x003b6478 in testing::internal::AssertHelper::operator=(testing::Message const&) const ()
#2  0x00303990 in ThreadFunctionalityTest_ThreadCreateAndCancel_Test::TestBody (this=0x4e1950) at ../../src/utils/tst/Thread.cpp:87
#3  0x003d413c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#4  0x003be6d8 in testing::Test::Run() ()
#5  0x003bf15c in testing::TestInfo::Run() ()
#6  0x003bf630 in testing::TestCase::Run() ()
#7  0x003c8144 in testing::internal::UnitTestImpl::RunAllTests() ()
#8  0x003d4a5c in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#9  0x003c7e24 in testing::UnitTest::Run() ()
#10 0x001d48d0 in RUN_ALL_TESTS () at /home/leonwang/opensrc/src/../../opensrc/include/gtest/gtest.h:2341
#11 0x001d48b4 in main (argc=1, argv=0x7efffde4) at ../../src/duration/tst/DurationTest.cpp:138

[MushMal]

@hsinghaowang your last post has two issues

1) Test failure in Thread.cpp. LN87. The received error was STATUS_THREAD_DOES_NOT_EXIST. This is an indication that something must have failed when creating the thread. Have you modified the code? TEST_THREAD_COUNT is set to 500 and that can cause your system to fail. At times, the kernel OOM killer can kill random threads depending on the OS (Android is more prone to doing that). Another more likely possibility is that the test timeout of 2 seconds to create all threads might not be enough on your small footprint device so your threads have not yet been fully created by the time we reach LN87. Could you lower the value to say 10.

2) Seems to be Google test internal crash in DurationTest.cpp LN 138. Perhaps, you could run this particular test individually by running something like ./kvspic_test --gtest_filter="ValidInput.*"

[hsinghaowang]

My change:

leonwang@ubuntu1804$ git df
diff --git a/src/utils/tst/Semaphore.cpp b/src/utils/tst/Semaphore.cpp
index 6daa7f3..f4ab645 100644
--- a/src/utils/tst/Semaphore.cpp
+++ b/src/utils/tst/Semaphore.cpp
@@ -118,7 +118,7 @@ TEST_F(SemaphoreFunctionalityTest, releaseBasicTest)

 TEST_F(SemaphoreFunctionalityTest, acquireAwaitingTest)
 {
-    const UINT32 SEMAPHORE_TEST_THREAD_COUNT = 500;
+    const UINT32 SEMAPHORE_TEST_THREAD_COUNT = 10;
     const UINT32 SEMAPHORE_TEST_DRAIN_THREAD_COUNT = 10;
     UINT32 i;
     TID threadId;
@@ -272,5 +272,8 @@ TEST_F(SemaphoreFunctionalityTest, freeWithoutReleaseAllThreadsTest)

     EXPECT_EQ(STATUS_SUCCESS, semaphoreFree(&handle));

+       printf("wait\n");
+       THREAD_SLEEP(1000 * HUNDREDS_OF_NANOS_IN_A_MILLISECOND);
+
     EXPECT_EQ(0, (UINT32) ATOMIC_LOAD(&threadCount));
 }
diff --git a/src/utils/tst/Thread.cpp b/src/utils/tst/Thread.cpp
index e5b3e2a..01de52d 100644
--- a/src/utils/tst/Thread.cpp
+++ b/src/utils/tst/Thread.cpp
@@ -3,7 +3,7 @@
 class ThreadFunctionalityTest : public UtilTestBase {
 };

-#define TEST_THREAD_COUNT       500
+#define TEST_THREAD_COUNT       10

 UINT64 gThreadCurrentCount = 0;
 UINT64 gThreadSleepTimes[TEST_THREAD_COUNT];

[root@GM]# ./kvspic_test --gtest_filter="ValidInput.*"
Note: Google Test filter = ValidInput.*
[==========] Running 6 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 6 tests from ValidInput
[ RUN      ] ValidInput.CompleteInput
[       OK ] ValidInput.CompleteInput (1 ms)
[ RUN      ] ValidInput.InputWithZeros
[       OK ] ValidInput.InputWithZeros (0 ms)
[ RUN      ] ValidInput.InputWithoutTime
[       OK ] ValidInput.InputWithoutTime (0 ms)
[ RUN      ] ValidInput.TimeOnlyInput
[       OK ] ValidInput.TimeOnlyInput (0 ms)
[ RUN      ] ValidInput.LargeIntegerInput
[       OK ] ValidInput.LargeIntegerInput (1 ms)
[ RUN      ] ValidInput.PreciseFractionalInput
[       OK ] ValidInput.PreciseFractionalInput (0 ms)
[----------] 6 tests from ValidInput (8 ms total)

[----------] Global test environment tear-down
[==========] 6 tests from 1 test case ran. (16 ms total)
[  PASSED  ] 6 tests.

[root@GM]# AWS_KVS_LOG_LEVEL=7 bin/gdbserver :12345 ./kvspic_test --gtest_break_on_failure --gtest_filter="-imerQueueFunctionalityTest.*:HeapPerfTest.*"
.
.
.
[ RUN      ] ViewApiTest.contentViewGetAllocationSize_NullPointer
[       OK ] ViewApiTest.contentViewGetAllocationSize_NullPointer (0 ms)
[ RUN      ] ViewApiTest.contentViewAddItem_InvalidTime
[       OK ] ViewApiTest.contentViewAddItem_InvalidTime (1 ms)
[ RUN      ] ViewApiTest.contentViewGetWindowDuration_NullPointer
[       OK ] ViewApiTest.contentViewGetWindowDuration_NullPointer (0 ms)
[ RUN      ] ViewApiTest.contentViewGetWindowItemCount_NullPointer
[       OK ] ViewApiTest.contentViewGetWindowItemCount_NullPointer (0 ms)
[ RUN      ] ViewApiTest.contentViewGetWindowAllocationSize_NullPointer
[       OK ] ViewApiTest.contentViewGetWindowAllocationSize_NullPointer (0 ms)
[----------] 21 tests from ViewApiTest (20 ms total)

[----------] 64 tests from PermutatedStreamInfo/AcksFunctionalityTest
[ RUN      ] PermutatedStreamInfo/AcksFunctionalityTest.CheckRollbackFromErrorAckTime/0

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 4734]
0x000c70e4 in instrumentedClientMemAlloc (size=104857600) at ../../src/client/tst/ClientTestFixture.h:117
117     *(PSIZE_T)pAlloc = size;
(gdb) bt
#0  0x000c70e4 in instrumentedClientMemAlloc (size=104857600) at ../../src/client/tst/ClientTestFixture.h:117
#1  0x00390e38 in aivHeapInit (pHeap=0x52e5fc, heapLimit=104857600) at ../../src/heap/src/AivHeap.c:185
#2  0x00394ac4 in heapInitialize (heapLimit=104857600, spillRatio=0, behaviorFlags=1, ppHeap=0x57d62c) at ../../src/heap/src/Heap.c:83
#3  0x00376b04 in createKinesisVideoClient (pDeviceInfo=0x574f00, pClientCallbacks=0x576020, pClientHandle=0x574ee8) at ../../src/client/src/Client.c:110
#4  0x000bf504 in ClientTestBase::CreateScenarioTestClient (this=0x574ed8) at ../../src/client/tst/ClientTestFixture.h:722
#5  0x000be474 in AcksFunctionalityTest_CheckRollbackFromErrorAckTime_Test::TestBody (this=0x574ed8) at ../../src/client/tst/AcksFunctionalityTest.cpp:28
#6  0x003d40e8 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#7  0x003be684 in testing::Test::Run() ()
#8  0x003bf108 in testing::TestInfo::Run() ()
#9  0x003bf5dc in testing::TestCase::Run() ()
#10 0x003c80f0 in testing::internal::UnitTestImpl::RunAllTests() ()
#11 0x003d4a08 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#12 0x003c7dd0 in testing::UnitTest::Run() ()
#13 0x001d48d0 in RUN_ALL_TESTS () at /home/leonwang/opensrc/src/../../opensrc/include/gtest/gtest.h:2341
#14 0x001d48b4 in main (argc=1, argv=0x7efffdd4) at ../../src/duration/tst/DurationTest.cpp:138
(gdb)

[MushMal]

OK, sorry, I was thrown off with DurationTest.cpp but it's just the linkage that links the main entry routine.

The main issue is that we are attempting to allocate 104857600 bytes in the common tests. The test fixture is adding it's own allocators to wrap around the systems and do some accounting so every test does verify the heap and potential memory leaks. This is the reason why we fail in the fixture as the fixture overrides the allocator. What's happening, is that the malloc fails to allocate 104857600 bytes (for the content store in this case) and returns NULL which gets dereferenced in the tests.

You can try to modify ClientTestFixture.h LN 701 and specify a smaller storage size but that might break some of the tests.

A more general question I would like to ask is why are you trying to run the unit tests on this device as it doesn't have enough capacity?

Could you please compile the C Producer video only sample and run it on your device - that might give you more information.

Can you describe more your scenario as it seems that we are going back and forth tweaking the unit tests for your device whereas the intent of the unit tests is to be run on a larger footprint device.

[hsinghaowang]

We want to make sure PIC with ALIGNED_MEMORY_MODEL works correctly. The final goal is to support Kinesis Video Streams in our platform.

I will try C Producer video only sample.

PS. After modify ClientTestFixture.h LN 701, the test failed at:

[ RUN      ] PermutatedStreamInfo/StateTransitionFunctionalityTest.FaultInjectUploadHandleAfterStopBeforeTokenRotation/14
[       OK ] PermutatedStreamInfo/StateTransitionFunctionalityTest.FaultInjectUploadHandleAfterStopBeforeTokenRotation/14 (2552 ms)
[ RUN      ] PermutatedStreamInfo/StateTransitionFunctionalityTest.FaultInjectUploadHandleAfterStopBeforeTokenRotation/15
[       OK ] PermutatedStreamInfo/StateTransitionFunctionalityTest.FaultInjectUploadHandleAfterStopBeforeTokenRotation/15 (2428 ms)
[ RUN      ] PermutatedStreamInfo/StateTransitionFunctionalityTest.FaultInjectUploadHandleAfterStopDuringTokenRotation/0
../../src/client/tst/StateTransitionFunctionalityTest.cpp:400: Failure
Expected equality of these values:
  ((UINT32) 0x00000000)
    Which is: 0
  mockProducer.timedPutFrame(currentTime, &didPutFrame)
    Which is: 1375731758

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 23690]
0x003b67c8 in testing::UnitTest::AddTestPartResult(testing::TestPartResult::Type, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
(gdb) bt
#0  0x003b67c8 in testing::UnitTest::AddTestPartResult(testing::TestPartResult::Type, char const*, int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) ()
#1  0x003b6424 in testing::internal::AssertHelper::operator=(testing::Message const&) const ()
#2  0x001301ac in StateTransitionFunctionalityTest_FaultInjectUploadHandleAfterStopDuringTokenRotation_Test::TestBody (this=0x562de8)
    at ../../src/client/tst/StateTransitionFunctionalityTest.cpp:400
#3  0x003d40e8 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#4  0x003be684 in testing::Test::Run() ()
#5  0x003bf108 in testing::TestInfo::Run() ()
#6  0x003bf5dc in testing::TestCase::Run() ()
#7  0x003c80f0 in testing::internal::UnitTestImpl::RunAllTests() ()
#8  0x003d4a08 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#9  0x003c7dd0 in testing::UnitTest::Run() ()
#10 0x001d48d0 in RUN_ALL_TESTS () at /home/leonwang/opensrc/src/../../opensrc/include/gtest/gtest.h:2341
#11 0x001d48b4 in main (argc=1, argv=0x7efffde4) at ../../src/duration/tst/DurationTest.cpp:138
(gdb) 

[MushMal]

This is STATUS_STORE_OUT_OF_MEMORY as defined in https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/producer-sdk-errors.html#producer-sdk-errors-client

As I mentioned the tests have been designed for larger footprint devices.

You can troubleshoot by checking the value of the status codes. However, the true test would be the running the sample on your platform

[MushMal]

One more note. A lot of the tests would EXPECT_EQ(STATUS_SUCCESS, operation) when setting up the tests. This means that if you are running without --gtest_break_on_failure, the test would continue even when it fails to set up assets for the tests and would inevitably cause issues like access violation. Perhaps, in the future, we could replace EXPEC_EQ with ASSERT_EQ for some of the tests but most tests share fixtures that do the setup and some tests are expecting failures.

[hsinghaowang]

The C Producer video only sample works in our platform with ALIGNED_MEMORY_MODEL enabled. I will close this. Thanks for your help.