Figure out possible problem with sqlite3mc and the heap_limit functions

[ericsink]

During #563 there was a problem with the test cases involving heap_limit. These tests have been temporarily disabled, but need to be brought back.

[ericsink]

cc/ @utelle

[ericsink]

FWIW, I get the same error when running the test suite in my local dev environment, on a Mac.

[ericsink]

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:01.12]     SQLitePCL.Tests.test_cases.test_sqlite3_soft_heap_limit64 [FAIL]
  Failed SQLitePCL.Tests.test_cases.test_sqlite3_soft_heap_limit64 [11 ms]
  Error Message:
   Assert.Equal() Failure
Expected: 20971520
Actual:   10485760

[utelle]

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
[xUnit.net 00:00:01.12]     SQLitePCL.Tests.test_cases.test_sqlite3_soft_heap_limit64 [FAIL]
  Failed SQLitePCL.Tests.test_cases.test_sqlite3_soft_heap_limit64 [11 ms]
  Error Message:
   Assert.Equal() Failure
Expected: 20971520
Actual:   10485760

I inspected SQLite's code and the test code.

SQLite has a data structure which holds the soft and the hard heap limit. Both values are initialized to 0, unless the symbol SQLITE_MAX_MEMORY is defined with a different value at compile time. As far as I can tell, thecb` repo doesn't set this symbol.

That is, SQLite starts with soft and hard heap limit equal 0.

However, obviously the first call to sqlite3_soft_heap_limit64() returns the value 10485760. Otherwise it could not request a new limit of 20971520. How can the soft heap limit be already 10485760, although sqlite3_soft_heap_limit64() was not called anywhere else before the soft heap limit test started (or did I miss a call tosqlite3_soft_heap_limit64())?

So, if sqlite3_soft_heap_limit64() was not called elsewhere, something else must have happened, before sqlite3_soft_heap_limit64() was called for the first time. What could that be? (I will give the answer in a minute.)

Another question is, under which circumstances can it happen, that sqlite3_soft_heap_limit64() does not set the requested soft heap limit. Inspecting the code of sqlite3_soft_heap_limit64() reveals that this can only happen, if the hard heap limit has a value > 0 and is smaller than the requested soft heap limit. In that case the soft heap limit is set to the same value as the hard heap limit.

Now, how can the hard heap limit have the value of 10485760? This has to be the case. Otherwise you would not get this value back on querying for the actual soft heap limit value on the first call.

The answer can only be that the hard heap limit test (which is also included in the test suite) ran before the soft heap limit test. If this is the case, the result of the soft heap limit test can be explained easily:

When a new soft heap limit of 20971520 is requested, the value is checked against the hard heap limit value (which is 10485760). Because this value is smaller than the new requested soft heap limit, the soft heap limit is set to the value of the hard heap limit. And that's why the second query of the soft heap limit returns 10485760 instead of the expected value 20971520.

My explanation is that the hard heap limit test runs before the soft heap limit test, causing the observed results. What I can't explain is, why this happens only for sqlite3mc and not sqlite3 or sqlcipher.

However, I'm quite confident that it has nothing to do with sqlite3mc, but with the behaviour of the test environment.

[ericsink]

Nice detective work.

I tried changing both tests to reset their limits back to 0 after doing their checks. That seems to fix the problem.

[utelle]

Please do the following:

Add this call at the beginning of the test test_sqlite3_soft_heap_limit64() before calling raw.sqlite3_soft_heap_limit64(-1):

var previousHardLimit = raw.sqlite3_hard_heap_limit64(0);

This way you should get a clean initial state, and the test should no longer fail.

If possible show the value of previousHardLimit in the output.

[utelle]

I saw your post only after I submitted my previous one.

Nice detective work.

My specialty. 😄

I tried changing both tests to reset their limits back to 0 after doing their checks. That seems to fix the problem.

I'm not surprised. 😆