Encounter problem with memory copy for record delete.

[mightylastingcode]

I ported unqlite to ARM M4 chip. I am able to use Jx script to add new records and fetch new records by ID. But, I can't drop a record. The firmware will crash for sure. It crashes when it tries to copy some data with this below macro in the syBlobAppend function.

Here is the header of the macro.

define SX_MACRO_FAST_MEMCPY(SRC, DST, SIZ) {...}

UnQlite works well on Ubuntu Linux. So, I am not sure what I may have done wrong in porting it to an ARM chip. I set the page size to 512 bytes instead of 4096 bytes.

I printed the pointers' address and data size for where the firmware crashed.

pData: 3140 ( 3140, 0x c44) //Internal RAM zBlob: -1877127508 ( 2417839788, 0x901d4eac) // External SDRAM nSize: -4 ( 4294967292, 0xfffffffc) ??? This is not good!

This is the only clue I have. I suspect some sort of calculation errors. But, it is not always -4 when it crashes.

Thank you for any help you can provide, Michael

[symisc]

This has to do with byte alignment I suppose. You should remove the old macro and define this one:

#define SX_MACRO_FAST_MEMCPY(SRC, DST, SIZ) memcpy(DST,SRC,SIZ)

On the next release which is scheduled for January 2020, all byte swapping, copying functions shall rely on the C standard library instead of implementing their own.

[mightylastingcode]

Unfortunately memcpy does not fix the problem.

I checked with the chip vendor and shared your thought with them. Here is what they suggested.

"Cortex M4 supports unaligned access however some instructions require aligned data (such as load/store multiple). Is there a way to force 32-bit alignment in Unqlite?"

Is it possible?

I also notice that besides the negative nSize value, the SRC address is not in the RAM area. For 30% of the time, pSrc->pBlob pointer value is not right. Where is this pointer get the value? Currently I am studying the code for how this pointer is assigned. Thank.

Thank you, Michael

[mightylastingcode]

I inserted a print statement to display all the pointer values from malloc and realloc. The addresses are all good. These two functions from the standard C lib are good.

The strange pBlob values are likely caused by something else.

[mightylastingcode]

First, sorry, the SRC address is ROM address. I am wrong. This low SRC address is not the problem.

For example: SyblobAppend(buf,"]",n); // "]" is stored in ROM. ROM is placed at the bottom. On PC, everything is stored in the same RAM space, but not in the embedded MCU.

Currently I think that the exception is probably caused by buffer overflow. It happens even for small normal n values (i.e. 28). I am not sure the reason.

[symisc]

The SyBlob is just a container for raw bytes string (i.e without the nul terminator) which grow dynamically via malloc/realloc (in fact it uses a memory pool for faster memory allocation). It hold a size/count counter that is doubled when the current allocated memory chunk size limit is reached.

If you want to bypass the SyBlob, you can rely on unqlite_kv_fetch_callback() which call the supplied callback instead of storing the data in a private SyBlob.

[mightylastingcode]

Thank you so much for taking time to write back to me. The problem seems to be processor dependent because UnQlite runs without this problem on x86 processor. The ARM M4+ CPU will hang whenever I delete a record. The ARM CPU hanged only twice when I insert a new record. For adding records, it is just not often for me to encounter this problem, but it does prevent me from creating a large database.

Is SyBlob storing the database records? What happens to the SyBlob container when you delete a record? Does the size of the container ever shrink when you delete data? When the size is doubled, is the container guaranteed to have enough memory allocated to take in new data. How can I check that? Because sometimes I get -4 for nSize, I am concerned that there may be memory size calculation issue.

I still want to keep using your code because I am concerned if this problem is related to ARM core or ARM GNU compiler. It is better to know what caused the issue if possible.

Best regards, Michael

[symisc]

@mikelisfbay, Could you compile the library without any optimization flags like -O2, -Ofast, etc. enabled and check if the issue is reproducible. Sometimes, aggressive optimizations can induct this kind of issue.

[mightylastingcode]

Thank you for your idea. I have tried it with -O0. I get the same issue. I also try :MEM: and get the same issue.

I am planning to try a clean copy next week. Because I put many printf statements in the code to narrow down the source of the issue, maybe I should start a clean version. The good thing about record deletion is that I need to track only a few subroutines. If I understand the container, maybe I can pin point the source of the issue.

I am little surprised why this issue because most of other features (jx insert, fetch,..) are working pretty good. I believe that it will work well on ARM CPU as it does on x86 CPU.

[symisc]

Is the issue only triggered from the delete operation. If you just do only insertions and record fetching, the problem disappear, right? If so, could you share your C code snippet that trigger the issue.

[mightylastingcode]

Thank you so much.

It is very rare to have a hang up problem with insertions. But, it has happened twice. The deletions will always trigger the hang up problem . It happens inside SX_MACRO_FAST_MEMCPY function within syBlobAppend function. I changed the macro as you suggested. It does not help. I suspect that it might copy too much data beyond the limit size of the container.

As my log shows, not every memory copy triggers the hang up problem. It takes a few calls. ============================Begin (SyBlobAppend)========================

nSize: 4 ( 4, 0x 4) snSize: 4 ( 4, 0x 4)

---

SRC: 256672 ( 256672, 0x 3eaa0) DST: 536911844 ( 536911844, 0x20009fe4) SIZ: 4 ( 4, 0x 4)

---

============================End (SyBlobAppend) OK========================

---

SRC: 285784 ( 285784, 0x 45c58) DST: 536898012 ( 536898012, 0x200069dc) SIZ: 40 ( 40, 0x 28)

---

---

SRC: 537194268 ( 537194268, 0x2004ef1c) DST: 536898012 ( 536898012, 0x200069dc) SIZ: 12 ( 12, 0x c)

---

============================Begin (SyBlobAppend)========================

nSize: 2260 ( 2260, 0x 8d4) snSize: 2260 ( 2260, 0x 8d4)

---

SRC: 1818584175 ( 1818584175, 0x6c65646f) DST: 537055676 ( 537055676, 0x2002d1bc) SIZ: 2260 ( 2260, 0x 8d4)

---

JX9_PRIVATE sxi32 SyBlobAppend(SyBlob pBlob, const void pData, sxu32 nSize) { sxu8 zBlob; sxi32 rc; int snSize = nSize; puts("============================Begin (SyBlobAppend)========================\n"); printf("nSize: %15d \t(%15u, 0x%8x)\n",nSize, nSize, nSize); printf("snSize: %15d \t(%15u, 0x%8x)\n",snSize, snSize, snSize); //if( nSize < 1 ){ if( snSize < 1 ){ printf("snSize = %d < 1\n",snSize); puts("============================End (SyBlobAppend) OK========================\n"); return SXRET_OK; } rc = BlobPrepareGrow(&(pBlob), &nSize); if( SXRET_OK != rc ){ printf("============================End (SyBlobAppend) bad rc = %d ========================\n", rc); return rc; } if( pData ){ printf("pData: %15d \t(%15u, 0x%8x)\n",pData, pData, pData); zBlob = (sxu8 *)pBlob->pBlob ; printf("zBlob: %15d \t(%15u, 0x%8x)\n",zBlob, zBlob, zBlob); zBlob = &zBlob[pBlob->nByte]; pBlob->nByte += nSize;

printf("zBlob: %15d \t(%15u, 0x%8x)\n",zBlob, zBlob, zBlob);
SX_MACRO_FAST_MEMCPY(pData, zBlob, nSize);  // System hangs inside here.

} puts("============================End (SyBlobAppend) OK========================\n"); return SXRET_OK; }

[mightylastingcode]

The IDE debugger shows the stack. I think that this may be helpful to you.

[mightylastingcode]

I created myMemCpy() as an experiment to copy the data byte by byte because memcpy() cannot solve the problem. But, my custom function does not help solve the problem either.

[symisc]

Is your code multi-threaded? If multiple threads are sharing the same unqlite handle, then you have to enable multi-threading to prevent memory corruption problems like this. Check https://github.com/symisc/unqlite/issues/91#issuecomment-540608966 for additional information.

[mightylastingcode]

No, only one thread is handling the unqlite db handle currently. In the future, it may or may not be true.

I went ahead to make it multthread safe as suggested. It does not fix the problem. I ran it twice. Like before, it always crashes. However, the line of code that causes the hang up problem is different now. It is in jxMemObjStore().

Best, Michael

[symisc]

The most plausible scenario here is that you are running out of memory and the memory allocation routine returned NULL somewhere to the Jx9 layer. How much RAM is allocated to the process running this program?

[mightylastingcode]

I have 32M bytes of external RAM on my board and make all of these external RAM space available for Unqlite. I will try to print an error message if NULL is ever returned from malloc/realloc() calls.

Thank you for sharing your idea.

[mightylastingcode]

I checked that I did not run out of memory. I think it allocates below 1M bytes. I only have 4 records in my example.

So, I want to check with you on how I setup unqlite for ARM M4.

1. unqlite.h //#define UNIXES // I commented out this line. Any side effect from doing it. 2 unqlite.c //#define UNQLITE_DEFAULT_PAGE_SIZE 4096 / 4K /

   define UNQLITE_DEFAULT_PAGE_SIZE 512 / 512 /

2. unqlite.c (copy everything from unqlite_demovfs.c)
3. network_thread_entry.c
   unqlite_start(); // run this function to set up unqlite configuration.

Do you see any problem above? Did I port your code correctly? Thank you.. Michael

[symisc]

Does the same problem popup when you use the key/value storage functions only (i.e. unqlite_kv_store(), unqlite_kv_delete(), etc.) without using the document storage layer (i.e. Jx9)?

[mightylastingcode]

No. These kv functions work fine. The problem with the memory hang occurs only when I use Jx9 function calls. I did not do much testing with the kv functions because I really like the Jx9 and spend time to get it to work. As I recall, I did not experience a problem with the kv functions.

[mightylastingcode]

There is no issue with unqlite_kv_delete(). I deleted three different records successfully. The database files were updated correctly.

[mightylastingcode]

I am trying to use unqlite_kv functions. I notice that I cannot store the record in the order of the key string. When I read the record, they seem to be stored in random order.

Index : 349 Key length ==> 3 Key string ==> 475 Data string ==> axzbettvfrfptlyxtndzeugfbthduvdlxtnelxireessoshdcnpogjsayjw

Index : 350 Key length ==> 3 Key string ==> 556 Data string ==> kebxufxalfwitewddqouxpgupfqcjrudvanoydmlnegozmtovnqjccdyyeh

Index : 351 Key length ==> 3 Key string ==> 637 Data string ==> ftxiyankxqrlkmjqhunbeqnmuetmhiziooouygkrkazhkizkkzolhnphboo

Can I store them in the order according to the key string?

index 0: key = 0 index 1: key = 1 index 2: key = 2

index 500: key = 500

I like Jx9 functions. While I am trying to find the root cause, I want to try to use the kv functions to build a simple database.

Thank you, Michael

[symisc]

That's a normal behavior because the default storage engine is the hash based one which does not honor order of key insertions but provide O(1) disk lookup regardless of the number of items inserted (i.e. 1 or 1 billion).

For Jx9, as said earlier, the most plausible scenario here is the out-of-memory error. This is due to the fact that Jx9 require at least 64MB ~128MB of RAM to perform its task (i.e. lex, parse, compile, bytecode execution, database call, foreign function calls, etc...). But again, this is an assumption. You can download and run the Jx9 standalone interpreter from https://jx9.symisc.net for further analysis.

[mightylastingcode]

I see. Thank you for your explanation on how the hash work.

I guess I may have missed some malloc/realloc() calls in the code. I thought I had the ram usage under control. The on board 32MB DRAM is not enough if that is the RAM requirement.

Today I used the kv store function to build a 4000+ records table before the system crashed. I think I probably ran out of RAM again. Consider how limited RAM space I have. UnQlite allows me to build a good size database. I am also glad that I am able to perform most of Jx9 functions. I consider that I am very lucky or UnQlite is pretty low footprint database software.

Thank you for taking the time to help me. I will try to upgrade the external RAM chip on my development board.

Best, Michael

[symisc]

You can insert way more than 4K records using the KV functions. Just call unqlite_commit() say each 1K insertions to free up some memory or even unqlite_close() and open the DB again in case your memory is scare.

[mightylastingcode]

That is great. I will try it.

By the way, you said that Jx9 consumes about 64MB of memory space. Does all the memory allocated through malloc/realloc()? Do global variable take much RAM (probably not)? I have two RAM's: heap (external 32MRAM for malloc/realloc), on chip RAM (640KB for global/local variables). I am building a memory pool for Unqlite so that I can reserve enough RAM for other peripherals like LCD display.

I want to make sure that I replace every malloc/realloc() with my functions. Because I did not detect a NULL return for the out of memory condition (which I expect to encounter), I am concerned if I have missed something.

line 26900 (Here are the only functions that allocates/reallocates RAM for UnQlite) Is it correct? static void SyOSHeapAlloc(sxu32 nByte) { void pNew;

if defined(WINNT)

pNew = HeapAlloc(GetProcessHeap(), 0, nByte);

else

pNew = malloc((size_t)nByte);

endif

return pNew;

} static void SyOSHeapRealloc(void pOld, sxu32 nByte) { void *pNew;

if defined(WINNT)

pNew = HeapReAlloc(GetProcessHeap(), 0, pOld, nByte);

else

pNew = realloc(pOld, (size_t)nByte);

endif

return pNew;    

}

Thank you again. Michael

[symisc]

You can register your own memory allocation routines plus a virtual file system (VFS) of your own without hacking the internal of the engine. Refer to the documentation at https://unqlite.org/c_api_object.html#SyMemMethods for additional information.

[mightylastingcode]

Thank you. Very thoughtful!