Compile Errors

[manjus3]

Hi dryman,

Thanks for making this library available to the public. I'm eager to try but am running into build issues. I simply cloned the repo, ran bootstrap.sh, followed by ./configure, and make. Please take a look below and let me know if I'm missing anything. I do have log4c and cmocka installed. First, here's the output from the uname command: Linux manju-VirtualBox 4.13.0-38-generic #43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

And now the build errors, showing the last few lines: libtool: link: gcc -shared -fPIC -DPIC common/.libs/op_log.o malloc/.libs/op_malloc.o malloc/.libs/allocator.o malloc/.libs/deallocator.o malloc/.libs/init_helper.o malloc/.libs/lookup_helper.o hash/.libs/cityhash.o hash/.libs/op_hash_table.o hash/.libs/pascal_hash_table.o -g -O2 -Wl,-soname -Wl,libopic.so.0 -o .libs/libopic.so.0.0.0 libtool: link: (cd ".libs" && rm -f "libopic.so.0" && ln -s "libopic.so.0.0.0" "libopic.so.0") libtool: link: (cd ".libs" && rm -f "libopic.so" && ln -s "libopic.so.0.0.0" "libopic.so") libtool: link: ar cru .libs/libopic.a common/op_log.o malloc/op_malloc.o malloc/allocator.o malloc/deallocator.o malloc/init_helper.o malloc/lookup_helper.o hash/cityhash.o hash/op_hash_table.o hash/pascal_hash_table.o ar: u' modifier ignored sinceD' is the default (see `U') libtool: link: ranlib .libs/libopic.a libtool: link: ( cd ".libs" && rm -f "libopic.la" && ln -s "../libopic.la" "libopic.la" ) make[3]: Leaving directory '/home/manju/opic/opic' make[2]: Leaving directory '/home/manju/opic/opic' Making all in benchmark make[2]: Entering directory '/home/manju/opic/benchmark' Making all in malloc make[3]: Entering directory '/home/manju/opic/benchmark/malloc' gcc -DHAVE_CONFIG_H -I. -I../.. -g -O2 -Wall -MT malloc_bench.o -MD -MP -MF .deps/malloc_bench.Tpo -c -o malloc_bench.o malloc_bench.c mv -f .deps/malloc_bench.Tpo .deps/malloc_bench.Po /bin/bash ../../libtool --tag=CC --mode=link gcc -g -O2 -Wall -static -o malloc_bench malloc_bench.o ../../opic/libopic.la -latomic -L/usr/local/lib -llog4c libtool: link: gcc -g -O2 -Wall -o malloc_bench malloc_bench.o ../../opic/.libs/libopic.a -latomic -L/usr/local/lib /usr/local/lib/liblog4c.so /usr/lib/x86_64-linux-gnu/libexpat.so -lpthread make[3]: Leaving directory '/home/manju/opic/benchmark/malloc' Making all in robin_hood make[3]: Entering directory '/home/manju/opic/benchmark/robin_hood' Making all in spookyhash-c make[4]: Entering directory '/home/manju/opic/benchmark/robin_hood/spookyhash-c' make[4]: No rule to make target 'all'. Stop. make[4]: Leaving directory '/home/manju/opic/benchmark/robin_hood/spookyhash-c' Makefile:711: recipe for target 'all-recursive' failed make[3]: [all-recursive] Error 1 make[3]: Leaving directory '/home/manju/opic/benchmark/robin_hood' Makefile:363: recipe for target 'all-recursive' failed make[2]: [all-recursive] Error 1 make[2]: Leaving directory '/home/manju/opic/benchmark' Makefile:484: recipe for target 'all-recursive' failed make[1]: [all-recursive] Error 1 make[1]: Leaving directory '/home/manju/opic' Makefile:392: recipe for target 'all' failed make: *** [all] Error 2

Thanks, Manju

[dryman]

Did you try: git submodule update --init --recursive? The error was due to a dependency spookyhash-c didn't get compiled. Let me know how it goes.

[manjus3]

Hi Felix,

Thanks for your response. I tried the update command just now, and I get the same errors.

-Manju

On Thu, May 3, 2018 at 9:11 AM, Felix Chern notifications@github.com wrote:

Did you try: git submodule update --init --recursive? The error was due to a dependency spookyhash-c didn't get compiled. Let me know how it goes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dryman/opic/issues/78#issuecomment-386349875, or mute the thread https://github.com/notifications/unsubscribe-auth/AlF24HVCuFY8Jf_GbZUMxHk24J02FD4Vks5tuyw1gaJpZM4TuazU .

[manjus3]

Hi Felix,

I decided to get a fresh copy of opic, then ran the above update command, and now I get a clean build. One note about compiling the examples listed on your introduction page: In the .c files, you provide the following compilation instruction: // gcc -lopic $(log4c-config --libs) opic_write.c -o opic_write

But the above line gives me undefined references. Rearranging it like shown below fixes the problem: gcc opic_write.c -o opic_write -lopic $(log4c-config --libs)

I have not read a whole lot on OPIC's implementation, but I had a question about the user data that is being persisted. Does OPIC handle user data/structs, which have pointers to other data?

Thanks, Manju

On Thu, May 3, 2018 at 9:39 AM, Manju Shekhar manju.shekhar@gmail.com wrote:

Hi Felix,

Thanks for your response. I tried the update command just now, and I get the same errors.

-Manju

On Thu, May 3, 2018 at 9:11 AM, Felix Chern notifications@github.com wrote:

Did you try: git submodule update --init --recursive? The error was due to a dependency spookyhash-c didn't get compiled. Let me know how it goes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dryman/opic/issues/78#issuecomment-386349875, or mute the thread https://github.com/notifications/unsubscribe-auth/AlF24HVCuFY8Jf_GbZUMxHk24J02FD4Vks5tuyw1gaJpZM4TuazU .

[manjus3]

Hi Felix,

Have you run valgrind on the example code you have provided (opic_write.c)? I just did, and valgrind is giving some errors. Additionally, valgrind itself crashed when I ran it on opic_write. Are the errors something to worry about?

Please see below:

==2880== 160 bytes in 1 blocks are still reachable in loss record 46 of 46 ==2880== at 0x4C31A1E: calloc (vg_replace_malloc.c:711) ==2880== by 0x5070C20: sd_calloc (malloc.c:109) ==2880== by 0x5071008: sd_hash_new (hash.c:74) ==2880== by 0x5070CF1: sd_factory_new (factory.c:50) ==2880== by 0x506E982: log4c_category_get (category.c:59) ==2880== by 0x4E3DED6: init_logger (op_malloc.c:69) ==2880== by 0x4010AD9: call_init.part.0 (dl-init.c:72) ==2880== by 0x4010BEA: call_init (dl-init.c:30) ==2880== by 0x4010BEA: _dl_init (dl-init.c:120) ==2880== by 0x4000ED9: ??? (in /lib/x86_64-linux-gnu/ld-2.26.so) ==2880== ==2880== LEAK SUMMARY: ==2880== definitely lost: 0 bytes in 0 blocks ==2880== indirectly lost: 0 bytes in 0 blocks ==2880== possibly lost: 0 bytes in 0 blocks ==2880== still reachable: 2,431 bytes in 49 blocks ==2880== suppressed: 0 bytes in 0 blocks ==2880== ==2880== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0) ==2880== ==2880== 1 errors in context 1 of 1: ==2880== Invalid read of size 2 ==2880== at 0x4E3F093: atomic_check_in_16 (op_atomic.h:128) ==2880== by 0x4E3F093: USpanObtainAddr (allocator.c:300) ==2880== by 0x4E3FC08: DispatchUSpanForAddr (allocator.c:193) ==2880== by 0x4E40367: OPMallocAdviced (allocator.c:130) ==2880== by 0x4E405DC: OPCallocAdviced (allocator.c:167) ==2880== by 0x4E4D5AB: PHNew (pascal_hash_table.c:121) ==2880== by 0x1089F2: main (in /home/manju/opic/examples/opic_write) ==2880== Address 0x1000003008 is not stack'd, malloc'd or (recently) free'd ==2880== ==2880== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

On Thu, May 3, 2018 at 10:28 AM, Manju Shekhar manju.shekhar@gmail.com wrote:

Hi Felix,

I decided to get a fresh copy of opic, then ran the above update command, and now I get a clean build. One note about compiling the examples listed on your introduction page: In the .c files, you provide the following compilation instruction: // gcc -lopic $(log4c-config --libs) opic_write.c -o opic_write

But the above line gives me undefined references. Rearranging it like shown below fixes the problem: gcc opic_write.c -o opic_write -lopic $(log4c-config --libs)

I have not read a whole lot on OPIC's implementation, but I had a question about the user data that is being persisted. Does OPIC handle user data/structs, which have pointers to other data?

Thanks, Manju

On Thu, May 3, 2018 at 9:39 AM, Manju Shekhar manju.shekhar@gmail.com wrote:

Hi Felix,

Thanks for your response. I tried the update command just now, and I get the same errors.

-Manju

On Thu, May 3, 2018 at 9:11 AM, Felix Chern notifications@github.com wrote:

Did you try: git submodule update --init --recursive? The error was due to a dependency spookyhash-c didn't get compiled. Let me know how it goes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dryman/opic/issues/78#issuecomment-386349875, or mute the thread https://github.com/notifications/unsubscribe-auth/AlF24HVCuFY8Jf_GbZUMxHk24J02FD4Vks5tuyw1gaJpZM4TuazU .

[dryman]

Sorry for slow reply. I'll test it with valgrind and see what went wrong.

[dryman]

Sorry for slow reply

Regarding your questions:

1. Unfortunately, for user structs/data that has pointers to other data, it is not supported. You can define data structures using the pointer opic provides. However, any data that uses the pointer that opic doesn't recognize, it will segfault when reading it back from disk.

2. It is actually natural for valgrind to fail. The program writes the pointer to disk and does not free the address, which is intended. The idea of opic is to continue the use of pointer in next process. Therefore the pointer cannot be freed when program ends. Having non-freed pointer, or uninitialized pointer when reading it back from disk is a natural thing.

I wrote a blog post for how it works. http://www.idryman.org/blog/2017/06/28/opic-a-memory-allocator-for-fast-serialization/

[manjus3]

Hi Felix,

Thanks for your response. That sounds good. I'll take a look at your blog post. I actually did write a test program to integrate my existing structures with pointers and serialize them. What I did was to first write a couple of function to serialize/deserialize them in the traditional sense, i.e., serialize the structure members into a char* buffer, and then convert this buffer to an OPIC pointer type, and then call OPIC's serialize function. It worked out fine. I could read it back from disk into an OPIC pointer type, and call my deserialize function to reconstruct my structure.

-Manju

On Sun, May 20, 2018 at 1:54 AM, Felix Chern notifications@github.com wrote:

Sorry for slow reply

Regarding your questions:

1.

Unfortunately, for user structs/data that has pointers to other data, it is not supported. You can define data structures using the pointer opic provides. However, any data that uses the pointer that opic doesn't recognize, it will segfault when reading it back from disk. 2.

It is actually natural for valgrind to fail. The program writes the pointer to disk and does not free the address, which is intended. The idea of opic is to continue the use of pointer in next process. Therefore the pointer cannot be freed when program ends. Having non-freed pointer, or uninitialized pointer when reading it back from disk is a natural thing.

I wrote a blog post for how it works. http://www.idryman.org/blog/2017/06/28/opic-a-memory-allocator-for-fast- serialization/

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/dryman/opic/issues/78#issuecomment-390467055, or mute the thread https://github.com/notifications/unsubscribe-auth/AlF24M72kHMsaF1OJez5wJY6cVgBszlgks5t0S9EgaJpZM4TuazU .