fbnzk / leveldb

Automatically exported from code.google.com/p/leveldb
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

db_bench and crc32c_test does not work on powerpc (big-endian) #85

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
it seems that CRC32c calculation fails on MCP83xx embedded processor!
when i call "crc32c_test" the third test fails with different output than on 
i386 workstation. I added some printfs to visualize (see attached patch). 

I compile with a toolchain from openwrt with following options
"powerpc-openwrt-linux-gnu-g++ -mcpu=e300c3 -mbig-endian"

./crc32c_test                                                                   
==== Test CRC.StandardResults                                                   
          calc CRC over data @ 0x0xbfab6358 len 32 result 8a9136aa              
                                                          calc CRC over data @ 
0x0xbfab6358 len 32 result 62a8ab43                                             
                           calc CRC over data @ 0x0xbfab6358 len 32 result 
67022fc1                                                                        
util/crc32c_test.cc:26: failed: 1188919630 == 1728196545

the same code compiled on openSuse i386 Worktation works fine without any 
failure.

the same Problem with db_bench command which fails after some steps.

./db_bench                                                                      
LevelDB:    version 1.4                                                         
                    Date:       Tue Apr 24 15:22:40 2012                        
                                                    CPU:        0 *             
                                                                  CPUCache:     
                                                                  Keys:       
16 bytes each                                                                   
         Values:     100 bytes each (50 bytes after compression)                
                                                    Entries:    1000000         
                                                                RawSize:    
110.6 MB (estimated)                                                            
         FileSize:   62.9 MB (estimated)                                        
                             WARNING: Snappy compression is not enabled         

------------------------------------------------                                
calc CRC over data @ 0x0xbf96b7c8 len 1 result 527d5351                         
                                               calc CRC over data @ 
0x0xbf96b7c8 len 1 result a016d052                                              
                          calc CRC over data @ 0x0xbf96b7c8 len 1 result 
b34623a6                                                                        
calc CRC over data @ 0x0xbf96b7c8 len 1 result 412da0a5                         
                                               calc CRC over data @ 
0x0xbf96b7c8 len 1 result 95e7c44e                                              
                          calc CRC over data @ 0x0x1005def4 len 34 result 
47a1355b                                                                        
calc CRC over data @ 0x0x1005e0ae len 35 result 37c0e56f                        
                                                checksum actual 0x37c0e56f 
masked 0x6d615a59 expected 0xa71abbfb (0x7ae836c1)open error: Corruption: 
checksum mismatch 

Original issue reported on code.google.com by jschind...@arcor.de on 24 Apr 2012 at 1:47

Attachments:

GoogleCodeExporter commented 9 years ago
Seems to me like your port doesn't have the correct value for
port::kLittleEndian and therefore the routines in coding.{h,cc}
are not working correctly.

To verify this:

(a) Print port::kLittleEndian at the beginning of some test
It should be false on your machine, but if my hypothesis is
correct, it will be true.

(b) Add the following test to coding_test and see if it works:
// Test that encoding routines generate little-endian encodings
TEST(Coding, EncodingOutput) {
  std::string dst;
  PutFixed32(&dst, 0x04030201);
  ASSERT_EQ(4, dst.size());
  ASSERT_EQ(0x01, static_cast<int>(dst[0]));
  ASSERT_EQ(0x02, static_cast<int>(dst[1]));
  ASSERT_EQ(0x03, static_cast<int>(dst[2]));
  ASSERT_EQ(0x04, static_cast<int>(dst[3]));

  dst.clear();
  PutFixed64(&dst, 0x0807060504030201);
  ASSERT_EQ(8, dst.size());
  ASSERT_EQ(0x01, static_cast<int>(dst[0]));
  ASSERT_EQ(0x02, static_cast<int>(dst[1]));
  ASSERT_EQ(0x03, static_cast<int>(dst[2]));
  ASSERT_EQ(0x04, static_cast<int>(dst[3]));
  ASSERT_EQ(0x05, static_cast<int>(dst[4]));
  ASSERT_EQ(0x06, static_cast<int>(dst[5]));
  ASSERT_EQ(0x07, static_cast<int>(dst[6]));
  ASSERT_EQ(0x08, static_cast<int>(dst[7]));
}

(c) I suspect the endian detection logic in port/port_posix.h
is wrong (it is seeing a name-clash between its use of LITTLE_ENDIAN
and some header file also defining it).  Try applying the following
patch:

--- /tmp/port/port_posix.h#30      2012-03-05 10:18:09.000000000 -0800
+++ port/port_posix.h    2012-04-25 09:52:04.835123000 -0700
@@ -7,14 +7,15 @@
 #ifndef STORAGE_LEVELDB_PORT_PORT_POSIX_H_
 #define STORAGE_LEVELDB_PORT_PORT_POSIX_H_

+#undef PLATFORM_IS_LITTLE_ENDIAN
 #if defined(OS_MACOSX)
   #include <machine/endian.h>
 #elif defined(OS_SOLARIS)
   #include <sys/isa_defs.h>
   #ifdef _LITTLE_ENDIAN
-    #define LITTLE_ENDIAN
+    #define PLATFORM_IS_LITTLE_ENDIAN true
   #else
-    #define BIG_ENDIAN
+    #define PLATFORM_IS_LITTLE_ENDIAN false
   #endif
 #elif defined(OS_FREEBSD) || defined(OS_OPENBSD) || defined(OS_NETBSD) ||\
       defined(OS_DRAGONFLYBSD)
@@ -31,10 +32,8 @@
 #include <string>
 #include "storage/leveldb/port/atomic_pointer.h"

-#ifdef LITTLE_ENDIAN
-#define IS_LITTLE_ENDIAN true
-#else
-#define IS_LITTLE_ENDIAN (__BYTE_ORDER == __LITTLE_ENDIAN)
+#ifndef PLATFORM_IS_LITTLE_ENDIAN
+#define PLATFORM_IS_LITTLE_ENDIAN (__BYTE_ORDER == __LITTLE_ENDIAN)
 #endif

 #if defined(OS_MACOSX) || defined(OS_SOLARIS) || defined(OS_FREEBSD) ||\
@@ -54,7 +53,8 @@
 namespace leveldb {
 namespace port {

-static const bool kLittleEndian = IS_LITTLE_ENDIAN;
+static const bool kLittleEndian = PLATFORM_IS_LITTLE_ENDIAN;
+#undef PLATFORM_IS_LITTLE_ENDIAN

 class CondVar;

Original comment by san...@google.com on 25 Apr 2012 at 4:55

GoogleCodeExporter commented 9 years ago
There was a bug in the endian-ness detection code.
leveldb 1.5.0 has the fix.

Original comment by san...@google.com on 30 May 2012 at 5:10

GoogleCodeExporter commented 9 years ago
Thanks for fixing that but problems still remain.

Programm db_bench Segfaults 
i found out that sometimes in function "FindGreaterOrEqual" in skiplist.h
"level" becomes "-1" and then "x->Next(level);" goes crazy of course.

With debug options i found also that n passed to "KeyIsAfterNode"
is not really a pointer and this makes crash the whole thing.

It seems that something is wrong with atomic stuff in port_posix.h !

Thanks for your hints.

Original comment by jschind...@arcor.de on 8 Jun 2012 at 3:13