search

DGA-MI-SSI / YaCo

YaCo is an Hex-Rays IDA plugin. When enabled, multiple users can work simultaneously on the same binary. Any modification done by any user is synchronized through git version control.
GNU General Public License v3.0
313 stars 36 forks source link

Problem when merging array in structures #33

Closed cbayet closed 6 years ago

cbayet commented 6 years ago

Hi,

YaCo breaks a lot the arrays defined in structures.

It seems to happen a lot with arrays of structures, and when we have several arrays in the same structure ; when we define one, the other is not considered as an array anymore.

Here is a git diff of a commit that breaks structures:

diff --git a/cache/strucmember/65C3F15F2504B1BF.xml b/cache/strucmember/65C3F15F2504B1BF.xml
index 5217e0c..02e643e 100644
--- a/cache/strucmember/65C3F15F2504B1BF.xml
+++ b/cache/strucmember/65C3F15F2504B1BF.xml
@@ -7,7 +7,7 @@
       <parent_id>1D11CCBAE74D78E8</parent_id>
       <address>C0</address>
       <userdefinedname>internalArray</userdefinedname>
-      <proto>InternalStructArray /*%InternalStructArray#F594934A80AA7F38%*/[8]</proto>
+      <proto>InternalStructArray /*%InternalStructArray#F594934A80AA7F38%*/</proto>
       <flags>0x60000400</flags>
       <signatures/>
       <xrefs>

diff --git a/cache/strucmember/9F04AED9FA27A903.xml b/cache/strucmember/9F04AED9FA27A903.xml
index 1795d4c..6db84cd 100644
--- a/cache/strucmember/9F04AED9FA27A903.xml
+++ b/cache/strucmember/9F04AED9FA27A903.xml
@@ -3,7 +3,7 @@
   <strucmember>
     <id>9F04AED9FA27A903</id>
     <version>
-      <size>0x0000000000000020</size>
+      <size>0x0000000000000320</size>
       <parent_id>1D11CCBAE74D78E8</parent_id>
       <address>1930</address>
       <userdefinedname>ArrayConf</userdefinedname>

As you can see, sometimes it breaks the size, sometimes it breaks the prototype. We investigated a bit in the code, and we think it might come from the simple_tif_to_string() function, that removes the pointer and arrays, and only recovers pointers...

As a result of such a commit, the array is undefined and set as unique element:

00000000 MyStruct        MyStruct ?               
00000010                 db ? ; undefined
00000011                 db ? ; undefined
00000012                 db ? ; undefined
00000013                 db ? ; undefined
00000014                 db ? ; undefined
00000015                 db ? ; undefined

Tested with IDA 7.1

bamiaux commented 6 years ago

Thanks for the bug report I've reproduced it and will look at it soon

bamiaux commented 6 years ago

Please compile & try https://github.com/DGA-MI-SSI/YaCo/tree/issue_33 which should fix those bugs

cbayet commented 6 years ago

Hi,

Thanks for the quick response. Unfortunately, I can't manage to build the project, and for now I used compiled binaries.

I have troubles compiling the project for Windows, the configure_2017.cmd fails as it tries to fetch some standard libraries:

C:\Users\user\YaCo\build>configure_2017.cmd
-- Selecting Windows SDK version 10.0.16299.0 to target Windows 6.1.7601.
-- Configuring for nt_x64
-- Using IDASDK_DIR=C:/Users/user/YaCo/build/"C:/Users/user/Documents/Share/idasdk71"
-- Using IDA_DIR=C:/Users/user/YaCo/build/"C:/Program Files/IDA 7.1"
-- Looking for snprintf
-- Looking for snprintf - found
-- Looking for select
-- Looking for select - found
-- Looking for CryptDecodeObjectEx
-- Looking for CryptDecodeObjectEx - found
-- Looking for setlocale
-- Looking for setlocale - found
-- Looking for dlfcn.h
-- Looking for dlfcn.h - not found
-- Looking for inttypes.h
-- Looking for inttypes.h - found
-- Looking for memory.h
-- Looking for memory.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stdlib.h
-- Looking for stdlib.h - found
-- Looking for string.h
-- Looking for string.h - found
-- Looking for strings.h
-- Looking for strings.h - not found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for unistd.h
-- Looking for unistd.h - not found
-- Configuring yatools/deps/charset
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of mbstate_t
-- Check size of mbstate_t - done
-- Check size of wchar_t
-- Check size of wchar_t - done
-- Check size of long long int
-- Check size of long long int - done
-- Check size of sigset_t
-- Check size of sigset_t - failed
-- Check size of unsigned long long int
-- Check size of unsigned long long int - done
-- Check size of _Bool
-- Check size of _Bool - done
-- Looking for _NSGetExecutablePath
-- Looking for _NSGetExecutablePath - not found
-- Looking for canonicalize_file_name
-- Looking for canonicalize_file_name - not found
-- Looking for getc_unlocked
-- Looking for getc_unlocked - not found
-- Looking for getcwd
-- Looking for getcwd - found
-- Looking for lstat
-- Looking for lstat - not found
-- Looking for mbrtowc
-- Looking for mbrtowc - found
-- Looking for mbsinit
-- Looking for mbsinit - not found
-- Looking for memmove
-- Looking for memmove - found
-- Looking for readlink
-- Looking for readlink - not found
-- Looking for readlinkat
-- Looking for readlinkat - not found
-- Looking for realpath
-- Looking for realpath - not found
-- Looking for setenv
-- Looking for setenv - not found
-- Looking for strerror_r
-- Looking for strerror_r - not found
-- Looking for tsearch
-- Looking for tsearch - not found
-- Looking for wcrtomb
-- Looking for wcrtomb - found
-- Looking for mach-o/dyld.h
-- Looking for mach-o/dyld.h - not found
-- Looking for search.h
-- Looking for search.h - found
-- Looking for sys/bitypes.h
-- Looking for sys/bitypes.h - not found
-- Looking for sys/inttypes.h
-- Looking for sys/inttypes.h - not found
-- Looking for sys/param.h
-- Looking for sys/param.h - not found
-- Looking for sys/socket.h
-- Looking for sys/socket.h - not found
-- Looking for sys/time.h
-- Looking for sys/time.h - not found
-- Looking for wchar.h
-- Looking for wchar.h - found
-- Looking for winsock2.h
-- Looking for winsock2.h - found
-- Configuring yatools/deps/iconv
-- Looking for _stat
-- Looking for _stat - found
-- Looking for class
-- Looking for class - not found
-- Looking for finite
-- Looking for finite - not found
-- Looking for fp_class
-- Looking for fp_class - not found
-- Looking for fpclass
-- Looking for fpclass - not found
-- Looking for fprintf
-- Looking for fprintf - not found
-- Looking for ftime
-- Looking for ftime - found
-- Looking for gettimeofday
-- Looking for gettimeofday - not found
-- Looking for isnand
-- Looking for isnand - not found
-- Looking for localtime
-- Looking for localtime - found
-- Looking for printf
-- Looking for printf - not found
-- Looking for signal
-- Looking for signal - found
-- Looking for sprintf
-- Looking for sprintf - not found
-- Looking for sscanf
-- Looking for sscanf - not found
-- Looking for stat
-- Looking for stat - found
-- Looking for strdup
-- Looking for strdup - found
-- Looking for strerror
-- Looking for strerror - found
-- Looking for strftime
-- Looking for strftime - found
-- Looking for strndup
-- Looking for strndup - not found
-- Looking for vfprintf
-- Looking for vfprintf - not found
-- Looking for vsnprintf
-- Looking for vsnprintf - not found
-- Looking for vsprintf
-- Looking for vsprintf - not found
-- Looking for ansidecl.h
-- Looking for ansidecl.h - not found
-- Looking for arpa/inet.h
-- Looking for arpa/inet.h - not found
-- Looking for arpa/nameser.h
-- Looking for arpa/nameser.h - not found
-- Looking for ctype.h
-- Looking for ctype.h - found
-- Looking for dirent.h
-- Looking for dirent.h - not found
-- Looking for dl.h
-- Looking for dl.h - not found
-- Looking for errno.h
-- Looking for errno.h - found
-- Looking for fcntl.h
-- Looking for fcntl.h - found
-- Looking for float.h
-- Looking for float.h - found
-- Looking for fp_class.h
-- Looking for fp_class.h - not found
-- Looking for ieeefp.h
-- Looking for ieeefp.h - not found
-- Looking for inttypes.h.h
-- Looking for inttypes.h.h - not found
-- Looking for limits.h
-- Looking for limits.h - found
-- Looking for malloc.h
-- Looking for malloc.h - found
-- Looking for math.h
-- Looking for math.h - found
-- Looking for nan.h
-- Looking for nan.h - not found
-- Looking for ndir.h
-- Looking for ndir.h - not found
-- Looking for netdb.h
-- Looking for netdb.h - not found
-- Looking for netinet/in.h
-- Looking for netinet/in.h - not found
-- Looking for poll.h
-- Looking for poll.h - not found
-- Looking for resolv.h
-- Looking for resolv.h - not found
-- Looking for signal.h
-- Looking for signal.h - found
-- Looking for stdarg.h
-- Looking for stdarg.h - found
-- Looking for sys/dir.h
-- Looking for sys/dir.h - not found
-- Looking for sys/mman.h
-- Looking for sys/mman.h - not found
-- Looking for sys/ndir.h
-- Looking for sys/ndir.h - not found
-- Looking for sys/select.h
-- Looking for sys/select.h - not found
-- Looking for sys/timeb.h
-- Looking for sys/timeb.h - found
-- Looking for time.h
-- Looking for time.h - found
-- Looking for zlib.h
-- Looking for zlib.h - not found
-- Configuring yatools/deps/libxml2
-- Configuring yatools/deps/regex
-- Configuring yatools/deps/zlib
-- Configuring yatools/deps/http_parser
-- Configuring yatools/deps/mbedtls
-- Performing Test HAVE_O_NONBLOCK
-- Performing Test HAVE_O_NONBLOCK - Failed
-- Performing Test HAVE_FIONBIO
-- Performing Test HAVE_FIONBIO - Failed
-- Performing Test HAVE_IOCTLSOCKET
-- Performing Test HAVE_IOCTLSOCKET - Success
-- Looking for disabled_nonblocking
-- Looking for disabled_nonblocking - not found
-- Looking for evp_aes_128_ctr
-- Looking for evp_aes_128_ctr - not found
-- Looking for inet_addr
-- Looking for inet_addr - found
-- Looking for ioctlsocket_case
-- Looking for ioctlsocket_case - not found
-- Looking for longlong
-- Looking for longlong - not found
-- Looking for poll
-- Looking for poll - not found
-- Looking for so_nonblock
-- Looking for so_nonblock - not found
-- Looking for socket
-- Looking for socket - found
-- Looking for strtoi64
-- Looking for strtoi64 - not found
-- Looking for strtoll
-- Looking for strtoll - found
-- Looking for ntdef.h
-- Looking for ntdef.h - not found
-- Looking for ntstatus.h
-- Looking for ntstatus.h - found
-- Looking for sys/ioctl.h
-- Looking for sys/ioctl.h - not found
-- Looking for sys/uio.h
-- Looking for sys/uio.h - not found
-- Looking for sys/un.h
-- Looking for sys/un.h - not found
-- Looking for windows.h
-- Looking for windows.h - found
-- Looking for ws2tcpip.h
-- Looking for ws2tcpip.h - found
-- Configuring yatools/deps/ssh2
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Configuring yatools/deps/git2
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM - Failed
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC - Failed
-- Performing Test HAVE_STRUCT_STAT_MTIME_NSEC
-- Performing Test HAVE_STRUCT_STAT_MTIME_NSEC - Failed
-- Configuring yatools/deps/flatc
-- Configuring yatools/deps/flatbuffers
-- Configuring yatools/deps/gtest
-- Configuring yatools/deps/swig
-- Configuring yatools/deps/farmhash
CMake Error at C:/Program Files/CMake/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)
  (Required is at least version "2.7")
Call Stack (most recent call first):
  C:/Program Files/CMake/share/cmake-3.11/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  C:/Program Files/CMake/share/cmake-3.11/Modules/FindPythonLibs.cmake:261 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  yatools.cmake:40 (find_package)
  CMakeLists.txt:17 (include)

-- Configuring incomplete, errors occurred!
See also "C:/Users/user/YaCo/out/x64/CMakeFiles/CMakeOutput.log".
See also "C:/Users/user/YaCo/out/x64/CMakeFiles/CMakeError.log".

I tried to set PYTHON_LIBRARIES or PYTHON_INCLUDE_DIRS but nothing changes

Is there some dependances you did not mentioned ? I got Visual Studio Community 2017 and Cmake 3.11

bamiaux commented 6 years ago

You need python 2.7 64-bit headers & libraries. IDA used to install them on windows, but maybe not anymore. You should have them if you install the default 2.7 install from python site Anyway, I'll make a bugfix release

bamiaux commented 6 years ago

You can check 1.8.2 release

cbayet commented 6 years ago

Hi,

I finally managed to compile the project (I had to repair my python install, seems it was broken). Unfortunately, the tests doesn't work, it seems the Qt5Svgd.dll.i64 hasn't be created ; I didn't investigated much though.

Anyway I didn't test the fix yet, I'll do it soon with your release

We used YaCo last days and we discovered a bunch of bugs, do you want us to open a git issue for each one ? We thought it might be better if we could directly chat with you (on a irc for example) because some bugs seems tricky or hard to reproduce. Of course it's relevant only if the project is under active development.

Thanks for the quick response and fixes <3

bamiaux commented 6 years ago

Make sure you're using IDA 7.1 for tests, it won't work anymore for 7.0 with current test code Feel free to post any bugs you find, some are easy to fix and some are hard. I'd like to collect more feedback on which bugs are not fixable with the current design before breaking it. YaTools is definitely active, we will talk & publish YaDiff soon https://www.sstic.org/2018/presentation/yadiff/

cbayet commented 6 years ago

It seems that your fix resolved the issue !

Thanks again