failed to load pickle

[RocketMaDev]

I'd like to reproduce the pathsin pickles, but some errors occurred. At the beginning, I guessed it was because of my python is latest, but when I used virtualenv to setup a python 3.8 and it still failed, I was confused. It installed angr-9.2.102 automatically.

Here is the output:

In [3]: with open('_IO_file_finish.pickle', 'rb') as f:
   ...:     states = pickle.load(f)
   ...: 
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[3], line 2
      1 with open('_IO_file_finish.pickle', 'rb') as f:
----> 2     states = pickle.load(f)

File ~/angry-FSROP/lib/python3.8/site-packages/angr/engines/unicorn.py:58, in SimEngineUnicorn.__setstate__(self, args)
     57 def __setstate__(self, args):
---> 58     super().__setstate__(args[0])
     59     self._block_details_cache = args[1]
     60     self._stop_block_addrs_cache = args[2]

File ~/angry-FSROP/lib/python3.8/site-packages/angr/engines/vex/light/light.py:44, in VEXMixin.__setstate__(self, s)
     42 def __setstate__(self, s):
     43     self.__init_handlers()
---> 44     super().__setstate__(s[0])

File ~/angry-FSROP/lib/python3.8/site-packages/angr/engines/vex/lifter.py:412, in VEXLifter.__setstate__(self, state)
    411 def __setstate__(self, state):
--> 412     s, ostate = state
    413     self._use_cache = s["_use_cache"]
    414     self._default_opt_level = s["_default_opt_level"]

ValueError: too many values to unpack (expected 2)

I would like to know the correct version to load these pickles.

[Kyle-Kyle]

yeah. it looks like an angr version problem. I'd suggest you to just run the script again by python angry-fsrop.py

[RocketMaDev]

It seems that I have insufficient memory. I tried it, but soon the whole system stopped, then python is killed. I'm afraid my 16-gig laptop is not eligible to run the script.

[Kyle-Kyle]

I'm testing it, it uses about 38G of memory... But anyway, I'll update the pickle after it finishes running

[RocketMaDev]

Please let me know your angr version when publishing new pickles so that I could load them correctly. :heart:

[RocketMaDev]

I borrowed a server with 118G of RAM from my friend and rerun the script. I could load my pickles with angr 9.2.118 and python 3.12.6, I'll keep these pickles for some time, please let me know if I need to make a PR to upload them.

First I ran nm 389d485a9793dbe873f0ea2c93e02efaa9aa3d.debug | sort > symbols.txt to get sorted symbols, then I ran ll | awk '{print $9}' | sed 's/.*/"\0",/' > file to make them easy to load in Python, finally I wrote the script below to know the control flow:

import pickle
import bisect

offsets = []
symbols = []
with open('bins/symbols.txt', 'r') as symfile:
    for line in symfile:
        resolve = line.strip().split(' ')
        offsets.append(int(resolve[0], 16))
        symbols.append(resolve[2])

files = [
    '_IO_cookie_seekoff.pickle',
    '_IO_file_finish.pickle',
    '_IO_file_overflow.pickle',
    '_IO_file_seekoff.pickle',
    '_IO_file_setbuf_mmap.pickle',
    '_IO_file_sync.pickle',
    '_IO_file_underflow_maybe_mmap.pickle',
    '_IO_file_underflow.pickle',
    '_IO_file_xsgetn_maybe_mmap.pickle',
    '_IO_file_xsgetn_mmap.pickle',
    '_IO_obstack_overflow.pickle',
    '_IO_obstack_xsputn.pickle',
    '_IO_wdefault_xsgetn.pickle',
    '_IO_wfile_overflow.pickle',
    '_IO_wfile_seekoff.pickle',
    '_IO_wfile_sync.pickle',
    '_IO_wfile_underflow_maybe_mmap.pickle',
    '_IO_wfile_underflow_mmap.pickle',
    '_IO_wfile_underflow.pickle'
]
for picfile in files:
    print(f"paths for {picfile[:picfile.find('.')]}:")
    with open('outputs/' + picfile, 'rb') as pic:
        paths = pickle.load(pic)
        for faddr in paths[0].history.bbl_addrs:
            idx = bisect.bisect(offsets, faddr) - 1
            if faddr == offsets[idx]:
                print(f"    {symbols[idx]}")
            else:
                print(f"    {symbols[idx]} + {faddr - offsets[idx]}")
    print()

In next comment I'll post the result.

[RocketMaDev]

paths for _IO_cookie_seekoff:
    _IO_cookie_seekoff
    _IO_new_file_seekoff
    _IO_new_file_seekoff + 51
    _IO_new_file_seekoff + 232
    _IO_new_file_seekoff + 241
    _IO_new_file_seekoff + 269
    _IO_new_file_seekoff + 118
    _IO_new_file_seekoff + 128
    _IO_new_file_seekoff + 138
    _IO_new_file_seekoff + 177
    _IO_obstack_xsputn
    _IO_obstack_xsputn + 44
    _IO_obstack_xsputn + 144
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_file_finish:
    _IO_new_file_finish
    _IO_new_file_finish + 19
    _IO_new_file_finish + 29
    _IO_wdo_write
    _IO_wdo_write + 53
    _IO_wdo_write + 77
    _IO_wdo_write + 254
    _IO_wdo_write + 112
    __libio_codecvt_out
    __libio_codecvt_out + 106
    __GI__dl_mcount_wrapper_check
    __GI__dl_mcount_wrapper_check + 48
    __libio_codecvt_out + 119

paths for _IO_file_overflow:
    _IO_new_file_overflow
    _IO_new_file_overflow + 21
    _IO_new_file_overflow + 32
    _IO_new_file_overflow + 41
    _IO_new_file_overflow + 54
    _IO_new_file_overflow + 288
    _IO_new_file_overflow + 298
    _IO_wdo_write
    _IO_wdo_write + 53
    _IO_wdo_write + 77
    _IO_wdo_write + 254
    _IO_wdo_write + 112
    __libio_codecvt_out
    __libio_codecvt_out + 106
    __GI__dl_mcount_wrapper_check
    __GI__dl_mcount_wrapper_check + 48
    __libio_codecvt_out + 119

paths for _IO_file_seekoff:
    _IO_new_file_seekoff
    _IO_new_file_seekoff + 51
    _IO_new_file_seekoff + 232
    _IO_new_file_seekoff + 241
    _IO_new_file_seekoff + 269
    _IO_new_file_seekoff + 118
    _IO_new_file_seekoff + 128
    _IO_new_file_seekoff + 138
    _IO_new_file_seekoff + 177
    _IO_obstack_xsputn
    _IO_obstack_xsputn + 44
    _IO_obstack_xsputn + 144
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_file_setbuf_mmap:
    _IO_file_setbuf_mmap
    _IO_new_file_setbuf
    _IO_default_setbuf
    _IO_default_setbuf + 62
    _IO_new_file_sync
    _IO_new_file_sync + 26
    _IO_new_file_sync + 36
    _IO_wdo_write
    _IO_wdo_write + 53
    _IO_wdo_write + 77
    _IO_wdo_write + 254
    _IO_wdo_write + 112
    __libio_codecvt_out
    __libio_codecvt_out + 106
    __GI__dl_mcount_wrapper_check
    __GI__dl_mcount_wrapper_check + 48
    __libio_codecvt_out + 119

paths for _IO_file_sync:
    _IO_new_file_sync
    _IO_new_file_sync + 26
    _IO_new_file_sync + 36
    _IO_wdo_write
    _IO_wdo_write + 53
    _IO_wdo_write + 77
    _IO_wdo_write + 254
    _IO_wdo_write + 112
    __libio_codecvt_out
    __libio_codecvt_out + 106
    __GI__dl_mcount_wrapper_check
    __GI__dl_mcount_wrapper_check + 48
    __libio_codecvt_out + 119

paths for _IO_file_underflow_maybe_mmap:
    _IO_file_underflow_maybe_mmap
    _IO_file_underflow_maybe_mmap + 78
    _IO_obstack_xsputn
    _IO_obstack_xsputn + 44
    _IO_obstack_xsputn + 144
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_file_underflow:
    _IO_new_file_underflow
    _IO_new_file_underflow + 14
    _IO_new_file_underflow + 39
    _IO_new_file_underflow + 53
    _IO_new_file_underflow + 64
    _IO_new_file_underflow + 290
    _IO_switch_to_get_mode
    _IO_switch_to_get_mode + 78
    _IO_switch_to_get_mode + 85
    _IO_switch_to_get_mode + 103
    _IO_new_file_underflow + 325
    _IO_new_file_underflow + 377
    _IO_obstack_overflow
    _IO_obstack_overflow + 20
    _IO_obstack_overflow + 96
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_file_xsgetn_maybe_mmap:
    _IO_file_xsgetn_maybe_mmap
    _IO_file_xsgetn_maybe_mmap + 88
    _IO_obstack_xsputn
    _IO_obstack_xsputn + 44
    _IO_obstack_xsputn + 144
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_file_xsgetn_mmap:
    _IO_file_xsgetn_mmap
    _IO_file_xsgetn_mmap + 72
    _IO_file_xsgetn_mmap + 504
    __abi_tag + 163900
    __mempcpy_sse2_unaligned
    __memmove_sse2_unaligned + 7
    __memmove_sse2_unaligned_erms + 48
    __memmove_sse2_unaligned_erms + 53
    __memmove_sse2_unaligned_erms + 80
    _IO_file_xsgetn_mmap + 521
    _IO_switch_to_main_get_area
    _IO_file_xsgetn_mmap + 532
    _IO_file_xsgetn_mmap + 552
    _IO_file_xsgetn_mmap + 84
    _IO_file_xsgetn_mmap + 123
    _IO_obstack_xsputn
    _IO_obstack_xsputn + 44
    _IO_obstack_xsputn + 144
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_obstack_overflow:
    _IO_obstack_overflow
    _IO_obstack_overflow + 20
    _IO_obstack_overflow + 96
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_obstack_xsputn:
    _IO_obstack_xsputn
    _IO_obstack_xsputn + 44
    _IO_obstack_xsputn + 144
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_wdefault_xsgetn:
    _IO_wdefault_xsgetn
    _IO_wdefault_xsgetn + 107
    _IO_wdefault_xsgetn + 116
    _IO_wdefault_xsgetn + 131
    _IO_wdefault_xsgetn + 137
    _IO_wdefault_xsgetn + 395
    _IO_switch_to_wget_mode
    _IO_switch_to_wget_mode + 25

paths for _IO_wfile_overflow:
    _IO_wfile_overflow
    _IO_wfile_overflow + 24
    _IO_wfile_overflow + 32
    _IO_wfile_overflow + 608
    _IO_wdoallocbuf
    _IO_wdoallocbuf + 24
    _IO_wdoallocbuf + 36

paths for _IO_wfile_seekoff:
    _IO_wfile_seekoff
    _IO_wfile_seekoff + 56
    _IO_wfile_seekoff + 84
    _IO_wfile_seekoff + 101
    _IO_switch_to_wget_mode
    _IO_switch_to_wget_mode + 25

paths for _IO_wfile_sync:
    _IO_wfile_sync
    _IO_wfile_sync + 37
    _IO_wfile_sync + 51
    _IO_wdo_write
    _IO_wdo_write + 53
    _IO_wdo_write + 77
    _IO_wdo_write + 254
    _IO_wdo_write + 112
    __libio_codecvt_out
    __libio_codecvt_out + 106
    __GI__dl_mcount_wrapper_check
    __GI__dl_mcount_wrapper_check + 48
    __libio_codecvt_out + 119

paths for _IO_wfile_underflow_maybe_mmap:
    _IO_wfile_underflow_maybe_mmap
    _IO_file_underflow_maybe_mmap
    _IO_file_underflow_maybe_mmap + 78
    _IO_obstack_xsputn
    _IO_obstack_xsputn + 44
    _IO_obstack_xsputn + 144
    _obstack_newchunk
    _obstack_newchunk + 448

paths for _IO_wfile_underflow_mmap:
    _IO_wfile_underflow_mmap
    _IO_wfile_underflow_mmap + 37
    _IO_wfile_underflow_mmap + 57
    _IO_wfile_underflow_mmap + 78
    _IO_wfile_underflow_mmap + 262
    _IO_wfile_underflow_mmap + 282
    _IO_wdoallocbuf
    _IO_wdoallocbuf + 24
    _IO_wdoallocbuf + 36

paths for _IO_wfile_underflow:
    _IO_wfile_underflow
    _IO_wfile_underflow + 42
    _IO_wfile_underflow + 53
    _IO_wfile_underflow + 73
    _IO_wfile_underflow + 1200
    __libio_codecvt_in
    __libio_codecvt_in + 105
    __GI__dl_mcount_wrapper_check
    __GI__dl_mcount_wrapper_check + 48
    __libio_codecvt_in + 118

[RocketMaDev]

I searched out source code of _IO_switch_to_wget_mode:

int
_IO_switch_to_wget_mode (FILE *fp)
{
  if (fp->_wide_data->_IO_write_ptr > fp->_wide_data->_IO_write_base)
    if ((wint_t)_IO_WOVERFLOW (fp, WEOF) == WEOF)
      return EOF;
  if (_IO_in_backup (fp))
    fp->_wide_data->_IO_read_base = fp->_wide_data->_IO_backup_base;
  else
    {
      fp->_wide_data->_IO_read_base = fp->_wide_data->_IO_buf_base;
      if (fp->_wide_data->_IO_write_ptr > fp->_wide_data->_IO_read_end)
    fp->_wide_data->_IO_read_end = fp->_wide_data->_IO_write_ptr;
    }
  fp->_wide_data->_IO_read_ptr = fp->_wide_data->_IO_write_ptr;

  fp->_wide_data->_IO_write_base = fp->_wide_data->_IO_write_ptr
    = fp->_wide_data->_IO_write_end = fp->_wide_data->_IO_read_ptr;

  fp->_flags &= ~_IO_CURRENTLY_PUTTING;
  return 0;
}
libc_hidden_def (_IO_switch_to_wget_mode)

Similarly, it calls _IO_WOVERFLOW macro to control rip. In conclusion, there is only 3 primitives with FSROP: __libio_codecvt_*, _obstack_newchunk and _IO_switch_to_wget_mode/_IO_wdoallocbuf.

[Kyle-Kyle]

That's good to know! Thanks for sharing!