Error when creating type map from source code

[sp1end0r]

Hi, I used TikNib about findutils. To do this, I got findutils source code (version 4.6.0) and downloaded binaries from normal dataset in BinKit repo.

Then, using IDA Pro 7.4, I proceeded "Building Ground Truth" step and It could make .pickle files from findutils binaries.

And, it attempted to extract function type information for type features, but I was faced with error when creating type map.

Error log is following:

jaehyu@olaf:~/repo/TikNib$ python3 helper/extract_functype.py --source_list "mytest/source_list.txt" --input_list "mytest/findutils_list.txt" --ctags_dir "data/ctags" --threshold 1
2022-02-28 16:23:22 olaf __main__[909] INFO creating ctags ...
2022-02-28 16:23:22 olaf __main__[909] INFO done. (0.000s)
2022-02-28 16:23:22 olaf __main__[909] INFO creating type map ...
Traceback (most recent call last):
  File "helper/extract_functype.py", line 97, in <module>
    update_type_map(type_map, ctags_fname)
  File "/home/jaehyu/repo/TikNib/helper/../tiknib/debug/functype.py", line 196, in update_type_map
    ret_type = pattern[: pattern.rindex(name)].rstrip()
ValueError: substring not found

For more detail, I added print in tiknib/debug/functype.py and got a following output.

pattern : /^  const uint8_t ABIVersion;$/
name : ABIVersion

pattern : /^    AssumptionCache *AC;$/
name : AC

pattern : /^    GENERAL_NAME *location;$/
name : ACCESS_DESCRIPTION_st::location

Traceback (most recent call last):
  File "helper/extract_functype.py", line 97, in <module>
    update_type_map(type_map, ctags_fname)
  File "/home/jaehyu/repo/TikNib/helper/../tiknib/debug/functype.py", line 199, in update_type_map
    ret_type = pattern[: pattern.rindex(name)].rstrip()
ValueError: substring not found

What can I do to solve this error ?

[0xdkay]

Hi, I haven't seen such an issue. Could you pass me the problematic binary and describe me the exact steps that you had proceeded?

[sp1end0r]

There is google drive link using binaries and source codes about findutils. (There are binaries in normal set from BinKit repo.) https://drive.google.com/file/d/1gDxBJbL_iAPyM-pXQ4yH4e5S00ilwpRZ/view?usp=sharing

After 5 step in Building Ground Truth, I attempted to extract function type information for type features in Extracting Features. (I assumed that the first step of Extracting Features is replaced by Building Ground Truth proceed.)

[zhao441100]

There is google drive link using binaries and source codes about findutils. (There are binaries in normal set from BinKit repo.) https://drive.google.com/file/d/1gDxBJbL_iAPyM-pXQ4yH4e5S00ilwpRZ/view?usp=sharing

After 5 step in Building Ground Truth, I attempted to extract function type information for type features in Extracting Features. (I assumed that the first step of Extracting Features is replaced by Building Ground Truth proceed.)

Hi, I used TikNib about findutils. To do this, I got findutils source code (version 4.6.0) and downloaded binaries from normal dataset in BinKit repo.

Then, using IDA Pro 7.4, I proceeded "Building Ground Truth" step and It could make .pickle files from findutils binaries.

And, it attempted to extract function type information for type features, but I was faced with error when creating type map.

Error log is following:

jaehyu@olaf:~/repo/TikNib$ python3 helper/extract_functype.py --source_list "mytest/source_list.txt" --input_list "mytest/findutils_list.txt" --ctags_dir "data/ctags" --threshold 1
2022-02-28 16:23:22 olaf __main__[909] INFO creating ctags ...
2022-02-28 16:23:22 olaf __main__[909] INFO done. (0.000s)
2022-02-28 16:23:22 olaf __main__[909] INFO creating type map ...
Traceback (most recent call last):
  File "helper/extract_functype.py", line 97, in <module>
    update_type_map(type_map, ctags_fname)
  File "/home/jaehyu/repo/TikNib/helper/../tiknib/debug/functype.py", line 196, in update_type_map
    ret_type = pattern[: pattern.rindex(name)].rstrip()
ValueError: substring not found

For more detail, I added print in tiknib/debug/functype.py and got a following output.

pattern : /^  const uint8_t ABIVersion;$/
name : ABIVersion

pattern : /^    AssumptionCache *AC;$/
name : AC

pattern : /^    GENERAL_NAME *location;$/
name : ACCESS_DESCRIPTION_st::location

Traceback (most recent call last):
  File "helper/extract_functype.py", line 97, in <module>
    update_type_map(type_map, ctags_fname)
  File "/home/jaehyu/repo/TikNib/helper/../tiknib/debug/functype.py", line 199, in update_type_map
    ret_type = pattern[: pattern.rindex(name)].rstrip()
ValueError: substring not found

What can I do to solve this error ?

Hi, I used TikNib about findutils. To do this, I got findutils source code (version 4.6.0) and downloaded binaries from normal dataset in BinKit repo.

Then, using IDA Pro 7.4, I proceeded "Building Ground Truth" step and It could make .pickle files from findutils binaries.

And, it attempted to extract function type information for type features, but I was faced with error when creating type map.

Error log is following:

jaehyu@olaf:~/repo/TikNib$ python3 helper/extract_functype.py --source_list "mytest/source_list.txt" --input_list "mytest/findutils_list.txt" --ctags_dir "data/ctags" --threshold 1
2022-02-28 16:23:22 olaf __main__[909] INFO creating ctags ...
2022-02-28 16:23:22 olaf __main__[909] INFO done. (0.000s)
2022-02-28 16:23:22 olaf __main__[909] INFO creating type map ...
Traceback (most recent call last):
  File "helper/extract_functype.py", line 97, in <module>
    update_type_map(type_map, ctags_fname)
  File "/home/jaehyu/repo/TikNib/helper/../tiknib/debug/functype.py", line 196, in update_type_map
    ret_type = pattern[: pattern.rindex(name)].rstrip()
ValueError: substring not found

For more detail, I added print in tiknib/debug/functype.py and got a following output.

pattern : /^  const uint8_t ABIVersion;$/
name : ABIVersion

pattern : /^    AssumptionCache *AC;$/
name : AC

pattern : /^    GENERAL_NAME *location;$/
name : ACCESS_DESCRIPTION_st::location

Traceback (most recent call last):
  File "helper/extract_functype.py", line 97, in <module>
    update_type_map(type_map, ctags_fname)
  File "/home/jaehyu/repo/TikNib/helper/../tiknib/debug/functype.py", line 199, in update_type_map
    ret_type = pattern[: pattern.rindex(name)].rstrip()
ValueError: substring not found

What can I do to solve this error ?

Do you use crack ida pro 7.4 in ubuntu? I can not find crack ida 7.5 in ubuntu..

[sp1end0r]

No. I don't use crack IDA pro.

[0xdkay]

@sp1end0r, it's quite strange. I just downloaded your binaries from the google drive and successfully ran it without any modification.

dongkwan ~/tiknib master* $ python helper/extract_functype.py \
    --source_list "example/source_list.txt" \
    --input_list "example/input_list_find.txt" \
    --ctags_dir "data/ctags" \
    --threshold 1
2022-03-01 13:17:47 syssec144 __main__[99571] INFO creating ctags ...
2022-03-01 13:17:47 syssec144 __main__[99571] INFO done. (0.043s)
2022-03-01 13:17:47 syssec144 __main__[99571] INFO Processing 128 binaries ...
2022-03-01 13:17:48 syssec144 __main__[99571] INFO done. (0.626s)
dongkwan ~/tiknib master* $
dongkwan ~/tiknib master* $ cat example/source_list.txt
/home/dongkwan/tiknib/test/mytest/findutils-4.6.0
dongkwan ~/tiknib master* $ head example/input_list_find.txt
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_arm_32_O0_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_arm_32_O1_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_arm_32_O2_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_arm_32_O3_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_arm_64_O0_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_arm_64_O1_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_arm_64_O2_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_arm_64_O3_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_mips_32_O0_find.elf
/home/dongkwan/tiknib/test/mytest/findutils/findutils-4.6.0_clang-4.0_mips_32_O1_find.elf

One possible reason would be your default include directory is different. TikNib uses /usr/include/ by default; thus, your system may not use that path. To check the code, please see this link.

Adding your system's include path to the source_list.txt file would address the problem.

[sp1end0r]

Thank you for comment! As you said, it is my problem about include directory...

[0xdkay]

Glad you addressed it. Please close the issue if there is no further problem :)