search

joxeankoret / diaphora

Diaphora, the most advanced Free and Open Source program diffing tool.
http://diaphora.re
GNU Affero General Public License v3.0
3.58k stars 371 forks source link

Ida stuck on running python script after a while #210

Closed smikkel997 closed 3 years ago

smikkel997 commented 3 years ago

After running the python script, after a while, ida stopped drawing all my resources but it still showed the running python script screen. I have let it run for over 20 hours but it stll is on the same screen, how do i fix this?

joxeankoret commented 3 years ago

I don't know what to say. Kill the process and try again? I don't understand why it might happen. Does it happen every time you launch it?

joxeankoret commented 3 years ago

I'm closing this issue as I don't think it happened to you again. Feel free to reopen it if needed.

smikkel997 commented 3 years ago

it could be because i dont have enough diskspace, since my disk is almost full, how much diskspace do i need to compare 2 sqlite databases which are both sized at about 1.3gb?

http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avg.com http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Op vr 5 feb. 2021 om 14:53 schreef Joxean notifications@github.com:

I don't know what to say. Kill the process and try again? I don't understand why it might happen. Does it happen every time you launch it?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joxeankoret/diaphora/issues/210#issuecomment-774045902, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDEYVA7YBHORYX4VTOLS5PZ6TANCNFSM4XEIOEDQ .

liettua commented 3 years ago

I don't know what to say. Kill the process and try again? I don't understand why it might happen. Does it happen every time you launch it?

same happens here when i enable use slow heuristics. it looks like its working and using 10% of cpu + 3gigs of ram

then just stops using resources and just shows the screen

smikkel997 commented 3 years ago

So i need to disable slow heuristics?

http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avg.com http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Op wo 24 feb. 2021 om 16:00 schreef yEsPaP notifications@github.com:

I don't know what to say. Kill the process and try again? I don't understand why it might happen. Does it happen every time you launch it?

same happens here when i enable use slow heuristics. it looks like its working and using 10% of cpu + 3gigs of ram

then just stops using resources and just shows the screen

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joxeankoret/diaphora/issues/210#issuecomment-785136080, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDDWF5XPEPBA2XJ36TDTAUICBANCNFSM4XEIOEDQ .

liettua commented 3 years ago

So i need to disable slow heuristics? http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avg.com http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> Op wo 24 feb. 2021 om 16:00 schreef yEsPaP notifications@github.com: … I don't know what to say. Kill the process and try again? I don't understand why it might happen. Does it happen every time you launch it? same happens here when i enable use slow heuristics. it looks like its working and using 10% of cpu + 3gigs of ram then just stops using resources and just shows the screen — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#210 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDDWF5XPEPBA2XJ36TDTAUICBANCNFSM4XEIOEDQ .

you can try but i dont know

joxeankoret commented 3 years ago

It seems you folks are working with very big databases. SQLite uses files on disk as intermediate things when running some queries so, if you're working with huge databases, it will create a huge temporary files (and will take very long). Slow heuristics are more likely to cause/trigger this problem so, yes, I recommend you to disable slow heuristics if they are causing you problems.

smikkel997 commented 3 years ago

thanks for the help (it indeed took very long)

http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avg.com http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Op wo 24 feb. 2021 om 17:49 schreef Joxean notifications@github.com:

It seems you folks are working with very big databases. SQLite uses files on disk as intermediate things when running some queries so, if you're working with huge databases, it will create a huge temporary files (and will take very long). Slow heuristics are more likely to cause/trigger this problem so, yes, I recommend you to disable slow heuristics if they are causing you problems.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joxeankoret/diaphora/issues/210#issuecomment-785214999, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDD63E7KD3YGZPBPTVDTAUUZ7ANCNFSM4XEIOEDQ .

smikkel997 commented 3 years ago

how can i change the directory in which it creates the temporary files?

Op wo 24 feb. 2021 om 17:49 schreef Joxean notifications@github.com:

It seems you folks are working with very big databases. SQLite uses files on disk as intermediate things when running some queries so, if you're working with huge databases, it will create a huge temporary files (and will take very long). Slow heuristics are more likely to cause/trigger this problem so, yes, I recommend you to disable slow heuristics if they are causing you problems.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joxeankoret/diaphora/issues/210#issuecomment-785214999, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDD63E7KD3YGZPBPTVDTAUUZ7ANCNFSM4XEIOEDQ .

smikkel997 commented 3 years ago

I have diaphora diffing two executables (sqlite databases of both 850MB) but for some reason, it takes already more than 70GB of space of my usb. Is this supposed to happen or not?

Op ma 1 mrt. 2021 om 12:31 schreef Viktor Y smikkelbeer081@gmail.com:

how can i change the directory in which it creates the temporary files?

Op wo 24 feb. 2021 om 17:49 schreef Joxean notifications@github.com:

It seems you folks are working with very big databases. SQLite uses files on disk as intermediate things when running some queries so, if you're working with huge databases, it will create a huge temporary files (and will take very long). Slow heuristics are more likely to cause/trigger this problem so, yes, I recommend you to disable slow heuristics if they are causing you problems.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joxeankoret/diaphora/issues/210#issuecomment-785214999, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDD63E7KD3YGZPBPTVDTAUUZ7ANCNFSM4XEIOEDQ .

smikkel997 commented 3 years ago

the executables i tried to diff are both 16mb in size, yet ida took more than 110GB of storage before i decided to quit the program, do you know why this is happening? (i didn't use slow heuristics)

Op ma 1 mrt. 2021 om 15:59 schreef Viktor Y smikkelbeer081@gmail.com:

I have diaphora diffing two executables (sqlite databases of both 850MB) but for some reason, it takes already more than 70GB of space of my usb. Is this supposed to happen or not?

Op ma 1 mrt. 2021 om 12:31 schreef Viktor Y smikkelbeer081@gmail.com:

how can i change the directory in which it creates the temporary files?

Op wo 24 feb. 2021 om 17:49 schreef Joxean notifications@github.com:

It seems you folks are working with very big databases. SQLite uses files on disk as intermediate things when running some queries so, if you're working with huge databases, it will create a huge temporary files (and will take very long). Slow heuristics are more likely to cause/trigger this problem so, yes, I recommend you to disable slow heuristics if they are causing you problems.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joxeankoret/diaphora/issues/210#issuecomment-785214999, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDD63E7KD3YGZPBPTVDTAUUZ7ANCNFSM4XEIOEDQ .

joxeankoret commented 3 years ago

:eyes: That makes absolutely no sense at all. Can you please share the samples so I can take a look?

smikkel997 commented 3 years ago

https://drive.google.com/drive/folders/1_sIcmf5EBsr3_NSQC--VBB54sHHyXAsi?usp=sharing i uploaded the two executables to this folder, tell me if you rather want the sqlite databases of them. (am using diaphora 1.2 and ida pro 7.0 btw, so maybe this happens because i use this outdated version)?

Op ma 1 mrt. 2021 om 16:57 schreef Joxean notifications@github.com:

👀 That makes absolutely no sense at all. Can you please share the samples so I can take a look?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joxeankoret/diaphora/issues/210#issuecomment-788060215, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDCZRAQGHRAREND3X7DTBO2NRANCNFSM4XEIOEDQ .

joxeankoret commented 3 years ago

I'm afraid that is a problem with IDA and, also, that it has little to do with the version (although IDA 7.0 is pretty old). I have downloaded and opened one of the binaries and IDA is still performing the initial auto-analysis ~2 hours after it, and so far discovered +20K functions. I'll tell you when it's done, but I'm afraid there is nothing I can done.

joxeankoret commented 3 years ago

So... it took around 3 hours just for IDA to finish the initial auto-analysis, and it found 52k functions. As I thought, I don't think there is anything I can do from Diaphora.

smikkel997 commented 3 years ago

do you have any clue how i could fix this (i do not have a version of ida higher than 7.2 but i'm using ida pro because this version contains a x86 decompiler), and do you happen to know whats causing this problem?

http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail Virus-free. www.avg.com http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Op di 2 mrt. 2021 om 13:19 schreef Joxean notifications@github.com:

So... it took around 3 hours just to analyse the first binary where IDA found 52k functions. As I thought, I don't think there is anything I can do from Diaphora.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/joxeankoret/diaphora/issues/210#issuecomment-788868530, or unsubscribe https://github.com/notifications/unsubscribe-auth/AR5NEDEWUAWGMFNXKT6ILR3TBTJUHANCNFSM4XEIOEDQ .

Hati- commented 3 years ago

I'm using IDA Pro 7.5 SP3 and Diaphora 2.0.3 and for me I was dealing with two very large SQLite databases; One database at 6.2 GB (~143k functions with debug symbols and took 2.5 hours to export) and another database at 6.3 GB (~184k functions without any debug symbols and took 3.5 hours to export).

When looking for partial matches in parallel everything timed out. Afterward it got stuck on Finding with heuristic 'Small names difference' for several hours, with the temporary SQLite file increasingly getting larger past 120 GB before I decided to kill it.

To circumvent this I decided to set TIMEOUT_LIMIT = 60 * 120 and comment out self.search_small_differences(self.partial_chooser) in diaphora.py before trying again. And it completed with a success! The diff was done after about 1.5 hours. Perhaps you can try the same?

Tips: to change the location of where SQLite stores its temporary files you have to change the effective TEMP or TMP environment variable.

joxeankoret commented 3 years ago

There are ways to disable the call to "search_small_differences()", to set a different timeout, etc... But they aren't widely know, as they require Diaphora scripting (yes, this is a thing). I think I can change some defaults like, for example, disable "Small Names Difference" for big database, or simply dropping it because it causes many performance issues.