Enhancement: Problem with Line Numbers (Decompile/Debug Instrument/Recompile Cycle)

[GoogleCodeExporter]

This request comes from lessons learned in Black Box testing.

When taking an APK though a Decompile/Debug Instrument/Recompile/Sign/Align 
cycle, we can perform an apk install and run the resulting binary. With 
NetBeans 6.8, we can even create a project and import the smali files.

However, its difficult to set breakpoints and debug the re-engineered APK. Its 
difficult because there is a mis-match in line numbers between the smali source 
files and and annotated lines in the source files. An example is shown in the 
attached file.

For example, in the attached image, source file lines run from 248 to 270 for 
the function. However, the annotated line numbers are 45 and 46. When trying to 
set a break point through the IDE, the IDE will try and set the breakpoint at 
source line and not the smali line. This usually causes the Dalvik VM to reject 
the breakpoint.

Would it be possible to enhance baksmali/smali to synchronize line numbers so 
breakpoints can be managed through an IDE? That is, if there is an annotation 
".line 248", have that annotation actually point to line 248 in the source code 
(currently, ".line 45" corresponds to line 253 in source code lines).

Original issue reported on code.google.com by noloa...@gmail.com on 29 Aug 2012 at 1:33

Attachments:

• smali-breakpoint.png

[GoogleCodeExporter]

Doesn't apktool already do this? It did at some point, at least.

In any case, I would be willing to accept a patch implementing this into either 
baksmali or smali. If you don't want to implement this yourself, I can probably 
implement it at some point. But it may or may not be in the near future :)

Original comment by jesusfreke@jesusfreke.com on 29 Aug 2012 at 1:57

• Changed state: Accepted

[GoogleCodeExporter]

> Doesn't apktool already do this?
I know [at least] two folks who claim it works (Brut.all and Dima). The screen 
capture came from APKTOOL following Brut.all's directions at 
http://code.google.com/p/android-apktool/wiki/SmaliDebugging and Dima's 
directions at 
http://d-kovalenko.blogspot.com/2012/08/debugging-smali-code-with-apk-tool-and.h
tml. The only difference between the two appears to be a wrapper shell script.

But I've never been able to get the stuff from APKTOOL to work under NetBeans 
(http://code.google.com/p/android-apktool/issues/detail?id=339). Eclipse is a 
lost cause at this point (remote debugging appears to be that difficult).

> a patch implementing this into either baksmali or smali.
I already started looking at the sources for low hanging fruit :). Would you 
have any suggestions? (I'm motivated since I have an immediate need).

Original comment by noloa...@gmail.com on 29 Aug 2012 at 2:26

[GoogleCodeExporter]

> a patch implementing this into either baksmali or smali
Forgive my ignorance.... What drives the ".line XXX" annotation? I can't seem 
to figure out the rule(s) when the annotation shows up.

It does not look like they are issued 1-for-1 for invoke-*, other keywords, or 
even things such as comparisons (i.e., an 'if' statement).

Lines appear to be sequential, but numbering can start over within a smali file 
and functions within the file. For example, both the class constructor (clinit) 
and instance constructor (init) claim to be at ".line 10" after the ".prologue" 
(in the same file).

The virtual functions following the two constructors (clinit and init) tend to 
move sequentially without restarting. The second virtual function picks up 
where the first function left off during numbering.

Original comment by noloa...@gmail.com on 29 Aug 2012 at 3:45

[GoogleCodeExporter]

"But I've never been able to get the stuff from APKTOOL to work under NetBeans" 
- huh? From my understanding, it's just baksmali with some patches (that adds 
the correct .line directives, just like you mention). I.e. exactly what you are 
asking for.

---

The line information is normally used to associate a particular small section 
of bytecode with the line in the .java source file that it is associated with, 
for debugging. So for example, when the vm is executing the instruction at 
offset 0xbcd2, the debugger uses the line information to map that back to the 
equivalent line in the java file.

Since you actually want to debug the smali file itself, you need to generate 
new line data, such that each instruction is associated with the line it is at 
in the .smali file.

You could either implement this in baksmali or smali. In baksmali, you would 
ignore any existing line data (from the DebugInfoItem), and simply add a new 
.line directive before every instruction. The only tricky part would be 
figuring out the line number in the smali file. (i.e. .line 123 should be for 
an instruction on line 123, although .line 123 itself would be on an earlier 
line, because it comes before the instruction)

Alternately, you could implement this in smali, in which case you would discard 
any .line directives that are in the .smali file, and generate new line info in 
the DebugInfoItem.

Of the two, I think doing it in baksmali would be easier, because you don't 
have to deal with creating a new DebugInfoItem.

Original comment by jesusfreke@jesusfreke.com on 29 Aug 2012 at 6:37

[GoogleCodeExporter]

> huh? From my understanding, it's just baksmali with some patches
Well, I'm not sure what to say. I'm having problems and others are having 
problems. And some others are not having problems.

I can produce screen shots that show line numbering problems (i.e., the line 
numbers are not correct after running APKTOOL). I just ran APKTOOL on a simple 
CrackMe, and there are two distinct source file lines claiming to be ".line 10" 
(one is in a class constructor (clinit), and the other is in an instance 
constructor (init)).

Perhaps this is due my setup/environment: Mac OS X Lion and Mountain Lion, x64 
versus x86, Apple Java 6, real devices, etc.

> Since you actually want to debug the smali file itself, you need to
> generate new line data, such that each instruction is associated
> with the line it is at in the .smali file.
OK, thanks. This should be my last set of dumb questions. When I encounter the 
following:

255> .line XXX
256> #v1=(PosByte);
257> invoke-super {p0, p1}, 
Landroid/app/Activity;->onCreate(Landroid/os/Bundle;)V

From you ".line 123" example, XXX should be 257: ".line 257". Is this correct?

Is it OK to add a ".line XXX" preceding all statements which will be executed? 
In particular, the invoke-* directives?

Is it OK to add ".line XXX" before all functions/methods (including static and 
instance).

Original comment by noloa...@gmail.com on 29 Aug 2012 at 3:50

[GoogleCodeExporter]

>I can produce screen shots that show line numbering problems (i.e., the line 
numbers are not correct after running APKTOOL). I just ran APKTOOL on a simple 
CrackMe, and there are two distinct source file lines claiming to be ".line 10" 
(one is in a class constructor (clinit), and the other is in an instance 
constructor (init)).

Out of curiosity, can you pastebin a file showing the incorrect line numbering 
from apktool?

>From you ".line 123" example, XXX should be 257: ".line 257". Is this correct?

Yes.

Is it OK to add a ".line XXX" preceding all statements which will be executed? 
In particular, the invoke-* directives? Yes.

Yes. It doesn't matter what the instruction is. It can be associated with a 
line number.

Is it OK to add ".line XXX" before all functions/methods (including static and 
instance).

No. Line numbers are only associated with instructions, not methods.

Original comment by jesusfreke@jesusfreke.com on 29 Aug 2012 at 4:53

[GoogleCodeExporter]

> Out of curiosity, can you pastebin a file showing the incorrect line 
numbering from apktool?
Absolutely. Pastebin: http://pastebin.com/DBrrs0JA.

The APK and disassemble was created on a Windows 7 (x64) machine with Java 7. 
Notice both the class constuctor (clinit) and instance constructor (init) both 
claim to be ".line 18". If I do it on Mac OS X Lion with Apple Java 6, they 
both claim to be at ".line 10".

The program is a simple 'CrackMe' so its meant to be kicked around (and there 
is nothing proprietary). I would be happy to give you Eclipse sources (Exported 
Archive), a built APK (signed with a real key), APKTOOL output (including 
incorrect line numbers), and a re-engineered APK (signed with a Debug Key).

Original comment by noloa...@gmail.com on 29 Aug 2012 at 5:51

[GoogleCodeExporter]

The .line directives look consistent with what would normally be there, for a 
file that was compiled from java and then disassembled without any line number 
munging. 

Are you sure you using apktool's -d option, which does the line number munging? 
(as per http://code.google.com/p/android-apktool/wiki/SmaliDebugging)

Also, that's really strange that the .line numbers are different on 
windows/mac. Do you get the same behavior with "stock" baksmali?

Original comment by jesusfreke@jesusfreke.com on 29 Aug 2012 at 6:03

[GoogleCodeExporter]

> Are you sure you using apktool's -d option
Yes. A screen capture of the command line used: 
http://postimage.org/image/94p7cyab9/

> that's really strange that the .line numbers are different on windows/mac.
> Do you get the same behavior with "stock" baksmali?
To date, I have never used baksmali and smali directly. Could you cite a quick 
command? If so, I will check for you.

Original comment by noloa...@gmail.com on 29 Aug 2012 at 6:20

[GoogleCodeExporter]

> per http://code.google.com/p/android-apktool/wiki/SmaliDebugging
I just took some time to read comments on this page.

It appears APKTOOL worked around January, 2011
It appears NetBeans (6.x?) debugging broke around around March 2011 (or perhaps 
reports started trickling in for the next NetBeans)
It appears latest NetBeans (7.2?) was broke around September 2011
It appears `apktool d -d...` broke around March 2012

Original comment by noloa...@gmail.com on 29 Aug 2012 at 6:46

[GoogleCodeExporter]

For what it's worth, the new apktool versions that ibotpeaches has been 
releasing don't seem to have the extra line/local munging stuff that is talked 
about on http://code.google.com/p/android-apktool/wiki/SmaliDebugging

Original comment by jesusfreke@jesusfreke.com on 29 Aug 2012 at 6:51

[GoogleCodeExporter]

> the new apktool versions that ibotpeaches has been releasing don't seem to 
have the extra line/local munging
Ah, yes. I recall seeing that announcement on XDA Developers. 
http://forum.xda-developers.com/showthread.php?t=1755243 (my web cache was 
still warm).

APKTOOL (the base project) appears abandoned and some functionality is broken. 
Perhaps we need an APKTOOL-NG, where these next generation tools can be 
centrally located. I've seen at least two "new APKTOOLS", including Multitools 
by Gerald. Gerald hosts on a ad-laden site that performs web-page jacking, so I 
had no interest in trying to download his warez.

Original comment by noloa...@gmail.com on 29 Aug 2012 at 7:04

[GoogleCodeExporter]

Connor's (ibotpeaches) stuff - apktool-1.4.10 is broken too. It throws an 
exception on both Mac OS X/Lion (x64)/Java 6 and Windows 7 (x64)/Java 7.

Sigh.....

Original comment by noloa...@gmail.com on 29 Aug 2012 at 8:27

[GoogleCodeExporter]

I'm working on the problem now. I forgot to keep the dexlib changes during 
merges, and merging fixes from dexlib 1.2.0 isn't the easiest since a year of 
smali development has gone by.

Original comment by connor.tumbleson on 1 Sep 2012 at 2:08

[GoogleCodeExporter]

I wrote a program to loop over all smali files; delete the errant line numbers 
produced by APKTOOL; and add a [correct?] line number on all invoke-* methods. 
After a recompile/sign/align, there was still no joy.

I'm not sure what version of Baksmali/Smali that version of APKTOOL is using 
(APKTOOL 1.4.3). I am befuddled that me and my short list of friends are having 
debugging problems after re-engineering while many others are not.

Original comment by noloa...@gmail.com on 2 Sep 2012 at 3:11

[GoogleCodeExporter]

Why just invoke instructions? You seem to be preoccupied with them for some 
reason. As I had mentioned earlier: "simply add a new .line directive before 
every instruction.".

Original comment by jesusfreke@jesusfreke.com on 2 Sep 2012 at 3:16

[GoogleCodeExporter]

> Why just invoke instructions? You seem to be preoccupied with them for some 
reason.
Well, I was interested to see if the issue was incorrect line numbers or 
something else. Correct line numbers on invoke-* should have allowed me to rule 
out broken line numbers as the cause of the issues.

> simply add a new .line directive before every instruction.
Thats a lot more work, and would have been worth it if the test case above 
worked as expected. Its a lot more work because the dalvik instruction set is 
non-trivial: http://source.android.com/tech/dalvik/dalvik-bytecode.html.

If you think it will make a difference, I can add a more complete grammar to my 
simple parser.

I can set breakpoints from a non-re-engineered APK (i.e., a NetBeans or Eclipse 
project), so I'm beginning to think APKTOOLS is broken on the compilation side, 
too (or the assembler is broken).

Original comment by noloa...@gmail.com on 2 Sep 2012 at 3:50

[GoogleCodeExporter]

> Why just invoke instructions? You seem to be preoccupied with them for some 
reason.
Well, I was interested to see if the issue was incorrect line numbers or 
something else. Correct line numbers on invoke-* should have allowed me to rule 
out broken line numbers as the cause of the issues.

> simply add a new .line directive before every instruction.
Thats a lot more work, and would have been worth it if the test case above 
worked as expected. Its a lot more work because the dalvik instruction set is 
non-trivial: http://source.android.com/tech/dalvik/dalvik-bytecode.html.

If you think it will make a difference, I can add a more complete grammar to my 
simple parser.

I can set breakpoints from a non-re-engineered APK (i.e., a NetBeans or Eclipse 
project), so I'm beginning to think APKTOOLS is broken on the compilation side, 
too (or the assembler is broken).

Original comment by noloa...@gmail.com on 2 Sep 2012 at 3:50

[GoogleCodeExporter]

> Why just invoke instructions? 
OK, I added a ".line XXX" for everything that started with something other than 
".", "#". and ":". I think that got all instructions.

Here's what typical constructors look like after the line number fix:

# direct methods
.method static constructor <clinit>()V
    .locals 1

    .prologue
    .line 26
    const-class v0, Lcom/example/crackme/MainActivity;

    #v0=(Reference,Ljava/lang/Class;);
    .line 30
    invoke-virtual {v0}, Ljava/lang/Class;->desiredAssertionStatus()Z

    .line 33
    move-result v0

    #v0=(Boolean);
    .line 37
    if-nez v0, :cond_0

    .line 40
    const/4 v0, 0x1

    :goto_0
    .line 44
    sput-boolean v0, Lcom/example/crackme/MainActivity;->$assertionsDisabled:Z

    .line 47
    return-void

    :cond_0
    .line 51
    const/4 v0, 0x0

    #v0=(Null);
    .line 55
    goto :goto_0
.end method

.method public constructor <init>()V
    .locals 0

    .prologue
    .line 63
    invoke-direct {p0}, Landroid/app/Activity;-><init>()V

    #p0=(Reference,Lcom/example/crackme/MainActivity;);
    .line 67
    return-void
.end method

Below are the results on APKTOOL. I can share the C++ source code for 
fixing/adding lines so you can see how Smali handles it during assembly, if you 
are interested.

apktool-crackme$ ./apktool b -d crackme.out/
I: Checking whether sources has changed...
I: Smaling...
Exception in thread "main" java.lang.NullPointerException
    at org.jf.util.PathUtil.getRelativeFile(PathUtil.java:44)
    at org.jf.smali.smaliFlexLexer.getSourceName(smaliFlexLexer.java:2922)
    at org.antlr.runtime.CommonTokenStream.getSourceName(CommonTokenStream.java:345)
    at org.antlr.runtime.Parser.getSourceName(Parser.java:88)
    at org.jf.smali.smaliParser.getErrorHeader(smaliParser.java:358)
    at org.antlr.runtime.BaseRecognizer.displayRecognitionError(BaseRecognizer.java:192)
    at org.antlr.runtime.BaseRecognizer.reportError(BaseRecognizer.java:186)
    at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:606)
    at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
    at org.jf.smali.smaliParser.annotation(smaliParser.java:5907)
    at org.jf.smali.smaliParser.parameter_directive(smaliParser.java:7718)
    at org.jf.smali.smaliParser.statements_and_directives(smaliParser.java:1902)
    at org.jf.smali.smaliParser.method(smaliParser.java:1609)
    at org.jf.smali.smaliParser.smali_file(smaliParser.java:595)
    at brut.androlib.mod.SmaliMod.assembleSmaliFile(SmaliMod.java:71)
    at brut.androlib.src.DexFileBuilder.addSmaliFile(DexFileBuilder.java:43)
    at brut.androlib.src.DexFileBuilder.addSmaliFile(DexFileBuilder.java:33)
    at brut.androlib.src.SmaliBuilder.buildFile(SmaliBuilder.java:64)
    at brut.androlib.src.SmaliBuilder.build(SmaliBuilder.java:48)
    at brut.androlib.src.SmaliBuilder.build(SmaliBuilder.java:35)
    at brut.androlib.Androlib.buildSourcesSmali(Androlib.java:222)
    at brut.androlib.Androlib.buildSources(Androlib.java:179)
    at brut.androlib.Androlib.build(Androlib.java:170)
    at brut.androlib.Androlib.build(Androlib.java:154)
    at brut.apktool.Main.cmdBuild(Main.java:182)
    at brut.apktool.Main.main(Main.java:67)

Original comment by noloa...@gmail.com on 2 Sep 2012 at 4:31

[GoogleCodeExporter]

From, the stack trace, This appears to be an issue parsing a parameter 
annotation, that gets complicated later by a likely bug in reporting the error. 
I would guess the problem is likely that you added a .line directive in the 
middle of a parameter annotation.

(although, the NPE while attempting to report the original error does seem to 
be a bug)

Original comment by jesusfreke@jesusfreke.com on 2 Sep 2012 at 7:10

[GoogleCodeExporter]

I just used baksmali to dump a re-engineered APK (more correctly, classes.dex). 
The re-engineered APK was created form a "Release Build" CrackMe, and went 
through a decompile/debug instrument/fix line numbers/recompile/debug 
sign/align cycle.

Line numbers are totally borked - as if I never performed the "fix line number" 
step. This could be due to APKTOOL (1.4.3), Smali assembler (1.3.3), or 
Baksmali (1.3.3) disassembler. Apktools could be screwing up the fix-up, smali 
could be incorrectly assembling with line number information, or baksmali could 
be incorrectly disassembling.

I've tried this exercise countless times on Windows, Mac OS X, and Linux; using 
Java 6 and Java 7; and using various revisions of the tools.

I think its impossible to complete this task with the tools in their current 
state. And I think the folks who claim they are doing it (debugging a 
re-engineered APK) are full of shit.

Original comment by noloa...@gmail.com on 2 Sep 2012 at 10:28

[GoogleCodeExporter]

"as if I never performed the "fix line number" step"

From my discussions with iBotPeaches, it seems that apktool disregards any 
existing .line info when you reassemble with -d. It replaces it all with 
whatever it is that it generates. 

If you're wanting to add the debug info manually (or with your own tool, 
whatever), you should use smali/baksmali directly - otherwise, I can't help, 
because I'm not all that familiar with apktool.

Original comment by jesusfreke@jesusfreke.com on 2 Sep 2012 at 10:52

[GoogleCodeExporter]

> From my discussions with iBotPeaches, it seems that apktool disregards any
> existing .line info when you reassemble with -d. It replaces it all with
> whatever it is that it generates.
Connor is right.

I took the [apktool] re-engineered APK and (1) extracted classes.dex (line 
numbers were borked); (2) ran baksmali over it; (3) ran my program over *.smali 
files to fix line numbers (Connor has it from a private email); (4) ran smali 
over it; and (5) dumped it again with baksmali.

After baksmali/fix line info/smali/baksmali cycle, line numbers appears to be 
in tact.

The APK was installed via ADB (`adb install ...`) and run via launcher. 
Unfortunately, no joy under NetBeans. I need to verify NetBeans is trying to 
submit the breakpoint; try starting via `adb am ...`; or try something like a 
uses-permission for SET_DEBUG_APP. I'm about out of ideas.

Original comment by noloa...@gmail.com on 3 Sep 2012 at 8:04