Closed yogiam closed 11 years ago
Can you please send me the apk file to Steven.Arzt@cased.de? I'll then have a look at it.
I dont have the exact same file, but I tested another file and found similar issue. Will mail that file to you.
Log
D:\Tests>java -Xms512m -jar soot.jar -allow-phantom-refs -src-prec apk -android -jars C:/android/sdk/platforms -outjar -d errfiles\Kitchen Timer.apk.jar -proces s-dir "errfiles\Kitchen Timer.apk" Soot started on Tue Oct 08 18:48:05 IST 2013 Using 'C:/android/sdk/platforms\android-3\android.jar' as android.jar Warning: Timer.apk.jar is a phantom class! Warning: android.accounts.Account is a phantom class! Warning: android.accounts.AccountManager is a phantom class! Transforming com.example.android.service.R$string... Transforming com.example.android.service.KitchenTimerService$1... Transforming com.example.android.service.R$drawable... Transforming com.example.android.service.KitchenTimerService$KitchenTimerBinder. .. Transforming com.example.android.service.R$id... Transforming com.example.android.service.R$raw... Transforming com.example.android.service.vew2... Transforming com.example.android.service.R$attr... Transforming com.example.android.service.Main$1... Transforming com.example.android.service.R$layout... Transforming com.example.android.service.KitchenTimerService... Transforming com.example.android.service.R... Transforming com.example.android.service.vew2$1... Transforming com.example.android.service.Main... Transforming Timer.apk.jar... Writing to com\example\android\service\R$string.class Jasmin:1: Warning - Syntax error. .source Kitchen Timer.apk ^ Jasmin:1: Error - Couldn't repair and continue parse. .source Kitchen Timer.apk ^ Jasmin: Found 2 errors Writing to com\example\android\service\KitchenTimerService$1.class Jasmin:1: Warning - Syntax error. .source Kitchen Timer.apk ^ Jasmin:1: Error - Couldn't repair and continue parse. .source Kitchen Timer.apk ^ Jasmin: Found 2 errors Writing to com\example\android\service\KitchenTimerService$KitchenTimerBinder.cl ass Jasmin:1: Warning - Syntax error. .source Kitchen Timer.apk ^ Jasmin:1: Error - Couldn't repair and continue parse. .source Kitchen Timer.apk ^ Jasmin: Found 2 errors Writing to com\example\android\service\R$id.class Jasmin:1: Error - Couldn't repair and continue parse. .source Kitchen Timer.apk ^ Jasmin: Found 2 errors Writing to Timer\apk\jar.class Soot finished on Tue Oct 08 18:48:07 IST 2013 Soot has run for 0 min. 1 sec.
D:\Tests>
Attached APK file. Password is 123.
http://www.flickr.com/photos/yogiam/
On Tue, Oct 8, 2013 at 6:47 PM, Steven Arzt notifications@github.comwrote:
Can you please send me the apk file to Steven.Arzt@cased.de? I'll then have a look at it.
— Reply to this email directly or view it on GitHubhttps://github.com/Sable/soot/issues/131#issuecomment-25888469 .
Filename is the issue. If I remove the space in file name, e.g make "Kitchen Timer.apk" to "KitchenTimer.apk", everything works fine.
The problem is the source file tag associated with a class when we fill it with the name of the APK file. Fixed in commit d2d4f65ab3b5d01d7cfc0ce93d3e7f23505d65e0
Please confirm the fix and close the issue if it works.
Will test with the nightly build and confirm.
http://www.flickr.com/photos/yogiam/
On Wed, Oct 9, 2013 at 1:45 PM, Steven Arzt notifications@github.comwrote:
The problem is the source file tag associated with a class when we fill it with the name of the APK file. Fixed in commit d2d4f65https://github.com/Sable/soot/commit/d2d4f65ab3b5d01d7cfc0ce93d3e7f23505d65e0
— Reply to this email directly or view it on GitHubhttps://github.com/Sable/soot/issues/131#issuecomment-25953678 .
Tested. Fixed.
Hi Steven,
Same issue arises with another file. Log below. Will email the file to you.
D:\Tests>java -Xms512m -jar soot.jar -allow-phantom-refs -src-prec apk -android -jars C:/android/sdk/platforms -outjar -d errfiles\appinventor.ai_T10D78.ZombieH ighway-1-1.7.apk.jar -process-dir errfiles\appinventor.ai_T10D78.ZombieHighway_1 _1.7.apk > out.txt Jasmin:20: Warning - Syntax error.
^ Jasmin:20: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:19: Warning - Syntax error.
^ Jasmin:19: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:20: Warning - Syntax error.
^ Jasmin:20: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:25: Warning - Syntax error.
^ Jasmin:25: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:22: Warning - Syntax error.
^ Jasmin:22: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:19: Warning - Syntax error.
^ Jasmin:19: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:21: Warning - Syntax error.
^ Jasmin:21: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:23: Warning - Syntax error.
^ Jasmin:23: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:23: Warning - Syntax error.
^ Jasmin:23: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:22: Warning - Syntax error.
^ Jasmin:22: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:26: Warning - Syntax error.
^ Jasmin:26: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:22: Warning - Syntax error.
^ Jasmin:22: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:22: Warning - Syntax error.
^ Jasmin:22: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:21: Warning - Syntax error.
^ Jasmin:21: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:24: Warning - Syntax error.
^ Jasmin:24: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:19: Warning - Syntax error.
^ Jasmin:19: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:24: Warning - Syntax error.
^ Jasmin:24: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:19: Warning - Syntax error.
^ Jasmin:19: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:23: Warning - Syntax error.
^ Jasmin:23: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:24: Warning - Syntax error.
^ Jasmin:24: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:22: Warning - Syntax error.
^ Jasmin:22: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:25: Warning - Syntax error.
^ Jasmin:25: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:19: Warning - Syntax error.
^ Jasmin:19: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:24: Warning - Syntax error.
^ Jasmin:24: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Jasmin:21: Warning - Syntax error.
^ Jasmin:21: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors
D:\Tests>
I cannot reproduce this issue. Are you sure that you are using the newest version of Soot?
Tested with the nightly build. Attached zip file with apk and complete log. Search for "jasmin" in the logs. Password is 123.
http://www.flickr.com/photos/yogiam/
On Thu, Oct 10, 2013 at 8:30 PM, Steven Arzt notifications@github.comwrote:
I cannot reproduce this issue. Are you sure that you are using the newest version of Soot?
— Reply to this email directly or view it on GitHubhttps://github.com/Sable/soot/issues/131#issuecomment-26061027 .
The issue is now reproducible for some reason. I'll have a look at it.
@Alexandre-Bartel This error is caused by Jasmin complaining about an annotation originally read from a DEX file. You said something about this being work in progress on the phone today because of Jasmin and Dexpler using different representations. I think the following Jasmin code is the problem, can you confirm this?
.runtime_visible_annotation .annotation "Lcom/google/devtools/simple/runtime/annotations/UsesPermissions;" .elem .str_kind "permissionNames" "android.permission.READ_CONTACTS" .end .annotation .end .annotation_attr
To me, this simple looks as if the L in the beginning and the semicolon at then end are wrong since Jasmin does not use this format anywhere else, superclasses etc. are just of the format "a/b/c".
@StevenArzt Right, at the moment annotations read from Dalvik cause problem when converted to Jasmin. I'm working on this.
@StevenArzt This bug should have been fixed in commit a6d344200dbeb2fc64e9367f3efb2365fe6708ff. Could you confirm and send me the apk for reference?
@Alexandre-Bartel It seems to work now, thanks. I have sent you the APK file via e-mail
@yogiam Can you confirm the fix as well and close the issue?
OK.
http://www.flickr.com/photos/yogiam/
On Sat, Oct 19, 2013 at 5:39 PM, Steven Arzt notifications@github.comwrote:
@Alexandre-Bartel https://github.com/Alexandre-Bartel It seems to work now, thanks. I have sent you the APK file via e-mail
@yogiam https://github.com/yogiam Can you confirm the fix as well and close the issue?
— Reply to this email directly or view it on GitHubhttps://github.com/Sable/soot/issues/131#issuecomment-26648848 .
Tested with nightly build. Issue still exists. Attached zip file(password 123) with apk and log files.
http://www.flickr.com/photos/yogiam/
On Mon, Oct 21, 2013 at 11:52 AM, Yogesh A. Mujumdar < yogesh.mujumdar@gmail.com> wrote:
OK.
- yam
http://www.flickr.com/photos/yogiam/
On Sat, Oct 19, 2013 at 5:39 PM, Steven Arzt notifications@github.comwrote:
@Alexandre-Bartel https://github.com/Alexandre-Bartel It seems to work now, thanks. I have sent you the APK file via e-mail
@yogiam https://github.com/yogiam Can you confirm the fix as well and close the issue?
— Reply to this email directly or view it on GitHubhttps://github.com/Sable/soot/issues/131#issuecomment-26648848 .
This is a different issue than the one discussed above. In this case, Jasmin fails to write out the .class file because of a class named "is" which is a reserved token in Jasmin. The offending line seems to be:
.interface public abstract is
@ericbodden Could you please have a look at the issue?
Steven can you please attach the offending .jasmin file?
Cheers, Eric
On 21.10.2013, at 14:51, Steven Arzt notifications@github.com wrote:
This is a different issue than the one discussed above. In this case, Jasmin fails to write out the .class file because of a class named "is" which is a reserved token in Jasmin. The offending line seems to be:
.interface public abstract is
@ericbodden Could you please have a look at the issue?
— Reply to this email directly or view it on GitHub.
Note that this issue also occurs for Jasmin instructions such as
"new is"
jasmin/ReservedWords.java tells us that this can happen with the following reserved keywords: 89 // reserved_words used in Jasmin directives 90 reserved_words.put("from", new Symbol(sym.FROM)); 91 reserved_words.put("method", new Symbol(sym.METHOD)); 92 reserved_words.put("to", new Symbol(sym.TO)); 93 reserved_words.put("is", new Symbol(sym.IS)); 94 reserved_words.put("using", new Symbol(sym.USING)); 95 96 // Special-case instructions 97 reserved_words.put("tableswitch", new Symbol(sym.TABLESWITCH)); 98 reserved_words.put("lookupswitch", new Symbol(sym.LOOKUPSWITCH)); 99 reserved_words.put("default", new Symbol(sym.DEFAULT));
On Mon, 2013-10-21 at 05:51 -0700, Steven Arzt wrote:
This is a different issue than the one discussed above. In this case, Jasmin fails to write out the .class file because of a class named "is" which is a reserved token in Jasmin. The offending line seems to be:
.interface public abstract is
@ericbodden Could you please have a look at the issue?
Reply to this email directly or view it on GitHub: https://github.com/Sable/soot/issues/131#issuecomment-26714039
.source 002f537027830303e2205dd0a6106cb1b79fa704(AnserverBot)(1).apk
.interface public abstract is
.super java/lang/Object
.method public abstract a(Lir;Liu;)V
.end method
Sorry, that was the wrong button, I did not intend to close the issue...
Thanks for the info Alex. I guess we require some escaping there (or a stateful lexer). I am surprised that this did not occur before, actually. I guess the only thing that saved us so far is that class names typically start with an uppercase letter.
Eric
On 21.10.2013, at 15:01, Alexandre-Bartel notifications@github.com wrote:
Note that this issue also occurs for Jasmin instructions such as
"new is"
jasmin/ReservedWords.java tells us that this can happen with the following reserved keywords: 89 // reserved_words used in Jasmin directives 90 reserved_words.put("from", new Symbol(sym.FROM)); 91 reserved_words.put("method", new Symbol(sym.METHOD)); 92 reserved_words.put("to", new Symbol(sym.TO)); 93 reserved_words.put("is", new Symbol(sym.IS)); 94 reserved_words.put("using", new Symbol(sym.USING)); 95 96 // Special-case instructions 97 reserved_words.put("tableswitch", new Symbol(sym.TABLESWITCH)); 98 reserved_words.put("lookupswitch", new Symbol(sym.LOOKUPSWITCH)); 99 reserved_words.put("default", new Symbol(sym.DEFAULT));
On Mon, 2013-10-21 at 05:51 -0700, Steven Arzt wrote:
This is a different issue than the one discussed above. In this case, Jasmin fails to write out the .class file because of a class named "is" which is a reserved token in Jasmin. The offending line seems to be:
.interface public abstract is
@ericbodden Could you please have a look at the issue?
Reply to this email directly or view it on GitHub: https://github.com/Sable/soot/issues/131#issuecomment-26714039 — Reply to this email directly or view it on GitHub.
This problem is introduced by obfuscation tools such as proguard. This kind of tool renames class names to a, b, c, ... and if there are enough classes to aa, bb, ..., is, it, ...
On Mon, 2013-10-21 at 06:11 -0700, Eric Bodden wrote:
Thanks for the info Alex. I guess we require some escaping there (or a stateful lexer). I am surprised that this did not occur before, actually. I guess the only thing that saved us so far is that class names typically start with an uppercase letter.
Eric
On 21.10.2013, at 15:01, Alexandre-Bartel notifications@github.com wrote:
Note that this issue also occurs for Jasmin instructions such as
"new is"
jasmin/ReservedWords.java tells us that this can happen with the following reserved keywords: 89 // reserved_words used in Jasmin directives 90 reserved_words.put("from", new Symbol(sym.FROM)); 91 reserved_words.put("method", new Symbol(sym.METHOD)); 92 reserved_words.put("to", new Symbol(sym.TO)); 93 reserved_words.put("is", new Symbol(sym.IS)); 94 reserved_words.put("using", new Symbol(sym.USING)); 95 96 // Special-case instructions 97 reserved_words.put("tableswitch", new Symbol(sym.TABLESWITCH)); 98 reserved_words.put("lookupswitch", new Symbol(sym.LOOKUPSWITCH)); 99 reserved_words.put("default", new Symbol(sym.DEFAULT));
On Mon, 2013-10-21 at 05:51 -0700, Steven Arzt wrote:
This is a different issue than the one discussed above. In this case, Jasmin fails to write out the .class file because of a class named "is" which is a reserved token in Jasmin. The offending line seems to be:
.interface public abstract is
@ericbodden Could you please have a look at the issue?
Reply to this email directly or view it on GitHub: https://github.com/Sable/soot/issues/131#issuecomment-26714039 — Reply to this email directly or view it on GitHub.
Reply to this email directly or view it on GitHub: https://github.com/Sable/soot/issues/131#issuecomment-26715293
The keywords is, from, to are apparently only used in the following rule...
//
// .var
It seems to me like code matching this rule is never, however, generated by Soot. I wondr what it is even good for. Maybe an annotation of variable types?
Anyway, I would suggest to simply rename the keywords to something longer that obfuscators are less likely to generate.
Another problem with the code that Steven sent seems to be the source annotation: .source 002f537027830303e2205dd0a6106cb1b79fa704(AnserverBot)(1).apk This currently does not parse either as far as I can see. Is Dexpler generating this annotation?
This annotation is generated out of the file name of the APK file which has such a weird name. There is already some hacking in Soot to sanitize the APK file name to avoid issues with Jasmin, so if we need to exclude (i.e. replace) more characters here, just let me know and I'll add them to the substitution list. The idea is to give the user some hint as to where the source originally came from - though this requires some character substitutions.
Well, currently I just get:
jas.jasError: Badly formatted number
You can test this yourself by running jasmin.Main with the parameter is.jasmin where is.jasmin is a file containing the Jasmin code.
Eric
On 21.10.2013, at 16:44, Steven Arzt notifications@github.com wrote:
This annotation is generated out of the file name of the APK file which has such a weird name. There is already some hacking in Soot to sanitize the APK file name to avoid issues with Jasmin, so if we need to exclude (i.e. replace) more characters here, just let me know and I'll add them to the substitution list. The idea is to give the user some hint as to where the source originally came from - though this requires some character substitutions.
— Reply to this email directly or view it on GitHub.
Yes, Dexpler is generating this annotation. I have updated the code to make sure the name always starts with a letter.
@ericbodden I would suggest adding a dot '.' before the keywords if that still make sense.
Appropriate change pushed to jasmin/develop.
There still seems to be something wrong when trying to process the sample APK. The failing class is called fs. Jasmin code:
.source dalvik_source_002f537027830303e2205dd0a6106cb1b79fa704(AnserverBot)(1).apk
.interface public abstract fs
.super java/lang/Object
.method public abstract a(Ljava/lang/Object;Ljava/lang/String;)Ljava/io/InputStream;
.end method
.method public abstract a(Ljava/lang/String;)V
.end method
Error message:
Jasmin:5: Warning - Syntax error.
.method
^
Jasmin:5: Error - Couldn't repair and continue parse.
.method
^
Jasmin: Found 2 errors
Has some of the reserved word renaming maybe affected fixed parts of the Jasmin language?
This is weird. If I revert the previous change then Jasmin is running out of memory on this input! WTH!?
The problem is that the '.method' keyword already exists:
43 reserved_words.put(".method", new Symbol(sym.DMETHOD)); ... 91 reserved_words.put(".method", new Symbol(sym.METHOD));
On Wed, 2013-10-23 at 01:31 -0700, Eric Bodden wrote:
This is weird. If I revert the previous change then Jasmin is running out of memory on this input! WTH!?
Reply to this email directly or view it on GitHub: https://github.com/Sable/soot/issues/131#issuecomment-26888280
By the way the keywords FROM TO and USING are also used in the .catch rule:
src/jasmin/parser.cup
541 // .catch
I have reverted the previous change but I am also not seeing a solution right now. I don't understand why simply changing or removing the keyword definition is breaking the parser. The scanner is hand written, though, which might explain things. It's probably badly designed and comes with lots of internal constraints. I don't have much time to look into this any further right now. Any volunteers?
@Alexandre-Bartel even if I just rename "is" to ".is" in the reserved keywords or if I remove the line entirely does the parser run out of memory...
@ericbodden Renanming "is" to ".is" is working fine on my side.
$ java -classpath /home/alex/src/sable-jasmin.git/lib/jasminclasses-2.2.1.jar:/home/alex/src/sable-polyglot.git/lib/java_cup.jar jasmin.Main fs.jasmin $ file fs.class fs.class: compiled Java class data, version 46.0 (Java 1.2)
Thanks Alex. I pushed another change that just modifies "is" to be ".is" instead: c90a67d9aeb9f62ce94cdec98fb9e012f3f578d1
I guess something must be wrong with my Eclipse setup. I also cannot reproduce the memory issue on the command line.
Note that this does not fix the problem entirely. "to" is still a keyword used for traps.
The fix seems to resolve the issue with the APK.
@yogiam Can you confirm this?
I pushed another change that should take care of all Jasmin keywords posing problem (the ones not starting with a dot): https://github.com/Sable/jasmin/commit/254e08fc4e40a58fdae3e52a09e3343753e521b8
@StevenArzt @yogiam Could you confirm that the second app. is working with this?
Will test and get back.
http://www.flickr.com/photos/yogiam/
On Thu, Oct 24, 2013 at 10:20 PM, Alexandre-Bartel <notifications@github.com
wrote:
I pushed another change that should take care of all Jasmin keywords posing problem (the ones not starting with a dot): Sable/jasmin@254e08fhttps://github.com/Sable/jasmin/commit/254e08fc4e40a58fdae3e52a09e3343753e521b8
@StevenArzt https://github.com/StevenArzt @yoglam Could you confirm that the second app. is working with this?
— Reply to this email directly or view it on GitHubhttps://github.com/Sable/soot/issues/131#issuecomment-27009460 .
Tested. I dont get the warning anymore. Closing.
Thanks.
Command: java -Xms512m -jar soot.jar -allow-phantom-refs -src-prec apk -android-jars C:/android/sdk/platforms -outjar -process-dir a.apk
Error: r.class Writing to kawa\lang\Pattern.class Writing to com\google\devtools\simple\runtime\components\android\ContactPicker.c lass Jasmin:20: Warning - Syntax error.
^ Jasmin:20: Error - Couldn't repair and continue parse.
^ Jasmin: Found 2 errors Writing to com\google\devtools\simple\runtime\variants\IntegerVariant.class Writing to com\google\devtools\simple\runtime\components\android\ListPickerActiv ity.class Writing to twitter4j\TwitterStream$StreamHandlingThread.class
I also get the error at different points while writing different classes. below is the list.
Other places where it gives error: Jasmin:20: Warning - Syntax error. Jasmin:20: Error - Couldn't repair and continue parse. Jasmin: Found 2 errors Jasmin:19: Warning - Syntax error. Jasmin:19: Error - Couldn't repair and continue parse. Jasmin: Found 2 errors Jasmin:25: Warning - Syntax error. Jasmin:25: Error - Couldn't repair and continue parse. Jasmin: Found 2 errors Jasmin:22: Warning - Syntax error. Jasmin:22: Error - Couldn't repair and continue parse. Jasmin: Found 2 errors Jasmin:21: Warning - Syntax error. Jasmin:21: Error - Couldn't repair and continue parse. Jasmin: Found 2 errors Jasmin:23: Warning - Syntax error. Jasmin:23: Error - Couldn't repair and continue parse. Jasmin: Found 2 errors Jasmin:26: Warning - Syntax error. Jasmin:26: Error - Couldn't repair and continue parse. Jasmin: Found 2 errors Jasmin:24: Warning - Syntax error. Jasmin:24: Error - Couldn't repair and continue parse. Jasmin: Found 2 errors