search

EnSoftCorp / android-essentials-toolbox

A set of building blocks for analyzing Android apps with Atlas
https://ensoftcorp.github.io/android-essentials-toolbox
MIT License
14 stars 5 forks source link

Update mappings for recent Android APIs #3

Closed louison closed 4 years ago

louison commented 7 years ago

Hi, I just discovered you project a couple of days ago. I'm currently working on a project that involves a Permission-method mapping. I'va been so happy to find your 'all-mappings' directory. It's very useful for me. Would you be able to update this directory with the latest android APIs ? (22, 23, 24, 25) If not, is there an easy way to do it myself and contribute ?

Thank you in advance do your help

benjholla commented 7 years ago

The permission mappings are generated with the PScout tool by the University of Toronto (http://pscout.csl.toronto.edu/downloads.php). They've provided some mappings which are included in this tool and in some cases we've run the tool to fill in a few of the mappings.

If you are interested in contributing to this project that would be awesome! There are two main challenges.

1) Running PScout requires downloading the full Android source for the particular API you are interested in. For consistency I have chosen to use the last available version of the vanilla stock OS source for each Android API version. I just made another repository public where we performed these steps for the versions we performed (it can be a bit of tedious process). In general, you figure out which version you want to build, read Google's documentation for how to build that version (i.e. setup a VM with the specific OS and toolchain), built the release, then run PScout and fix any issues PScout has in producing the mapping. Annoyingly the Android build setup changes slightly from version to version and you can't just blindly run PScout (Example: https://github.com/EnSoftCorp/AnalyzableAndroid/tree/master/Android22#pscout). I forked and fixed a bug in PScout for APIs 21+ here but the merge request was never accepted (https://github.com/benjholla/pscout). There was a major change around API 23 or 24 that prevented me from running pscout again but I forgot what it was off the top of my head. I did see some activity on a PScout II tool, but I never saw anything officially released and no new updated mappings by the original researchers.

2) Some much easier that I've only seen done in this tool is to recover the Permission Group to Permission and Permission Level to Permission mappings. Fortunately that is much much easier and just requires parsing XML. I have a tool that does that in this repo already and documentation here: https://github.com/EnSoftCorp/android-essentials-toolbox#maintenance

In short if you have time and are willing to dig into it, I'd be happy to help out. I'm a little rusty on the process and a bit side tracked with other projects, but I'd really like to see as many of the permission mappings completed as we can get :)

benjholla commented 7 years ago

By the way, now that I looked at the all-mappings directory I can't remember exactly why that is there...the directory that contains the mappings used by the tool is in the plugin project com.ensoftcorp.open.android.essentials/src/com/ensoftcorp/open/android/essentials/permissions/mappings, which does include API 22.

louison commented 7 years ago

Thank you for your answer ! It clears up a lot of things. I will try do dig a bit on the subject, but I really have to move forward on my main project (which requires yours) First, let's try to build android API 25 :)

louison commented 7 years ago

So you advise me to use Pscout or Pscout 2 ? I don't really understand what's going on with this tool.

benjholla commented 7 years ago

I've only ever used the original PScout implementation. I saw updates to the PScout website that included PScout II, but I never saw any new published papers about it so I'm not really sure the state of the tools. Looking at the repository for PScout II it looks like the activity on the project has stalled out. There is an open issue for investigating if it works on Android Marshmallow (API 23). Probably we should be talking to the original authors (I'll try to reach out to them today).

It's worth reading through their paper quickly to get an idea of what the tool is doing and what problems the mapping may have (its not perfect). The original PScout tool documentation was enough that I was able to fire up the tool and get some results after a day or so of fiddling around and the AnalyzableAndroid repo I linked earlier has all of my step by step notes for the previous version where I ran the tool.

Just curious, are you planning to use this tool with Atlas or are you using a different framework and just using the mappings? We investigated creating the permission mappings ourselves when we first began this work, but came across Berkeley's Android Permissions Demystified paper which first recovered the mappings dynamically and was then later succeeded by a static analysis in University of Toronto's PScout paper. In the end we never attempted recovering the mappings ourselves aside from running existing tools.

louison commented 7 years ago

Oh, I don't even know Atlas, thank you for the reference. For now, I'm trying to see how the state of the art looks like. I'm aware of "Android Permissions Demystified". But it seems like all this work has been abandoned. So my goal is to try something on top of PScout or maybe a custom solution inspired from Pscout. But I'm not sure where to start. Moreover, documentation on Pscout seems not to cover new Android permission system.

I will try to run PScout on a 7.1 Android build by the end of the week, just to see the behavior of the tool.

I'm interested with the mappings because I'm currently working on some security issues on Android OS. So I've built my own custom tool which is able to check some security vulnerabilities. But I need to go forward and see if specific method calls in an Android apk bytecode are bound to permissions.

benjholla commented 7 years ago

In that case you might want to look at the tutorial for using this toolbox with Atlas to audit permission usage.

You are right the "Android Permissions Demystified" is outdated and PScout is the current state of the art for recovering permission mappings as far as I am aware of.

benjholla commented 7 years ago

Ok some good news.

The advising professor behind PScout responded and informed me that while the students behind PScout have graduated or moved on, there is a group in Germany that did produce a newer version of a tool that makes some improvements over PScout (see http://axplorer.org).

They do not appear to be sharing their code. However, mappings are available for more recent API levels though, which includes API 23. I'll take a look at parsing their API 23 results and integrating into our standardized format.

APIs 24 and 25 were not included so I'll try to reach out to them to and see if they have the results or if they would be willing to share the source code to their tool.

benjholla commented 7 years ago

Commit d3316a7dc9949a1180a74ea98aa341b2d90bf900 adds support API 23. I am making a new issue to update permission level and permission group mappings as well as the permission descriptions from the Google documentation.

louison commented 7 years ago

Hi !, Very cool for the news, so we know that Pscout will not be maintained anymore ! I already saw axplorer, I'm sorry, I totally forgot to give you the link. What I know so far from Axplorer :

But, you're right, it's a good idea to reach them, I should do that more often.

louison commented 7 years ago

My bad, ContactsContract is a content provider, so the permission is mapped with the associated CONTENT_URI, which is documented in the content-provider mapping by axplorer

benjholla commented 7 years ago

I think the mappings are complete at least from what I saw, just they were split into 3 categories. I concatenated the framework-map-23.txt and the sdk-map-23.txt files to produce the API23.xml mapping.

Note that the API23.xml mapping does not include the content provider protected permissions in cp-map-23.txt because there are no corresponding methods. I believe the previous works missed these as well.

I reached out to the group in Germany to ask if they would share their code or release results for the updated APIs, but they have not responded yet. If they are not willing to do either I am not sure what we will be able to do at this time, without a lot of work.

louison commented 7 years ago

I agree, let me know if you have some answers :)

benjholla commented 7 years ago

Eric from Saarland University who maintains the Axplorer work responded and said he intends to upload updated mappings for APIs 24 and 25 in ~2 weeks if he can find the time. As for releasing his source code, he said it is not ready for release yet, but would like to release it eventually.