pkumza / LibRadar

LibRadar - A detecting tool for 3rd-party libraries in Android apps.
Apache License 2.0
256 stars 51 forks source link

Dataset is not available #45

Closed yoavst closed 7 years ago

yoavst commented 7 years ago

Where is the full dataset stored?

pkumza commented 7 years ago

Highly suggest you to try LiteRadar, a lite version of LibRadar that with no compromise to library number.

Or try online detecting

As github only accepts files that less than 100MB, I could only upload a small set of data as dataset

The latest dataset(of more than 2,400,000 apps) has reached 4.4GB, that I can't afford a fast CDN. I've upload a 2.5GB dataset (of 1 million apps) to Baiduyun, but I am not sure about your network speed to China.

yoavst commented 7 years ago

Thank for your answer!

yoavst commented 7 years ago

Just a small question @pkumza , What is the difference between the full and lite one?

pkumza commented 7 years ago

Please check issue 1 of LiteRadar.😃

On 06/15/2017 19:29, Yoav Sternberg wrote:

Just a small question @pkumza , What is the difference between the full and lite one?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

yoavst commented 7 years ago

Sorry for interrupting you, but it seems almost impossible to download anything from Baidu if you are not Chinese. Can you please upload it to something more international?

Thanks again!

pkumza commented 7 years ago

Size-optimized dataset is available at https://www.dropbox.com/s/w31gig6msdo3cdy/dump-20170515-shrink.rdb.tar.gz?dl=0

See docs/QuickStart.md for detailed installation steps.

yoavst commented 7 years ago

I'm running LibRadar with the full dataset from dropbox and it returns empty data [].LiteRadar works. Any idea?

pkumza commented 7 years ago

Please check that if you are using the latest version first. The features I use is different in recent commits.

yoavst commented 7 years ago

Repo is up to date.

pkumza commented 7 years ago

Interesting... I've tried again and the shrink data works well.


As you are using Redis, you must have installed redis-cli. try it!

$ redis-cli
127.0.0.1:6379> select 2
OK
127.0.0.1:6379[2]> keys *
1) "un_ob_pn"
2) "feature_weight"
3) "un_ob_cnt"
4) "feature_cnt"

Please check if your output is just the same.

yoavst commented 7 years ago
127.0.0.1:6379[2]> keys *
1) "feature_cnt"
2) "feature_weight"
3) "un_ob_pn"
4) "un_ob_cnt"
pkumza commented 7 years ago

Oh yes, your redis works well too. =_=

pkumza commented 7 years ago

Sorry, I've no idea what happened to LibRadar.

I'll test the steps of my QuickStart.md in a purely new machine soon. And I'd better create a docker image for LibRadar latter.

yoavst commented 7 years ago

Thanks!

pkumza commented 7 years ago

I've checked a pure environment and it works well too.

I got a reason that Redis needs several seconds to listen a port as the database is large. How about checking DB settings in _settings.py and trying it again?


btw, my testing env is Ubuntu 16.04 64-bit

yoavst commented 7 years ago

The settings seems OK. I'm running it on latest OS X.

yoavst commented 7 years ago

Seems to be an OS X problem, works great on Ubuntu VM.

pkumza commented 7 years ago

LibRadar is deployed on Linux now. However it is always developed on OS X. 🙉 It works well on my Mac too. 🐵 There's gotta be something wrong with your macOS Env. 🙈

AlexKoshulyan commented 5 years ago

The link https://www.dropbox.com/s/w31gig6msdo3cdy/dump-20170515-shrink.rdb.tar.gz?dl=0 is broken by now.

pkumza commented 5 years ago

@AlexKoshulyan Yes, it is broken. Could you have a try for https://www.dropbox.com/s/ljtzw74twt8xgy6/d.tar.gz?dl=0 ?

AlexKoshulyan commented 5 years ago

Fine