search

upanya / android-marketplace-crawler

Automatically exported from code.google.com/p/android-marketplace-crawler
0 stars 0 forks source link

RuntimeException: Response code = 509, msg = unknown #4

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. I have problems with any instruction, no one works
2. In this case I introduce
java -jar crawler.jar -pname pak.txt
3.

I get like answer
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.R
untimeException: Response code = 509, msg = unknown
        at java.util.concurrent.FutureTask$Sync.innerGet(Unknown Source)
        at java.util.concurrent.FutureTask.get(Unknown Source)
        at com.marketplace.Main.execute(Unknown Source)
        at com.marketplace.Main.resolveArgs(Unknown Source)
        at com.marketplace.Main.main(Unknown Source)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Response code
 = 509, msg = unknown
        at com.gc.android.market.api.MarketSession.executeRawHttpQuery(MarketSes
sion.java:306)
        at com.gc.android.market.api.MarketSession.executeProtobuf(MarketSession
.java:252)
        at com.gc.android.market.api.MarketSession.flush(MarketSession.java:221)

        at com.marketplace.io.Fetcher.getAppsByPackageName(Unknown Source)
        at com.marketplace.service.PackageThread.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.RuntimeException: Response code = 509, msg = unknown
        at com.gc.android.market.api.MarketSession.executeRawHttpQuery(MarketSes
sion.java:287)
        ... 10 more
Oct 09, 2011 1:20:42 PM com.marketplace.io.Fetcher getAppsByPackageName

What is the expected output? What do you see instead?
The apps(?)

What version of the product are you using? On what operating system?
The last one, so far. Windows 7

Please provide any additional information below.

Original issue reported on code.google.com by queenscr...@gmail.com on 9 Oct 2011 at 5:24

GoogleCodeExporter commented 8 years ago 
The only times you see a 509 Response if you abuse/overuse the API.

Original comment by raunakgu...@gmail.com on 10 Oct 2011 at 3:16

GoogleCodeExporter commented 8 years ago 
I have not been able to run once this application, I do not know what I am 
doing wrong, I hope you can help me.

Ammmhh I wonder if it is related to this error
org.apache.http.conn.HttpHostConnectException: Connection to http://0.0.0.0:3000
 refused

Do you know what can be happenning here?

Ok, so far, I have run the server, is this ok? ---->

++++++++
C:\Android-Market-Crawler\server>rake db:drop
(in C:/Android-Market-Crawler/server)

C:\Android-Market-Crawler\server>rake db:drop
(in C:/Android-Market-Crawler/server)
Couldn't drop Marketplace_development : #<Mysql2::Error: Unknown database 'marke
tplace_development'>

C:\Android-Market-Crawler\server>rake db:create
(in C:/Android-Market-Crawler/server)
Marketplace_test already exists

C:\Android-Market-Crawler\server>rake db:migrate
(in C:/Android-Market-Crawler/server)
==  CreateApps: migrating =====================================================
-- create_table(:apps)
   -> 0.0960s
-- add_index(:apps, :packagename, {:unique=>true})
   -> 0.1350s
==  CreateApps: migrated (0.2330s) ============================================

==  CreateRatings: migrating ==================================================
-- create_table(:ratings)
   -> 0.1000s
-- add_index(:ratings, :app_id, {:unique=>true})
   -> 0.1490s
==  CreateRatings: migrated (0.2520s) =========================================

==  CreateVestalVersions: migrating ===========================================
-- create_table(:versions)
   -> 0.0720s
-- change_table(:versions)
   -> 0.7860s
==  CreateVestalVersions: migrated (0.8620s) ==================================

==  CreateVisuals: migrating ==================================================
-- create_table(:visuals)
   -> 0.0700s
-- add_index(:visuals, :app_id)
   -> 0.1180s
==  CreateVisuals: migrated (0.1910s) =========================================

==  AddAttachmentImageToVisual: migrating =====================================
-- add_column(:visuals, :image_file_name, :string)
   -> 0.1990s
-- add_column(:visuals, :image_content_type, :string)
   -> 0.1650s
-- add_column(:visuals, :image_file_size, :integer)
   -> 0.1740s
-- add_column(:visuals, :image_updated_at, :datetime)
   -> 0.2070s
==  AddAttachmentImageToVisual: migrated (0.7500s) ============================

==  CreateTargets: migrating ==================================================
-- create_table(:targets)
   -> 0.0730s
-- add_index(:targets, :name, {:unique=>true})
   -> 0.1260s
==  CreateTargets: migrated (0.2010s) =========================================

==  CreateAppTargets: migrating ===============================================
-- create_table(:app_targets)
   -> 0.0720s
-- add_index(:app_targets, :app_id)
   -> 0.1590s
-- add_index(:app_targets, :target_id)
   -> 0.1260s
==  CreateAppTargets: migrated (0.3610s) ======================================

==  CreatePermissions: migrating ==============================================
-- create_table(:permissions)
   -> 0.0710s
-- add_index(:permissions, :name, {:unique=>true})
   -> 0.1290s
==  CreatePermissions: migrated (0.2030s) =====================================

==  CreateAppPermissions: migrating ===========================================
-- create_table(:app_permissions)
   -> 0.0520s
-- add_index(:app_permissions, :app_id)
   -> 0.0960s
-- add_index(:app_permissions, :permission_id)
   -> 0.1070s
==  CreateAppPermissions: migrated (0.2610s) ==================================

==  CreateComments: migrating =================================================
-- create_table(:comments)
   -> 0.0740s
-- add_index(:comments, :app_id)
   -> 0.1220s
-- add_index(:comments, :authorId)
   -> 0.1180s
==  CreateComments: migrated (0.3210s) ========================================

C:\Android-Market-Crawler\server>rake db:seed
(in C:/Android-Market-Crawler/server)

C:\Android-Market-Crawler\server>
++++++++

In code I changed src/com/marketplace/io/Secure.java just these lines

++++++++++++++++
preferences.put("username_key", "queenscrawler@gmail.com");
                // preferences.put("more username", "username1@gmail.com");
                // preferences.put("more username 2", "username2@gmail.com");
                preferences.put("password_key", "mypassword");

++++++++++++++++

Do I have to change more lines?

Then I execute
java -jar crawler.jar -c

It starts to create threads, and then I get the error I mentioned :/

++++++++++++++++++
Oct 10, 2011 1:04:50 PM com.marketplace.service.CategoryThread run
INFO: Creating thread for category TOOLS
Oct 10, 2011 1:04:50 PM com.marketplace.io.Fetcher getAppByCategory
INFO: Mon Oct 10 13:04:50 EDT 2011 Getting App(s) for Category REFERENCE
 --> On Index Id:0
org.apache.http.conn.HttpHostConnectException: Connection to http://0.0.0.0:3000
 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnect
ion(DefaultClientConnectionOperator.java:159)
        at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.ja
va:149)
        at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPool
edConnAdapter.java:108)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultReq
uestDirector.java:415)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpCl
ient.java:641)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpCl
ient.java:576)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpCl
ient.java:554)
        at com.marketplace.io.Sender.doBasicHttpPost(Unknown Source)
        at com.marketplace.io.Sender.appExists(Unknown Source)
        at com.marketplace.io.Sender.addAppToCollection(Unknown Source)
        at com.marketplace.service.CategoryThread.run(Unknown Source)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
        at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
        at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
        at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
        at java.net.PlainSocketImpl.connect(Unknown Source)
        at java.net.SocksSocketImpl.connect(Unknown Source)
        at java.net.Socket.connect(Unknown Source)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSoc
ketFactory.java:123)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnect
ion(DefaultClientConnectionOperator.java:133)
        ... 16 more
Oct 10, 2011 1:04:51 PM com.marketplace.service.CategoryThread run
INFO: Creating thread for category ARCADE
Oct 10, 2011 1:04:51 PM com.marketplace.io.Fetcher getAppByCategory
INFO: Mon Oct 10 13:04:51 EDT 2011 Getting App(s) for Category NEWS      --> On
Index Id:0
++++++++++++++++++

Do you see so far anything I can be making wrong?

Original comment by israelmo...@gmail.com on 10 Oct 2011 at 5:18

GoogleCodeExporter commented 8 years ago 
Ok, again never mind I just had to modify

public class Constants {

    public static String url = "http://127.0.0.1:3000/";

instead of 0.0.0.0:3000

Ok, this is closed... but still I have issues, so..., ok I will continue trying

Original comment by israelmo...@gmail.com on 10 Oct 2011 at 5:52

GoogleCodeExporter commented 8 years ago 
Hi,
Did you manage to make the crawler work ?

Original comment by fadwa.fa...@gmail.com on 25 Oct 2011 at 3:18

GoogleCodeExporter commented 8 years ago 
Yes, I did
But, unfortunately it was not what I was looking for, even though, it is really 
smart the way that was tricked the android API

Original comment by queenscr...@gmail.com on 28 Oct 2011 at 8:57

GoogleCodeExporter commented 8 years ago 
Hi, Raunak.

It's all ok for me to crawl top 480 apps, however, if I try to download the 
481th app, respond 509 will occur.

Do you think I've overused the API?

Original comment by clzq...@gmail.com on 9 Nov 2011 at 7:12