Anyone else getting a lot json invalids lately (403 Forbidden)?

[MSII]

Is it something with my local files or is everyone getting a lot of json is not valid notifications in the last day or so?

[wazerstar]

Yes already reported it to Maxx, it started 15-20 hours ago, I'm not sure if its the latest builds.

[MSII]

Okay, was just checking. Recently listed queries still run fine. It's just specific items. There are two queries that work fine every time, and then 2-3 errors follow. It's this way with every item; always 2 that work fine and then 2-3 json errors.

[wazerstar]

Just start 4 or more and you will see it goes nuts.

[MSII]

I have a lot more than 4 running, I meant that each individual page scan has two successful scans and then 2-3 json invalid errors.

[Maxx53]

I can't affect at this situation. If Steam server sends back broken data, It's problem on his side.

[KaliBit]

Hello, two days ago steam implemented this restriction, in which the server replies with the code 403 (Forbidden) to the listing page request when the user makes a lot of requests to their servers. I couldn't determine exactly how many, but I guess it is around 3 requests per second, and it will block you for X seconds. So, if you are listening to many items this will obviously be exceeded. I tried to solve this by playing with cookies/sessions, but it seems that this is an IP based restriction, meaning that this restriction also applies if you are not logged. In my opinion this is a crappy way of dealing with bots because it is already affecting legitimate users (http://www.reddit.com/r/Steam/comments/26u6jx/steam_market_blocking_on_lan/). I don't yet know if this can be solved by throttling the requests between some proxy's, I guess we just have to try :)

[MSII]

What KaliBit said seems right, I've noticed that if you take the amount of items you are scanning for on page down to less than 9, the json errors do not happen.

It's a restriction by steam on how many requests you can make, presumably no more than 3 per second, since 3000ms delay = 3sec every item, 3 sec / 9 items = 3 requests per second average...

Is there any way to get around this?

[dmarcuse]

Let's answer the question with a question. Is there a way to check multiple items with a single listing check? (Similar to the way browsing a search works - showing a list of items and more basic data)

If that's not possible, then how about being able to set a priority for each item, meaning it is checked first every check, so less important ones are affected by the block if Steam decides you're refreshing too fast.

[Maxx53]

@KaliBit Thanks for information. I think proxy is believable solution.

[KaliBit]

@apemanzilla your first observation is good, however, I do not believe that is possible. Because if you could build a query that would output exactly the items that you want to listen (I tried for a bit using the new advanced search options, but I was not able to mix appids), you would have to make an additional request to the listing page itself because the search would not give you the information that is needed to make a purchase, also, you would be assuming that the price given by the search would be the lowest possible. The second suggestion is possible, but we would still be limited by the restriction :(

@Maxx53 No problem. I also believe that the proxy would work, however, we will probably run into some session problems. Other aspect is that steam is probably logging the IPs, and if they are smart, they have something to warn then if a user is jumping IP's like a rabbit, making it easier to catch bots.

[jksjksjks]

yep, just checked, with up to 9 items, there is no problem, no json!

[Maxx53]

Added proxy list support in the last commit. Need testing with fast proxies! Using: Create text file "proxy.txt" next to "SCMBot.exe". Fill with proxies. String format - IP:port. For example:

[ghost]

getting "too much requests per second" with 7 proxies

[Maxx53]

How much proxies and scan items? I believe, optimum may be reached if "proxy list count > scans"

[ghost]

7 poxies 1 scan at 100 delay

Edit: I added more proxies and its fine now. Thanks great update.

[Maxx53]

Maybe not whole list was loaded. Need to check, add to this line, then rebuild: https://github.com/Maxx53/SCMBot/blob/master/scr/SCMBot/Main.cs#L136 Code:

MessageBox.Show("Proxies loaded: " + proxyList.Count.ToString()); 

[ghost]

great, that fixed it.

Edit: Okay I'm running 3 scans, in one it displays "parsing fail" because there are no listings for that item, but in the other 2 it displays "content empty"

[ghost]

Update: still getting Too much requests per second with 17 proxies and scanning one item @ 500 delay

[dmarcuse]

How does the proxy integration work? Is it round-robin, different proxy each request, or "assigns" one item scan to each proxy if possible? I feel like the second may be more stable if we go on the 9 item scan rule, assign 9 items to a proxy, and hope Steam accepts it.

[ghost]

9 item scan rule? what's that? I was scanning one item with 17 poxies and I still get "too many requests per second"

[dmarcuse]

@fcastro16 In earlier comments, we found that 9 items at default timing seem to work fine without triggering the 403 lockout. If we could get it working with 9 items per proxy, I think that would suffice for almost unlimited items (provided you have proxies to accomodate for them all)

[Maxx53]

Yes, different proxy every request. Load is shared. You can see short visualization of algorithm principles here: http://youtu.be/Djzq3RzIqwU

If all proxies in use, request is going directly (without proxy).

UPD: Proxy List featue updated!

[wazerstar]

hmm anyone know how we could make our OWN proxy if we have a web domain? i know you can make a proxy "website" for your personal use but how would i manage to make ip:port for my personal domain i bought?, i mean there must be some php scripts do to that?

I dont see how using public proxies would be great, they are trashy, too long delay and would not be up the all time.