very slow speed, timeouts

[pepev12]

Hi, Its been many weeks since im experimenting a very slow speed in how the tool interacts with the api, i think its mkm api's problem, but i would like to check with other people, is it working fine for you? regards,

[fverdoja]

same here, the tool (v8.0.2) makes about one API request every 20sec on average... it's way slower than before...

[tomasjanak]

It has been like that almost a month now (reported here: https://github.com/alexander-pick/MKMTool/issues/46). Just for sanity I tried some old versions of MKMTool and it is equally slow, so it is definitely because of Cardmarket. If somebody manages to get some explanation from Cardmarket support please post it here, thanks.

[pepev12]

I have a ticket with them from two weeks ago , lets see if they can do something, Maybe its a good idea everybody with problems open a ticket...

[pepev12]

I noticed that its only slow with some processes, like updating prices (full update); if you use only ensure minimun price, it is blazing fast, Its also fast when uploading cards. Seems like reading prices could be where the bottleneck is?

[pepev12]

Appraising external lists and checking deals of display value are also slow; clearly is an issue when reading prices.

[tomasjanak]

There is an update in the API documentation from 15.12.2020., which is around the time we started seeing the slow downs:

Api Request Limitations (II)

December 15th, 2020

From now on - and in addition to our normal request limits - all requests to the Marketplace are limited to 30.000 per day. Exceeding this limit will return a 429 Too Many Requests and it applies to the 1.1 versions respectively.

Example: While professionals have a request limit of 100.000 requests per, only 30.000 of those can be used for requests from the marketplace group. If the 30k limit is reached, but not the 100k limit, the remaining requests still can be spent on requests from the order, stock, etc. groups.

The requests we use for Price update / List Appraisal belong to the Marketplace group (and check cheap deals too, basically whenever we do "get prices of other sellers"). While it does not explain the slow downs, I think it is likely related, since clearly they have been doing some changes with those requests. So I think they either made some mistake, or added some additional speed limitation they are not telling us about. I will write a question to MKM support.

I will add a counter for the Marketplace limit so that MKMTool does not make more than 30k of those requests per day once its cleared up (currently it looks like nobody can get nowhere near 30k per day anyway).

[pepev12]

Clearly is something related to that change. I updated my ticket with the suspicious, urging them for a solution, It's possible any kind of workaround to use other api calls?

[tomasjanak]

Clearly is something related to that change. I updated my ticket with the suspicious, urging them for a solution,

Thanks, let us know if you get some reply.

It's possible any kind of workaround to use other api calls?

No, there are the following types of requests:

Account Management Market Place Information Order Management Shopping Cart Manipulation Stock Management Wants List Management Cardmarket/API Services

The only ones that get you to actual cards on sale are Marketplace Information and Stock Management, but as the name suggests, Stock Management lets you operate only on cards you have in your inventory. So there is no way to get to any prices from other sellers, or even just the trends, without using the Marketplace Information.

[pepev12]

I have received this answer from support, honestly im not sure it makes any sense:

Due to the problems, the API and the massive usage of the marketplace requests, we had to implement some additional limits to marketplace requests. We suggest to

1. Synchronously use marketplace requests, i. e. wait for one request to be responded before requesting the next endpoint. Simultaneous requests to marketplace requests from the same app will be queued on our side - sometimes resulting in time-outs.

2. Keep your own persistence tier of non-volatile data. This helps massively reducing the amount of requests to marketplace data.

3. Make sure, not more than 600 requests are send within a period of 60 seconds.

Please accept our sincere apologies for those necessary changes. We're constantly working on providing powerful tools and we're working hard on making the API more accessible and comfortable.

[tomasjanak]

Thanks. It does make sense, but does not answer any of our questions. We already do all three of those things. Could you please send them another message like the following (since you already have a opened ticked with someone assigned maybe it will go faster)? You can tell them it is from the developer of your app. Or should I send it?

First, to your suggestions:

1. Our application does all API requests synchronously.
2. Recently we have done several updates to limit as much as possible the amount of requests made, including making a more detailed copy of the card database locally so that we do not have to re-fetch non-volatile data.
3. a) we have implemented checks to ensure this limit and b) we are nowhere near being able to make 600 requests per 60 second.

Currently, we are able to do about 5 Marketplace requests per 60 seconds, which is a drastic reduction from previous speed (before 15.12.2020), which was about 200-300 per 60 seconds. This happens even at the beginning of the day with 0 previous requests done (so the "30000 Marketplace request per day" limit implemented on 15.12.2020 definitely does not limit us). We double checked that this is not time spent by our application, it is solely time spent waiting for responses to requests - the wait times are in the order of tens of seconds for a single Marketplace request. This makes them practically unusable.

Therefore, there must be either (A) some other limits in addition to those already published on https://api.cardmarket.com/ws/documentation/API_Main_Page or (B) some issues with your systems. In case A, we need to know what those additional restrictions NOT already described on your website are to be able to assess how we can or cannot use Marketplace requests. As far as we know, we are using them in a way that should be in accordance with your restrictions. But the results we get are unexpected. In case B, we would appreciate more transparent communication about those issues and time estimates on when we can expect them to be fixed, thank you.

Note that this significant performance drop happened virtually overnight, i.e. it is not a (direct) result of continuously increasing amount of users: some changes clearly happened on your API back-end in the middle of December and regardless of whether those changes were planned or unplanned, we cannot properly react to them without more information.

To sum up, from our side it seems that something is wrong on Cardmarket's side with the Marketplace requests. Please let us know if we are mistaken and you are actually satisfied with the current state and have no plans to rectify it. We have seen that you are trying to improve the performance of Cardmarket (https://www.cardmarket.com/en/Magic/News/Temporary-Suspension-of-Price-Guides-on-Cardmarket), it is just not clear if improving the Marketplace requests is part of those efforts or if their current speed is here to stay for foreseeable future.

Best Regards, [signature]

edit: originally I wrote a "P.S." question which is actually almost surely not relevant, removed

[Lelong-vincent]

Any news on this issue by any chance ?

Should we all open a ticket to try to pressure them to get an answer ?

[tomasjanak]

This thread is obsolete now, I will close it. If you want to discuss the current API outage, you can use thread https://github.com/alexander-pick/MKMTool/issues/63