alexander-pick / MKMTool

MKMTool ist a helper application I wrote for tinkering around with optimization of sale processes on magiccardmarket.eu and the idea of automisation of some tasks most people wouldn’t be able to get done by pure manpower.
GNU Affero General Public License v3.0
65 stars 15 forks source link

Am I missing something? #5

Closed MeEgt closed 4 years ago

MeEgt commented 6 years ago

Hi there.

About two months ago I used the tool to update the prices after uploading cards with 'MKMEye' and didn't even try another function. Have about 2000 cards in my offers. After new hardware and fresh windows installation (+MVS2015) every action crashes the tool, except for: settings account info Check Display Value Check for cheap deals (check wantslist crashes as well)

i.e. Update Prices MKMInteract.cs line 63 var response = request.GetResponse() as HttpWebResponse; An exception of type 'System.Net.WebException' occurred in System.dll but was not handled in user code

Mainview.cs line 121 await Task.Run(() => updatePriceRun()); An exception of type 'System.Net.WebException' occurred in mscorlib.dll but was not handled in user code

Program.cs line 47 Application.Run(new MainView()); An unhandled exception of type 'System.Reflection.TargetInvocationException' occurred in mscorlib.dll

What am I missing?

Regards

tomasjanak commented 6 years ago

Hi,

could you modify the MKMInteract.cs to write the message of the exception that is happening on the line 63? Maybe it can tell some details. Just replace the makeRequest method (lines 42 to 68) as follows, it will write the exception into a error_log.txt file besides the executable (MKMTool/bin/DebugOrRelease/) and then let the software crash again:

        public static XmlDocument makeRequest(string url, string method, string body = null)
        {
            var doc = new XmlDocument();
            try
            {
                var request = WebRequest.CreateHttp(url);
                request.Method = method;

                var header = new MKMAuth.OAuthHeader();
                request.Headers.Add(HttpRequestHeader.Authorization, header.getAuthorizationHeader(method, url));
                request.Method = method;

                if (body != null)
                {
                    request.ServicePoint.Expect100Continue = false;
                    request.ContentLength = body.Length;
                    request.ContentType = "text/xml";

                    var writer = new StreamWriter(request.GetRequestStream());

                    writer.Write(body);
                    writer.Close();
                }

                var response = request.GetResponse() as HttpWebResponse;
                doc.Load(response.GetResponseStream());
            }
            catch (System.Net.WebException e)
            {
                using (var sw = File.AppendText(@".\\error_log.txt"))
                {
                    sw.WriteLine(e.Message);
                }
                throw e;
            }

            return doc;
        }
MeEgt commented 6 years ago

Thank you for your fast response. I am fighting with a severe flue at the moment... I will post the log in a few days.

MeEgt commented 6 years ago

The error_log.txt has only one line in it.

Der Remoteserver hat einen Fehler zurückgegeben: (403) Unzulässig.

The language is german but the essence is the error code 403 I guess.

This is from about 3 days ago. I just forgot to comment it here. The interesting thing is that the "Price Update" is working right now. Could it be possible that this has something to do with the amount of cards in my stock? I'm down to 1650 cards and it is working.

I will let the program run for now and test the other features after that. I will add about 100 cards to my stock and will test again tomorrow.

tomasjanak commented 6 years ago

The MKM says in the API documentation that "You get a 403 Forbidden HTTP status, when you try to access valid resources, but don't have access to it, i. e. you try to access /authenticate with a dedicated or widget app, or resources specifically written for widget apps with a dedicated app. "

Not sure what that actually means, but it might be somehow related to your fresh windows installation, I would try to renew the app tokens (in "My profile" on MKM) and make sure it says to set the "App type" to "dedicated".

tomasjanak commented 6 years ago

So I have a little on-topic story. It will be long, but hopefully worth it.

I don't know if MeEgt managed to resolve his issue, but this Monday I encountered the same problem myself - nothing works, a 403 response is returned to everything. So as I said above, I thought I will try to renew my app token and went to the MKM's website. There I found out that not only is my token removed, but I also don't have the option to create a new one. So I wrote to the customer support asking what is happening and later I received a reply that rather surprised me.

Your account was blocked from using the API by our security mechanisms running to detect possible abusive use of the API.

You have exceeded your daily request limits with nearly only using requests from the marketplace group, i.e. identified as grabbing data only with a dedicated app.

They were also kind enough to unblock me, but it still left me a bit confused. Why is there an API call limit if using it to the max is still considered abusive - why isn't it lower then? Is the catch the "only using requests from the marketplace" clause? So I sent another series of questions.

At his point I will add a little background: I am just a regular small seller, so I have the 5000 API requests limit per day. I also don't sell many things, I have just about 700 cards in stock right now, so the limit wasn't really any issue for price updating, which is the only thing I have been using until recently. However, a couple weeks ago I modified the "check for cheap deals" to be able to check cards of a specific seller (by username) so that if I am buying something from somebody, I can also automatically check if that seller has some good bargains that I could add to my order so that I get the best value of the shipping cost that I am already going to pay anyway. I was ordering a bunch of cards for some decks I am building, I made a total of 10 orders and for most of the sellers, when there was some "room" to add something to the shipment, I checked if they have something well-priced. Even though I filtered quite a lot of cards (max price, condition), I obviously still ran out of API requests eventually. I was kind of expecting it, so then I just turned off MKMTool, finished my orders manually and went on to do other things. And then on Monday I got blocked.

Right, so this is what I asked them (shortened) :

Is my account always going to be blocked whenever I exceed the daily limit (if I am not mistaken, it happened to me only once, this weekend, and I have been using the API for a few months now)? Are there some cases that would not be considered abusive even when the limit is breached (you said that it was because I used nearly only marketplace requests....)? If I for example just added a counter in my software that counts how many API calls I made today and shut itself down when I reach 4999, would that be OK? Can my account be blocked for "grabbing data only" even if I use less than the call limit?

The gist of their answer was that the way I am using the app is "not supported". They obviously don't want to reveal their security mechanism, which I think is reasonable as it would make it easier to exploit, but it would be nice to have some more precise guidelines as to what will be tolerated and what will get you blocked, since I don't see what I am doing as abusive - using the "check cheap deals" functionality will even potentially lead me to make deals through MKM I wouldn't otherwise do, which I would expect would be encouraged by MKM since they get a cut from all of it.

I also asked if it's OK to make their answer public to which I got no response so I will just assume "no", but the answer was rather short and didn't reveal much that isn't obvious anyway, so I will take the liberty of citing perhaps the most interesting bit of it:

  1. Some decisions are taken manually after a user being on a watch list for a while. We decide about taking actions on a lot of indicators, which include past account activities, revenues, and a lot more.

I would also like to point out that the "update price" uses the same type of API requests, so you can easily get blocked for that as well. And apparently, that can happen without any warning the day somebody at MKM decides that you are using it too much. I don't know if I was "on a watch list" for using MKMTool everyday for the past few months or if it was all based on me breaching the request limits. I don't know if it's ok to keep using MKMTool everyday to update the prices of my couple hundreds cards in stock or if it will get me blocked again. The only thing I know is that nobody will tell me that I am doing something wrong until they block me. Actually, nobody will tell me that even AFTER they block me, I will have to ask. Can't say that I approve of this policy...

They pointed me to the part of the documentation that gives at least some guidelines - https://api.cardmarket.com/ws/documentation/API:Auth_Overview . The important part from that is this - in the description of what is a Dedicated App:

Attention: We explicitely do not allow, that Dedicated App users constantly only request the public Marketplace resources (products, articles, prices, etc.) on consecutive days and especially not with exhausting the request limits. The purpose of Dedicated Apps is the support of the user with their normal MKM activities, which explicitely includes managing their stock, order, wantslists, and handling their shopping carts. We implemented extensive security mechanisms to withdraw a user's possibility to request all resources by blocking a user's API access. If you plan to only request marketplace data for the use on your website, plan to develop an application that shows MKM offers and prices, you have the option to apply for a Widget App.

On the first glance, MKMTool violates that, because for most of its functionality it uses a lot of "Marketplace resources", they make the bulk of all the API requests. However, they are not used for any purpose outside MKM, such as setting prices in your personal e-shop etc. They are used quite directly for "managing their stock [MKMTools's update price], order, wantslists, and handling their shopping carts [MKMTools's check cheap deals]". So the way I see it, MKMTool is not against the idea of Dedicated Apps "ideologically", but nevertheless, the implementation will make MKM's security measures trigger. But I don't know if something can be done about that. The support person made it rather clear that they are not interested in discussion about the security measures so the only way to get some kind of an exception would be to make MKMTool itself approved by the MKM. To do that, there are 2 options, the widget app or the 3rd-party app.

The Widget App has to NOT use user credentials, so by that definition, MKMTool cannot be considered a Widget App?

The "3rd-party apps" description says this:

If you want to develop an application, that provide services to all other active MKM users and let them use the functionality through the API with your app, e. g. you retrieve their stock and let them manipulate their stock, this app type [3rd-party App] is your solution.

MKMTool fits into this category well. However, "3rd-party Apps" are available only to professional sellers, which at least for me makes it completely not interesting. I am not a professional seller. In fact, it is BECAUSE I am not a professional seller why I want to use an app like MKMTool - I don't have the time (or desire) to everyday spend hours analyzing how is the magic market developing and make business decisions based on that.

To conclude, if you are not a professional seller, your API access tokens can get blocked anytime you use MKMTool. My guess is that if you use only couple hundred API requests per day (e.g. have couple hundred cards and update prices at most once per day) you will be fine, but it is only a guess. Using the check cheap deals functionality is likely to get you blocked. You will not get any warning from MKM.

Tomas

P.S.: just practical notes: MKMTool should probably shut itself down if a 403 error is encoutered during initialization, as many subsequent actions will lead to a crash anyway. Something in the sense of "403 encountered. Your MKM App Token is probably invalid, check it in your account settings. Nothing will work anyway, so MKMTool will now shut down, bye."

alexander-pick commented 5 years ago

403 is normaly an error you get due to invalid auth, Tomas is right here. Please check the token and stuff.