server error when trying to refresh mules

[Robonau]

when trying to refresh mules i get loading data server error full response:{"count":0,"created":"2017-08-13T20:46:20Z","lang":"en-GB","results":null}

[Zeroeh]

Deca limits the rate at which you can refresh char/list now. Since the actual IP that sends the request is yahoo, nobody will be able to refresh.

[zaqxswcdevfrbgtnhy]

What is the maximum rate they set?

[zaqxswcdevfrbgtnhy]

It'd be nice if there was some way to make muledump not have to send the requests through the central yahoo IP, as well as some rate limiter for how quickly it attempts to refresh the accounts.

[Zeroeh]

You get 9 refreshes every minute. Currently working with some buddies to create a xml to json converted so we don't need yahoo.

[zaqxswcdevfrbgtnhy]

Issue 114 may have some relevance. Thank you for working on a fix.

[zackpollard]

So I've spent a bit of time looking into the code for this to see if I can PR a fix. I'm running into some issues (I don't write javascript much so if people know an immediate fix to the problem i'm facing please let me know).

I have removed the call to the yahoo API and instead make the call directly to the realm API and convert the XML to JSON locally instead of using the yahoo API. Now the problem with this is the AJAX request that was being made was using the jsonp datatype, which gets around any kind of CORS (Cross Origin Requests) issues. When using XML data type, it applies CORS rules, and therefore you have to send the Access-Control-Allow-Origin: * header along with the request. If you don't send this, then the browser won't allow the response to be processed. If you do send this, then it triggers the browser to send pre-flight checks in the form of an OPTION request to the realm API. The issue with this is that the realm API rejects OPTION requests as it doesn't support that request type, therefore the pre-flight check fails and the request doesn't go through. I believe this may be the reason the yahoo API was being used in the first place, as it negates these issues. I can't see a way around this currently, so unless somebody has an idea on how to get around this, we could have a bit of a problem with muledump going forwards unless realm reverts this change...

Could be fun times ahead xD

[zackpollard]

@Zeroeh just read that you're working on a fix, XML to JSON converters exist already but how do you plan to avoid the CORS issues when you make the XML request to realm?

[Zeroeh]

@zackpollard It doesn't have to be written in JS. Just sayin'. And what xml request? It's an http request.

[zackpollard]

@Zeroeh Doesn't matter what it's written in, it's the browser that does the CORs checks. It's an HTTP request with an XML response type, which is subject to CORs. The reason that the current one is not subject to CORs is because it uses JSONP which is simply padded json, but this is let through without any kind of checks for where the origin of the request was. Calling the realm API directly we can only get an XML response, which means we will be subject to CORs, and the realm API doesn't respond correctly to the CORs checks so everything fails.

[Nillad]

if it gives you 9 refreshes a min should it refresh 9 of my mules before giving me server error or let me individually refresh at my own pace... it doesnt let me refresh any of them

[Zeroeh]

@Nillad That's because there are thousands of people trying to refresh through one ip

[Nillad]

I see.. is there some way to make my own ip for it to refresh through?

[Zeroeh]

@Nillad That's more or less what we're trying to fix here :P

[Nillad]

This is the issue right? } $.ajax({ dataType: 'jsonp', url: 'https://api.github.com/repos/thestdman/muledump/tags', complete: function(xhr) { xhr.done(checkupd); }

[Zeroeh]

@Nillad Nope, it has to do with everyone using the yahoo query server.

[Nillad]

is there only 1 query server?

[Zeroeh]

@Nillad There are others but if people start using those then it will just be like it is now. The goal is to have our own hosted query server so we aren't relying on 3rd parties.

[Nillad]

well these ones had to be made by someone right? like how would I make my own, or can you not have your own yahoo query server. Sorry if I'm sounding dumb

[Zeroeh]

@Nillad You can make your own "yql" server, but if you look above these guys are having issues doing just that.

[Nillad]

I've looked at several threads and they are all having issues fixing it but they also all seem to be trying to not use the yql in general instead of just making their own yql to request through

[Nillad]

queue_request({ dataType: 'jsonp', url: 'https://query.yahooapis.com/v1/public/yql', data: { q: 'select * from xml where url="' + url + '"', format: 'json' }, complete: callback }) so we are all using the same public api, if you can make your own private one it seems like this should be a simple fix

[Zeroeh]

@Nillad It's certainly much easier said than done

[Nillad]

https://developer.yahoo.com/yql/guide/yql_url.html seems like I just have to put the time in and make one then

[tischepe]

This might help: https://github.com/atomizer/muledump/pull/115

[Nillad]

@tischepe I've looked at it but if it were a real fix I feel like it would've been incorporated into base muledump code by now.. rather old post

[zaqxswcdevfrbgtnhy]

Does this mean every muledump user will have to manually set up their own YQL server? Or are you guys still working on a better fix?

[zackpollard]

@nillad your suggestion of making your own yahoo query instance wouldn't work unless they provide you with your own reserved IP, as otherwise you will still be rate limited, which is what I assume would be the case as I don't expect they will separate the public service and the custom service in respect to IP pools. Also the code in #115 won't work @tischepe as all the linked PR within there does is use another service which would have the exact same IP issue as the yahoo api has.

[Nillad]

@zackpollard Yeah I realize that, I was looking for another online url service like Yahoo that may provide private ones (most likely for money) but I couldnt find one

[Nillad]

I just got 2 complete refreshes.. fixed?

[zaqxswcdevfrbgtnhy]

@Nillad not for me, still gives

loading data
server error
full response:{"count":0,"created":"2017-08-15T15:13:53Z","lang":"en-US","results":null}

for me.

[Nillad]

yeah only worked for about 30 mins for me

[P1MGU1M]

Will still have muledump???

[zaqxswcdevfrbgtnhy]

@P1MGU1M It appears deca is also working on a fix:

Edit: We are aware of people having issues with Muledump not working properly. We're working on a fix.

Source

[Dragonlogesh]

@zackpollard I believe that by starting chrome, or any other browser, with web security disabled, you can bypass the CORS restrictions.

You can start chrome, specifically with web security disabled with the following command prompt Chrome.exe --disable-web-security

[P1MGU1M]

Some news about the muledump ???

[atomizer]

I am away until the 29th, so I can't really do anything at the moment. Sorry!

[P1MGU1M]

Has anyone managed to fix it yet?

[Zeroeh]

@P1MGU1M A few people have private working versions but none are public

[zackpollard]

Yea, a proper fix is basically not possible to do without deca fixing either their API to work with CORS or having muledump now require you run your own little server to proxy through requests to realm api, which is what I'm doing on my own server for now. Deca said they are planning to fix muledump so I would expect this is why nobody has released their fixes yet, it certainly is the reason why I haven't released mine.

[P1MGU1M]

Zackpollard could teach me

[P1MGU1M]

°????????????????????????????????

[zaqxswcdevfrbgtnhy]

There is a reddit post with a public fix now. Not made by Deca, so I hope they will release a fix of their own too.

[Nillad]

@zaqxswcdevfrbgtnhy link it?

[Zeroeh]

link

[UnicoRush]

@Nillad

https://www.reddit.com/r/RotMG/comments/6ut0tg/muledump_2_electric_boogaloo/

https://github.com/snarticuno/muledump-2

but I have not tested it...

can someone check?

2017-08-20 15:02 GMT+02:00 Nillad notifications@github.com:

@zaqxswcdevfrbgtnhy https://github.com/zaqxswcdevfrbgtnhy link it?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/atomizer/muledump/issues/148#issuecomment-323583803, or mute the thread https://github.com/notifications/unsubscribe-auth/AG8IDj4QsOHU-FRi3yCxzOyn5fIae_Sgks5saC6CgaJpZM4O1y78 .

[Nillad]

I'm testing it now, having some issues

[Nillad]

Update.. takes a bit to get it running but it works, creator is still ironing out last few bugs and rate limit still exists but it is a only your accounts requesting instead of all of the muledump users through yql

[Nillad]

as long as you have 38 or less mules and you dont refresh them like a mad man you'll have no issues Discord Invite: discord.gg/fHXFNgh https://github.com/snarticuno/muledump-2

[Nillad]

Discord Invite: discord.gg/fHXFNgh support page for all your questions