Fifa 18 update and release schedule

[oczkers]

I've updated code to work with new version of the game but not all methods are yet implemented - there is no config with urls like it used to be before so all urls have to be detected manually. It would be great if You can help me with this.

Release schedule:

• [x] Update login & request methods to work with new version

• [x] Detect rest of urls & release beta for brave users (today/tomorrow)

• [x] Implement pinEvents (this week)

• [x] Final touch and final stable release (end of week, next week)

• [ ] (possible) implement emulate feature once again - it should be much easier and safe when pinEvents gonna work (end of month?)

PS. Be careful with sell method - buy now price is required now and i've set it to ea default value (10000). Should we boost it to max value maybe?

[ghost]

Nice to see the API updated

Can I suggest something? I think it should be a good idea to clean the old unused branches and maybe also clean up the open issues

[oczkers]

Yes indeed, would You like to help me with managing issues?

[KmanProdz]

What's left for step 4?

[converge]

@oczkers I can help you cleaning the issues

[oczkers]

What's left for step 4?

Find rest of urls and add missing pinEvents calls.

[oczkers]

Done, most pinEvents are made so it should be safe, I have to test timestamps (and add some random here probably) and then it'll be very safe :-).

WARNING: When You go from club items to transfermarket in webapp there is 3 pinevents called (club -> home -> transfermarket) BUT if You're simply using club and tradepile/watchlist method there will be only 2 pinevents (club -> transfermarket) so they can potentially detect this anomaly. Making this automatic would generate a lot of additional code (saving where did You get it from etc.) so please do it manually if You plan to bombard ea servers.

Enjoy :-)

[jsarasti]

I don't know if you've noticed, but before each GET to a specific URL, the Web App does an OPTIONS request to that same URL. Should we do the same?

[oczkers]

Yes but this options are made without any data, just to ask server what kind of request is accepted so waste of time in my opinion.

[rafaelget]

Maybe, but they can track that requests too.

[fifa2017player]

Thanks a lot for updating this to work with Fifa 18!

Would it be possible to add some documentation to the code or the repo page about how pinEvents work? I am not sure I understand what you meant by "do it manually" - do you mean we should write code to trigger those pinEvents, and when do we need to do them?

[oczkers]

Would it be possible to add some documentation to the code or the repo page about how pinEvents work?

We need a lot of work on documentation but as we all know writing this is kinda boring so yes there'll be nice, big, up to date and easy to understand documentation, someday :-)

So far i can tell You that pinEvents are just reports sent to ea - not every request triggers it, basically only when You change "page" like going hub -> transfermarket or club -> hub etc. (little bit different on mobile apps).

do you mean we should write code to trigger those pinEvents

Yes, pin class with necessary methods are done, look at code to know how to trigger it.

and when do we need to do them?

Depends on what You're doing, but probably You don't need it when staying all the time on transfermarket screen.

[fifa2017player]

Thanks for the response! I have filed a separate issue to track updating of the documentation.

[jsarasti]

@oczkers First of all, thanks for all your hard work on this. It's more than appreciated.

Secondly, you added the OPTIONS request where it's actually not needed. I don't see the Web App invoking OPTIONS on pin, only before actions. So "OPTIONS /tradepile" -> "GET /tradepile", "OPTIONS /transfermarket" -> "GET /transfermarket", "OPTIONS /watchlist" -> "GET /watchlist", etc.

[oczkers]

I don't see the Web App invoking OPTIONS on pin,

I thought You were talking about pinevents, anyway there are options here, at least in my browser (chrome). https://i.imgur.com/UoEfIfV.png

[jsarasti]

Thanks for adding that. Looks good now!

BTW, I've also noticed another thing that the Web App includes in the query string and that we're currently missing: a final, underscore parameter for all requests other than pinEvents (e.g. "/watchlist?_=0123456789012"). It seems to be 13 digits, doesn't look random but incremental, and the value stays the same during one session. Any idea what that could be?

[oczkers]

As far i remember it was just timestamp for get requests only and it's already implemented, something changed? https://github.com/futapi/fut/blob/master/fut/core.py#L586

[ricklhp7]

@oczkers I find out that the first value of the parameter _ is time() * 1000 + 3 (I dont know if i missed 3 first gets, i need to check), but after that it only increments 1.

So if we connect when the timestamp is 1508021577, the first get will have the parameter = 1508021577003, than the second get will have the parameter = 1508021577004, doesn't matter if we do this request 1 minute later.

[ricklhp7]

Not time() 1000 + 3... the first one is just the timestamp in miliseconds (I would try time() 1000 + randint(0, 999), because the first one always finishing with 000 is suspect imo)... than increments +1 every next GET. The OPTIONS request will have the same _ value than the GET request.

[ricklhp7]

In pinevents:

the ts_post aways have 500ms more than ts_post value. Sometimes 501 or 502. I think we need to put this difference to be safer.

PS: maybe there are some pinevents without delay (sometimes 1 or 2 ms). The only one i found is the first one when we are logging, when the session didnt start yet.

[ricklhp7]

I stressed my cpu and my internet and still same results... the page took about 2 seconds to load, so this delay has nothing to do with our lag our hardware. Its always between 500 and 502ms.

[oczkers]

but after that it only increments 1.

Oh crap, i'm almost sure it wasn't working like this before. Beautiful and clever work! Pushing fix in few minutes.

the first one is just the timestamp in miliseconds (I would try time() * 1000 + randint(0, 999), because the first one always finishing with 000 is suspect imo)

When is the first request with proper _? I've logged with incognito mode (fresh start) and it's counting just after You enter credentials (remoteconfig), before it's not frozen and depends on actual time. In my chrome it's not rounded to 1000 so it's probably browser (javascript engine) case. http://prv.get24.org/ss/2017.10.15-23.49.14.png

the ts_post aways have 500ms more than ts_post value. Sometimes 501 or 502. I think we need to put this difference to be safer.

I've implemented it few days ago here and here.

[ricklhp7]

I did a lot of work on pinevents today, I will post everything tomorrow in the morning.

I think we will need to work with threads with that. If we are in the search page, when we click in search and male a get request in 0:00:00.000, the pin event is posted exactly 500 ms later (sometimes 501, 502 ms...). The ts_event is about 0:00:00.000 and the ts_post is about 0:00:00.500. Between 0 and 500 is possible to have some requests, but im pretty sure a bid request is asking to have your account flagged.

In the first pin event you called it before verifying the answer question, it is posted about 500 ms after that. I think we need to have this delay to post the pinevent. The ts_event is the moment we called the OPTIONS request, not the post or get.

Another thing. There is only one OPTIONS request for pinevents every 600 seconds. The initial time is the moment of the pinevent post, not the on the options request.

Every 60 seconds we go to transfer hub page we call the watchlist and tradepile, but not their pinevents, unless we go to tradepile or watchlist page, obviously.

I will post more thing and data soon. Sorry for my english.

[jsarasti]

Oh crap, i'm almost sure it wasn't working like this before. Beautiful and clever work! Pushing fix in few minutes.

It sure wasn't. I was looking at this on Friday/Saturday and the "_" parameter stayed the same the whole time after the initial value increase.

[oczkers]

@ricklhp7

Correct if I understood wrong:

1. get request is made (watchlist for example)
2. after 500ms (+ 0-2 if rendering page is slow) options and post is sent to pinevents with get request timestamp as event and actual timestamp as ts_post.

There is only one OPTIONS request for pinevents every 600 seconds.

I cannot verify this - in my browser options are made before every request, no matter when and how much is made http://prv.get24.org/ss/2017.10.16-10.37.47.png There is some kind of "refresh" time in remoteconfig but it is for refreshing this config file only and i don't think we have to worry about this - You can change card in browser and chrome will freeze all scripts + config is downloading without any personal data (nucleus, persona etc.)

Every 60 seconds we go to transfer hub page we call the watchlist and tradepile, but not their pinevents, unless we go to tradepile or watchlist page, obviously.

What do You mean exactly? I've opened random hub page, stay on this without touching any button for ~10 minutes and only requests being made is for remoteconfig refreshing. http://prv.get24.org/ss/2017.10.16-10.56.44.png

ps. sleep added to pin, timestamps corrected https://github.com/futapi/fut/commit/3ddea2cacbd966e5d1f16daa0bac4be5d9115e14

[ricklhp7]

I will post some data to explain. Its valid for any kind of request, but I tested it in search.

a) 0:00:06.114 we did a OPTIONS request. The PIN request is done 0:00:06.615 (501 ms later). ts_event = 0:00:06.113, ts_post = 0:00:06.614 (501ms)

b) 0:00:00.312 we did a OPTIONS request. The PIN request is done 0:00:00.813 (501 ms later). ts_event = 0:00:00.311, ts_post = 0:00:00.812 (501ms)

c) 0:00:08.396 we did a OPTIONS request. The PIN request is done 0:00:08.898 (502 ms later). ts_event = 0:00:08.394, ts_post = 0:00:08.896 (502ms)

So what I think. We need to get the time right before we are doing the OPTIONS request and use it as ts_event time. The delay to make the pin request is the same of ts_post and ts_event. Sometimes it can be 1ms different for rounding reasons.

---

So if you are always requesting OPTIONS for any pinevent, i really dont think we need to worry about that. In my case, ONLY IN PINEVENTS, only once every 600 seconds they request a OPTIONS request. For others requests there are always a OPTIONS request (actually not 100% of the time), only for pinevent is different.

---

Sorry if i couldnt explain the last thing. I will try to say with examples, because its hard to explain in english.

We are logged for a while and we are in the Home tab.

In 4:06:12.087 we click in Transfers tab. (probably couple ms earlier, this is the request time) In 4:06:12.087 we requested the OPTIONS for tradepile In 4:06:12.090 we requested the OPTIONS for watchlist PS: my ping is about 200ms and both requests where done simultaneously, but I dont think it matters.

In 4:06:12.346 we requested the GET for tradepile In 4:06:12.589 we requested the pinevent (ts_event=4:06:12.087, ts_post=4:06:12.588) In 4:06:12.919 we requested the GET for watchlist

The pinevent could be done earlier, but it have a right time to be done (about 500ms after request), but my ping is very high. It makes quite obvious it is in a pararell thread.

In 4:07:00.000 we went to transfer list page and made a request for tradepile again.

If we to to transfer tab again in:

a) 4:07:13.918 -> no request are made for tradepile or watchlist, because it will only happen 60s after the last one b) between 4:07:13:919 and 4:07:59:999 -> only the watchlist request is going to be done c) after 4:08:00.000 -> both watchlist and tradepile list requests are done again

PS2: Find out another thing... When we log we actually request the transferlist, we can work it in higher layer only. I dont think its very important, but I will do it in my script to look more real.

[oczkers]

So what I think. We need to get the time right before we are doing the OPTIONS request and use it as ts_event time. The delay to make the pin request is the same of ts_post and ts_event. Sometimes it can be 1ms different for rounding reasons.

Looke like ts_event is just time when user clicked button and ts_post is 500 ms when action is being reported to server. Right now we are making request and immediately after saving timestamp for ts_event (depends on cpu but should be around 0ms ;-) ) so before ea server receives connection we already have timestamp saved (~0 ms for cpu calulation < ~30 ms for connection being established). Sure we can add complexity to remember exact ts_event time but it's irrelevant. Don't get me wrong but we're talking about milliseconds so it is simply not possible they can track this - every browser, javascript engine in browser, cpu or internet connection acts little bit different but most important this is user timestamp so completely different and not comparable with ea servers timestamp.

We have to make sure user timestamps are correct one to each other (500ms delay between ts_event ts_post) and sequence of connections that ea server gets. Sequence won't be ideal even when browser makes it because even if two connections are supposed to be made parallel there is always internet lag, cpu lag, browser connection limit etc. so server will get one or another randomly. I'm much more worried about not downloading javascript files or not refreshing images but it's probably irrelevant too (browser cache).

About third case, that's kinda what i was talking about in warning before - to get this automatic we have to remember what was user doing are what page was he looking at so it should be implemented on higher level in my opinion. Also 60 seconds delay between pile refresh is not always preserved - when You move item between piles/sell item or sth like this it's gonna refresh immediately so they won't track this in my opinion and it's just to relieve servers.

When we log we actually request the transferlist

One more request on start is barely nothing so we can add it but like as You i think it's not very important. Up to You guys ;-).

PS. Like always this is just my opinion and i'm opened to every suggestions, criticism and discussion.

EDIT: I think we should release new version soon so let me know if anyone have suggestions.

[ricklhp7]

I think you are right, Im just saying how its working, so we can make it look as real as possible.

I dont think the order of the requests is extremely important, but if it is easy to do it, so why not?

Are you going to implement refreshing images? In the past it wasnt a problem, but who knows.

[adamjeremiah]

This all sounds great, and I've started implementing this to my own bot. My one big concern currently though, is captchas.

We can make this seem are real/human as possible, if it fails on captchas then ultimately it'll still end in a ban.

I've tried figuring it out myself but I don't know enough about http requests and replicating them.

[ricklhp7]

Now I saw the updated core and pin. Its amazing. I really get it when you told that doesnt matter the order of those requests, it really doenst. I will call pin events in another thread just to be faster, but its not a priority.

About captchas, if I got 5 in all fifa seasons (since I bot - 2014) is too many. Maybe you rpm is to high for so much time. In my script I just raise an exception and close the account. So I do the captcha manually.

My script sends me a message when it happens, so I fix very fast.

[adamjeremiah]

My issue is, I used ultimatebroker, which handled things very poorly. And I fear that since my account has been temp banned once, I'll be on some sort of watch list. My RPM was set to 7. Whats a safe number?

[ricklhp7]

there is no safe number... and actually rpm is complex. When we do it manually, some minute i make 60 requests, 5 in another minute...

Temp ban is trigged when you search the market more than 1000 times in 60 minutes or 5000 in one day, so even humans get that. I dont think it will flag an account, because I had a lot temp bans in the past.

@oczkers you implemented 1s before bidding, right? I dont know if its too much or its really that. I was thinking in 500ms. In the past we got an error if we bid to things in a interval below 1 second, but I always snniped the first item in less than 500ms. Got all my accounts banned last season, so Im thinking in a safer value. 1 second is definetly safe, but maybe to much.

[ricklhp7]

Tested with an autoclicker that I made for this fifa, its absolutelly fast. It will click the buy now and the ok buttons in almost 0ms, i just hold a key and it keeps sending clicks events to both, so really its really fast.

After the get request of the transfer market, I was able to send an OPTIONS bid request in 926ms. The buy now button is locked for some time, even when we can see the players. 1s looks to much, but its pretty close. Its going to be so hard to snipe in a safe way.

[oczkers]

Im thinking in a safer value. 1 second is definetly safe, but maybe to much.

I was getting "426 Too many requests" (just one error, not temp ban) even with 1s delay between bids so i'll be testing ~1.1s.

Temp ban is trigged when you search the market more than 1000 times in 60 minutes or 5000 in one day,

Sounds right, after temp ban is gone and you make ~50 requests captcha is triggered, can You confirm this? I've made a little mistake in my script and instead of sending sms with captcha info it crashed so i solved it ~60 minutes too late, waiting for ban probably ;-/.

Guys it's best to avoid temp bans and captchas so stick to limits that ricklhp7 wrote. (1000 times in 60 minutes or 5000 in one day,)

[ricklhp7]

I dont think you are getting banned for that.

Last season they weren't flagging accounts to ban in a wave, they were banning in few hours, at least for me.

I have my search requests stored in a database, so its easy to me to limit that. For that reason I found out how temp bans work.

Between bids I think the minimum time is 1.1s, but when you search the market you can bid as soon as you can, but I dont recomend it, they ban pretty fast if you do that.

About the library. The OPTIONS headers are different from the request headers, shouldnt we implement it?

[ricklhp7]

In if 'var redirectUri' in rc.text: self.r.headers['Referer'] = rc.url rc = self.r.post(rc.url, {'_eventId': 'end'})

Isn't it a get request?

[ricklhp7]

OPTIONS requests dont carry any personal info, I dont know if Im going to use it. Its not hard to fix the headers, but seems useless.

@oczkers are you using it? Do you think they can track us for not using it?

[oczkers]

@randomsecguy

No, i don't have any changes stashed, just merging pull requests and i'll be releasing new version in a few minutes :-)

The OPTIONS headers are different from the request headers, shouldnt we implement it?

You mean x-ea-game-id etc.? We could get rid of this before making options but do You think it matters? If they want to track it they would have add new code checking if this specific params are sent here and attach this to specific account (no personal data is sent).

Isn't it a get request?

Fixed :-)