search

saltbot-org / saltbot

automated virtual betting bot
GNU General Public License v2.0
52 stars 45 forks source link

Problem Importing the Records #106

Closed KnightSickle closed 6 years ago

KnightSickle commented 8 years ago

So, im pretty sure of whats happening, the bot is not loading the records i am importing. When both Cowboy or Scientist try to make a bet the console displays that they dont have enough information on any character, have tested this on multiple matches and they always display:

PROBLEMS:

This causes the Scientist to only bet around 10$ per match, and the cowboy 400$. Both always on blue. After this began happening i have reinstalled both the extension and chrome and this are the steps i have been taking every time i try to use the bot.

1-Open SaltyBet, and login 2-I open the Bot and try to import the records from the GitHub archive SampleData_Mar_13_2016.zip. Nothing appears on the console after i have done so 3-Try to Import the Chromosome from the same archive as in Step 2. Again, nothing new is displayed on Console. 4-Tried Cowboy strategy first. It displays the Problem i mentioned above 5-Tried Scientist strategy. It displays the same problem.

Additional Info: 1-I am NOT importing the .zip file, i am importing the .txt records and the chromosome which are inside this .zip file. 2-The bot IS working, the Lunatic makes its bets fine and monk seems to be recording data 3-Both Cowboy and Scientist were working fine hours ago. I left the bot on during the night and something very unexpected happened, it crashed my computer, I cant say for sure if it was the bot but when i woke up everything on my computer stopped working and i had to give it a Hard Reset, i then reinstalled everything altough nothing seemed to be out of place after the crash. 4-I have uninstalled all other extensions, altough a few hours ago when everything was working fine those extensions did not seem to be a problem.

I am unsure if this is related to the bot at all but before i do anything with the bot but while i am at saltybet this is displayed at the console:

Failed to load resource: net::ERR_INSECURE_RESPONSE global-ec133f4….js:17 JQMIGRATE: jQuery.browser is deprecated chrome-extension://hfaagokkkhdbgiakmmlclaapfelnkoah/cast_sender.js Failed to load resource: net::ERR_FAILED global-ec133f4….js:29 No cast extension found 2global-ec133f4….js:29 document not ready yet, trying again in 500 milliseconds... (program):1 build: local emberchat-87bfdff….js:111 Uncaught SecurityError: Blocked a frame with origin "https://www.twitch.tv" from accessing a frame with origin "http://www.saltybet.com". The frame requesting access has a protocol of "https", the frame being accessed has a protocol of "http". Protocols must match.

salty.js:431 - changing strategy to Monk

One additional question, while i was browsing the comments for a solution i saw people mentioning a seed.zip file which was supposed to contain the records, altough the comments seemed to be old. I am importing from "SampleData_Mar_13_2016.zip", which is on the GitHub page, am i using the correct file? Also the file "65k records without exhibitions June 2016.txt" is not downloadable for it is too big, i have copied its contents to a .txt file inside my PC and tried to import it like that after the other import from the .zip file stopped working. I am unsure if that would be right way to do it but it presented the same problems as i mentioned above (the bot doesnt seem to load the file at all).

reconman commented 8 years ago

You can download the 65k match file just fine from https://raw.githubusercontent.com/synkarius/saltbot/master/65k%20records%20without%20exhibitions%20June%202016.txt

Right click and select save as.

There was a similar issue where everything regarding the Chrome storage suddenly didn't load anymore. The first 2 messages from saltbot after the site is loaded should be "settings applied" and "settings upgraded".

Try reinstalling Chrome, that fixed the issue for the other person.

KnightSickle commented 8 years ago

Tried reinstalling it again and the problem persists. The message "settings applied" and "settings upgraded are there", but this happens before they appear:

http://imasdk.googleapis.com/js/sdkloader/ima3.js Failed to load resource: net::ERR_BLOCKED_BYCLIENT http://imasdk.googleapis.com/js/sdkloader/ima3.js?=1469404069407 Failed to load resource: net::ERR_BLOCKED_BY_CLIENT

Also, a lot of other errors happen after that, but they might not be related to the bot, but to adblock.

Do you know where this Chrome Storage is located? Where exactly is it? Do you think theres any way in which i can load the resources by any other mean?

reconman commented 8 years ago

When you open the console click on the filter button which looks like a funnel. Then check "Hide network messages". With that you should be able to block most of the irrelevant error messages regarding web elements blocked by adblockers and similar stuff.

The chrome storage is just some file hidden in AppData. When you open it there's just unreadable data in it. That's what chrome uses to store the data of an extension.

I couldn't reproduce your problem at all because either it displays 0 records imported or x records imported. The only way how it can log nothing is when the local storage can't be written to.

Try changing some settings and check if they are the same if you reload the page. For example change the aggro limit.

KnightSickle commented 8 years ago

Yes, i can change certain settings (Bet on exhibitons, Agressive betting, all of them), what i cannot do is import both the records nor the chromosome, i actually CAN in fact export the records and export the chromossome, so i assume that the bot is storing the matches data somewhere it just wont import the ones im trying to. I have also tried importing the data that i have exported and it still doesnt seem to work, nothing appears on the console.

After i hid the network messages, some errors still appeared:

salty.js:301 - skipping team match salty.js:431 - changing strategy to Cowboy salty.js:441 - settings saved strategy.js:161 - Cowboy has insufficient information, W:L(P1)(P2)-> (0:0)(0:0) strategy.js:124 - betting without confidence: 405 salty.js:136 - match result code: c1:Frieza EX3|c2:Vegeta EX3|w:1|s:rc|p:a|t:A|m:e|o:3:1|t:117 salty.js:301 - skipping team match salty.js:198 - number of:: chars: 2, matches: 1, bettors: 354 salty.js:209 - records saved, matches this cycle: 3 global-ec133f4….js:16 Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user's experience. For more help, check https://xhr.spec.whatwg.org/. global-ec133f4….js:17 JQMIGRATE: jQuery.browser is deprecated Failed to execute 'postMessage' on 'DOMWindow': The target origin provided ('https://passport.twitch.tv') does not match the recipient window's origin ('http://www.saltybet.com'). global-ec133f4….js:29 No cast extension found emberchat-71c13d4….js:111 Uncaught SecurityError: Blocked a frame with origin "https://www.twitch.tv" from accessing a frame with origin "http://www.saltybet.com". The frame requesting access has a protocol of "https", the frame being accessed has a protocol of "http". Protocols must match.

2salty.js:301 - skipping team match

After this no more errors appeared.

Im starting to think it might have something to do with the permissions on my computer but i have nothing to back that up, just a guess. I have tried googling for problems with local storage and found nothing that could help me.

I have now tried manually deleting every single data from chrome after uninstalling it (It would still keep some data at AppData) but it still wont work.

Again, is there really no way i can get the bot to read these files by not using the UI? Is there any way i can work the bot in other browser? Maybe another version of google chrome?

reconman commented 8 years ago

Messing with the storage files probably won't affect anything positively.

All those error messages look pretty normal to me because you get them even without the extension.

I want you to try one thing:

What do you see?

What you should see is:

Attempting records import... popup.js:105 Upload successful. popup.js:108 Attempting to read file... popup.js:92 File read successful.

KnightSickle commented 8 years ago

Ok, this is going to sound crazy but somehow, doing THAT made it work. First of all, after doing what i said i had done on my previous post, i did nothing else to try to fix it. Second, before trying what you told me to do i had the bot open on cowboy settings for a few hours (It keep placing bets of 400$ on blue with no confindence). This is what i did when it worked:

1-Inspected the Import button on the bot and went to console 2-Tried to import the resources file, It happens to appear exactly what you said it should. 3-Went to check the console of the entire page (Without inspecting anything) and it appeared: records imported: 59068 match records 6295 character records 4-Tried the same with chromossome (Inspected the import chromossome button, checked its console and tried to import it) and it worked as well.

In doubt, i double checked it, closed all chrome instances and tried it from scratch. WITHOUT TRYING TO INSPECT ANYTHING i tried to import the records AND IT DID NOT WORK (Nothing appeared on the console), same with the chromossome. I then tried the same process that made it work, the same way i said i did above and it Worked again, both for the records and for the chromossome. I have NO idea at all why it works like that, but so far it does. This is what i am getting on the console now on Scientist:

» PREDICTION STATS for (Kamek EX3 VS Sailor mars) » strategy.js:501 » WINS/LOSSES: weighted totals as % (red:blue) -> (100 : 0) :: unweighted (red W:L)(blue W:L) -> (7:8)(2:13) :: details (red W:L)(blue W:L) -> (BBSBBBB:BBBBBBBB)(BB:BBBBBBBBBBBBB) strategy.js:505 » predicted odds -> (47822227.35 : 1) :: avg win time (red:blue) -> (142 : 177 ) :: avg loss time (red:blue) -> (122 : 107 ) strategy.js:510

strategy.js:47 - betting at level: 2, confidence: 99.99 salty.js:209 - records saved, matches this cycle: 7 salty.js:136 - match result code: c1:Kamek EX3|c2:Sailor mars|w:0|s:cs|p:t|t:B|m:m|o:4.8:1|t:117 salty.js:198 - number of:: chars: 6295, matches: 59074, bettors: 992

I assumed that that means its working (The bot is betting different amounts of money now on someone other than blue). (The confidence and predicted odds are changing from match to match)

One thing tough, i sometimes get huge amounts of lag, with a small computer freeze when a new match begins (When it tries to calculate the odds), it does not happen everytime but this appeared on the console one time when it happened:

global-ec133f4….js:15 Handling of 'wheel' input event was delayed for 1007 ms due to main thread being busy. Consider marking event handler as 'passive' to make the page more responive. VM42 extensions::uncaught_exception_handler:8 Error in event handler for runtime.onMessage: TypeError: Cannot read property 'push' of null at chrome-extension://bholoegapebhflljekancpcnajigaiih/salty.js:497:13

It doesnt seem to happen if the saltybet page is minimized or if i am on another tab tough, but is it supposed to happen? Also, thanks in advance for all the support you have provided so far.

reconman commented 8 years ago

That is really a strange behaviour.

The lag while calculating who to bet on is normal but I'll consider moving the calculation to a different thread with the Web Worker API. That way users don't notice that calculations are made.

Regarding the last error message with push it seems that saltbot couldn't handle Waifu4u's chat message about who is fighting who. I don't know how it can happen right now.

reconman commented 8 years ago

Sadly I have to postpone the optimization since it's all NEW tier battles right now.

calexil commented 7 years ago

This behavior is persisting, most times the data will not import unless the import button is right clicked and inspected.

pls investigate

ACIIL commented 7 years ago

calculation to a different thread with the Web Worker API.

I sorta been looking into ways to split the heavy work off the saltyBet page and popup. Perhaps moving bet prediction and simulation onto background page's is the solution? Then we only need to query from them.

calexil commented 7 years ago

anything to speed it up.. I have noticed it is now missing bets cause of how long the record sort/search takes

ACIIL commented 7 years ago

EDIT: i am wrong about the concurrency issue with character info.

It's unlikely missing bets is from the new match prediction run loop. I did a search for what could cause that and will explain findings.

Assuming you are using scientist mode, there is only a confidence function called. It uses the characters stats, which only remembers results of the last 15 matches of that character. These character scores are saved and recalled in memory by the character name. When ether new finished matches/Imported matches/new simulator loops are called, they will perform character stat updates (in records order) and stop.

I realized looking at the code now; connecting the characters stat tokens into both sim running and live match prediction is a horrible concurrency nightmare. For example, sim is running, its looping and testing match prediction for the chromosome, AND it will update character stats with the current match record tested; Character stats could be anywhere in history now, it could be locked and prevent betting for the moment, i do not know.

Its likely you are running the sim at the same time, which is constantly using and pushing new data into these character stats. There is undefined locking/priority here. Don't run the simulator when your priority is betting.

Following that, character records could be optimized if the standard javascript (key:value) -> ("character name":character_stats_object) methods are becoming slow by size. This area should be looked at by someone who knows better about it.

In your case, If the bot is not betting when the sim is not running, did you refresh/entered the page and enter a match? I think a whole match loop is needed before the system sets in. Did the console log report disconnects in this time or dropped twitch chat or failed SaltyBet UI updates?

calexil commented 7 years ago

I have never run a sim, I dont even know how

ACIIL commented 7 years ago

you press "update genetic weights" to run the sim.

ACIIL commented 7 years ago

Directed to @reconman or @synkarius

I think i am wrong now, because in the sim there is a read for the matches. chrome.storage.local.get(["matches_v1", "characters_v1", "chromosomes_v1"], function (results) {

Here only "matches_v1" are imported, and characters are built in a new "info" structure.

    var info = {
                "character1": updater.getCharacter(matches[i].c1, characterRecords, namesOfCharactersWhoAlreadyHaveRecords),
                "character2": updater.getCharacter(matches[i].c2, characterRecords, namesOfCharactersWhoAlreadyHaveRecords),
                "matches": results.matches_v1
            };

Which uses reference for characterRecords and namesOfCharactersWhoAlreadyHaveRecords in this function.

Characters are updated by: updater.updateCharactersFromMatch(matches[i], info.character1, info.character2);

And there is no write back to storage later. So These Characters stats live in structure "info" and independent of character stats used for live match prediction.

calexil commented 7 years ago

ah

ACIIL commented 7 years ago

Found a more probable reason why match character prediction is slow: In tracker.js

recswill hold a record of ALL characters. Its a javascript style array. Here the code attempts to do a COMPLETE ARRAY search for the character names and it does not break out early. There are more optimal ways to do this. For reference, characters in my database are +7k now.

//Attempt to get character objects from storage, if they don't exist create them
    chrome.storage.local.get(["matches_v1", "characters_v1", "settings_v1"], function (result) {
        var self = s;
        var baseSeconds = 2000;
        var recs = result.characters_v1;
        self.multiplier = result.settings_v1.multiplier;

        //self.fillCharacters(result);//get character record objects or make new ones
        if (recs)
            for (var i = 0; i < recs.length; i++) {
                var c = recs[i];
                if (c.name == self.names[0]) {
                    self.character1 = c;
                }
                if (c.name == self.names[1]) {
                    self.character2 = c;
                }
            }

Also given javascript uses hashtables for everything, it might be possible to do just a direct name check.

reconman commented 7 years ago

Thanks for the hint, the run time for this part can be improved by using a sorted list with binary search. I'll take a look at it tomorrow or on Sunday.

ACIIL commented 7 years ago

I'm sure if characters_v1 was a Character_name=>character_data pair instead of array_index=>character_data pair, it would be much faster for lookup and maximize how it is normally used.

Irrelevant side-note, matches_v1 changed from a index=>data pair, to a date=>data pair might be useful in other ways. I never found the date stamps correct enough to sort with. It is a shame it was not year-month-day-hour-min, as here it can help with historical match ordering and testing samples.

reconman commented 7 years ago

For character data a Map would help but for the matches it would not work. Accessing matches with a date string does not provide any benefits since the matches array is sorted anyway.

The date format was introduced by me 1 1/2 or 2 years ago. Before that the date was not logged at all so you had no idea on which date the matches were recorded. This was done way before the match window existed so there was no requirement for storing the exact times of the matches.

I could probably add the time to newly recorded matches and use 00:00 if the time is not set.

ACIIL commented 7 years ago

I see, and thank you for the history lesson @reconman. I found the current time format did not sort correctly when sorted by string, it relied on spreadsheet parsers reading the time format correctly (which is a nice feature when it exists). If the time format became year-month-day-etc, it would work nicer when used with dumber tools. I understand if the time is too precise, like to seconds, it may be overkill.

reconman commented 7 years ago

I implemented the binary search and insert in https://github.com/synkarius/saltbot/commit/6c297849940f979ec946ea83916446badfc34ef0.

calexil commented 7 years ago

confirming that data import is pretty much instant now, as is record import.

even on my pos celeron betting box, records are now pulled in a couple seconds

calexil commented 7 years ago

hmm, this is more of a feature request issue now... a verbose notification that the import was successful would be key

156

calexil commented 6 years ago

closing and reopening a more specific instance occurring in the new indexedDB in 2.5