TestApp mini-project

[ThePokerDude]

Hi,

I'd like to give you some update about my progress with the test app (I'm at the very beginnig of this project). I use python and SQLite.

1. Running the actual test

Thx @Thorvald. In https://github.com/EdwardPiwowar/BBA/issues/888 you'll find a description

2. DB structure (see BBATesterDBcreate.py) It's 3 tables

Bidding_Test (for storing basic information about the test which was run) ID integer PRIMARY KEY AUTOINCREMENT, BBA_Version integer, NS_CC text, EW_CC text

Board_Evaluation with correct final contracts to compare the test with BBA_Board_Number varchar(28), Vulnaribility text, NS_CC text, EW_CC text, Player_Site text, Contract text, PASS_X_XX text, Score_Evaluation integer

Test_Bidding_Results for the results of the current test Test_ID, BBA_Board_Number varchar(28), Player_Site text, Contract text, PASS_X_XX text

The SQL for joining the 3 tables:

c.execute("select \ BT.ID, \ TBR.BBA_Board_Number, \ BT.NS_CC, \ BT.EW_CC, \ TBR.Player_Site, \ TBR.Contract, \ case \ when BE.Score_Evaluation is not null then BE.Score_Evaluation \ else '0' \ end as Score_Evaluation \ from Bidding_Test BT \ join Test_Bidding_Results TBR on BT.ID = TBR.Test_ID \ left join Board_Evaluation BE on TBR.BBA_Board_Number = BE.BBA_Board_Number and TBR.Contract = BE.Contract and TBR.Player_Site = BE.Player_Site")

I need to think a little bit about the SQL which actually compares results of different BBA versions. It should check only results if the used convention cards were the same.

3. parsing the test result PBNs (PBNParseAndWritIntoDB.py) Here I'm at the beginnig but it workes quite well. The currect version parses only for Test_Bidding_Results. The parsing for Bidding_Test is much easier and I'll implement in the next days. Also I did not yet implement writing into the DB. During the project I'll improve it (include some tests) as at the moment I'm blindly trusting that the PBN file is complete and correct.

file = open('test.pbn', 'r') read = file.readlines() modified = []

for index, line in enumerate(read): if line.startswith('[Board'): # No need to index with i

Check if the next line exists before accessing it

    if index + 1 < len(read):
        modified.append(read[index + 1][2:].strip())
elif line.startswith('[Dealer "S"]'):
    modified.append('NS')
elif line.startswith('[Dealer "N"]'):
    modified.append('NS')
elif line.startswith('[Dealer "E"]'):
    modified.append('EW')
elif line.startswith('[Dealer "W"]'):
    modified.append('EW')
elif line.startswith('[Contract'):
    contract_value = line.split('"')[1]
    modified.append(contract_value)
    if "XX" in contract_value:
        modified.append("R")  # If it contains "XX"
    elif "X" in contract_value:
        modified.append("D")  # If it contains "X" but not "XX"
    else:
        modified.append("P")  # If it does not contain "X"
elif line == "\n":
    modified.append("\n")

print(modified) BBATester.zip

[EdwardPiwowar]

Fine, can you already attach a sample database?

[ThePokerDude]

Hi, technically yes. SQLite creates a .db file which behaves like a real SQL database. In one of the attached files there are the table create commands. The Board_Evaluation table will contain all the sample hands with a score. For example if in a hand the best contract is 6n - it would have a score of 10, if 6c is the 2nd best it would have an 8 for example. So I'd import:

BBA_Board_Number | Vulnaribility | NS_CC | EW_CC | Player_Site | Contract | PASS_X_XX | Score_Evaluation 0621BC3F900EE2E4357B78964476 | NS | Sayc | Sayc | NS | 6N | P | 10 0621BC3F900EE2E4357B78964476 | NS | Sayc | Sayc | NS | 6C | P | 8 0621BC3F900EE2E4357B78964476 | NS | Sayc | Sayc | NS | 5N | P | 6 0621BC3F900EE2E4357B78964476 | NS | Sayc | Sayc | NS | 5N | P | 6 0621BC3F900EE2E4357B78964476 | NS | Sayc | Sayc | NS | 5N | P | 6 0621BC3F900EE2E4357B78964476 | NS | Sayc | Sayc | NS | 5C | P | 4

The evaluation is of course a little bit subjective.

If the actual test is run, we would store the BBA version and the CC information in Bidding_Test for example for 3 consecutive tests with 8643, 8644, 8645:

ID | BBA_Version | NS_CC | EW_CC 1 | 8643 | Sayc | Sayc 2 | 8644 | Sayc | Sayc 3 | 8645 | Sayc | Sayc

and the test results in Test_Bidding_Results: Test_ID | BBA_Board_Number | Player_Site | Contract | PASS_X_XX

1 | 0621BC3F900EE2E4357B78964476 | NS | 3C | P 2 | 0621BC3F900EE2E4357B78964476 | NS | 5C | P 3 | 0621BC3F900EE2E4357B78964476 | NS | 6C | P

We then can query for this hand and the used convention card so we would see the progress from version to version. 8643 would have 0 points as ther is no hit for 3c in the Board_Evaluation table. 8644 would have 4 points and 8645 would have 8 points.

So if the technical part works, we nee to create a list of hands with final correct and acceptable final contracts and assigned points to it.

[ThePokerDude]

I will change Board_Evaluation add a column: Topic. remove the column Vulnaribility and PASS_X_XX (redundant) and create a new table Board_Information BBA_Board_Number, Vulnaribility, NS_HCP, NS_Spades_Count, NS_Hearts_Count, NS_Diamonds_Count, NS_Clubs_Count

This it will be easier to analyze the progress by topic (for example slam bidding, Transfer, Stayman etc. With the Board_Information table it will be possible to analyze if fits are found, games are missed even if HCP is sufficent, opponents can play undoubled with very few points and so on.

[ThorvaldAagaard]

When having multiple files, it would be nice to have a key, so you can select a subset of deals. Like being able to test a single convention with out to have to have a database pr. convention

[ThePokerDude]

That true. This what I intended with adding the column topic. But we could add an additional column "Convention" So in we would like to retest for this convention (for example stayman). As BBA test works perfectly fine with an input file which contains BBA board numbers for a Stayman test we could create a test file by extracting all boards for this convention (select BBA_Board_Number from Board_Evaluation where Convention = 'Stayman').

So the DB structure would be like this:

c.execute('''CREATE TABLE Bidding_Test ( ID integer PRIMARY KEY AUTOINCREMENT, BBA_Version integer, NS_CC text, EW_CC text )''')

c.execute('''CREATE TABLE Board_Information ( BBA_Board_Number varchar(28) PRIMARY KEY, Vulnaribility text, NS_HCP integer, NS_Spades_Count integer, NS_Hearts_Count integer, NS_Diamonds_Count integer, NS_Clubs_Count integer )''')

c.execute('''CREATE TABLE Board_Evaluation ( BBA_Board_Number varchar(28), Topic text, Convention text, NS_CC text, EW_CC text, Player_Site text, Contract text, Score_Evaluation integer )''')

c.execute('''CREATE TABLE Test_Bidding_Results ( Test_ID, BBA_Board_Number varchar(28), Player_Site text, Contract text, PASS_X_XX text )''')

[ThorvaldAagaard]

I think you should store the bidding sequence and not just the contract. Then it would be possible to find deals matching a specific bidding sequence.

I am creating a pickle (in memory DB) with all sequences from the training and I will use that (work in progress) as stats for hands.

As an example I found that after 1N-P-P-P responder could have an 8-card minor So here it would be interesting to find all boards with the bidding 1N-P-P-P

[ThePokerDude]

Ok, I'll implement storing the bidding sequence, too. I'll add a column in Test_Bidding_Results (have to check which data type: BLOB or text)

c.execute('''CREATE TABLE Test_Bidding_Results ( Test_ID, BBA_Board_Number varchar(28), Bidding_Sequence, Player_Site text, Contract text, PASS_X_XX text )''')

and add a new table

c.execute('''CREATE TABLE Full_Bidding_Evaluation ( BBA_Board_Number varchar(28), Topic text, Convention text, NS_CC text, EW_CC text, Bidding_Sequence, Bidding_Sequence_Score )''')

and I'll rename Board_Evaluation in Final_Contract_Evaluation

@ThorvaldAagaard for the Final_Contract_Evaluation do you think we should differentiate for different NS and EW convention cards? The idea with the score is from bidding challenges you find everywhere in bridge magazines. There they have one score set for a board. For Full_Bidding_Evaluation I think the convention cards are essential as the way to the final contract highly depends on the system used.

Now that I mentioned bidding challenges, I think the information about the source of the board is useful like "Bidding challenge magazine XY from December 1999" or "Computer Match BEN vs WBridge 2024".

I will either add a new column in table Board_Information. If the "event" field is not empty I could extract the information from there. If it's empty it has to be inserted manually. Or I'll create a new table containing the source information. Not sure which one is better.

[ThePokerDude]

I made good progress an the test app is usable. I'm a rookie programmer so I did the app with the help of ChatGPT o1 - so the code might be not super good, but it runs and does what it shall do :)

Steps for test: 1) run Thorvalds RunBBA.cmd and create a pbn output file with archive setting "as a.pbn file (room1) 2) run BBATesterDBcreate.py to create a SQLite DB and all tables 3) run BBABoardInformation.py to parse the created output.pbn files and store general board information into the DB 4) run BBATestResultImport.py to parse the created output.pbn files and store the test results like (full bidding, final contract ..) 5) run ReportExample.py (first report created)

ToDo import board evaluations (correct final contract, correct bidding, description (topic, convention) improve reports

First findings based on 601 issues (pbn was once created by Thorvald)

version1 / version2 / different final contract / different bidding 8643 vs 8662 / 143 / 184 8662 vs 8663 / 68 / 102 8663 vs 8667 / 1 / 2

So we see that changes in version have huge influence on a lot of boards. At the moment I see only a huge variation but as I did not import yet the evaluation (will be more or less manual) I cannot tell how much of the diffrence is an improvement and where the model git worse. 1st release.zip

[ThorvaldAagaard]

ChatGPT is a fine programmer, but sometimes we are not clear telling it what to do :-)

I recommend you create a repository on Github with you application, let me know if you need help.

When I train on latest version I have 1.1 mill deals, and after bidding those I remove bad scores, where the result is far away from par, and I remove boards, that was doubled and making

These 2 (1400 and 20.000 boards) files contain boards, where almost any change will be for the better, and then I have the rest, that should only change after an issue has been corrected.

[EdwardPiwowar]

Thorvald, send me these two tables, maybe I can find a rule to improve them.

[ThorvaldAagaard]

Due to the size of the files we can't save them in GitHub, so I have created a directory on my FRP-server

Use FileZilla with these settings:

In the folder Training8663 you can see these files:

All.PBN is the interesting deals collected in the BEN-repository GIB-Thorvald-8663.pbn is All.PBN bid by BBA GIB-Thorvald-8663-Optimum.pbn is GIB-Thorvald-8663.pbn with Optimum scores added by Bridge Composer

Using https://github.com/ThorvaldAagaard/Bridge-Robot-Utilities/blob/main/src/Split_PBN.py the deals are examined, and split into three files (4 but doubled_not_making is just for stats)

GIB-Thorvald-8663-Optimum_OK.pbn are the files used for training - each bidding sequence will be max 30 times db_making.pbn are deals discarded because it was doubled and making db_not_making.pbn is just to get a count of how many deals was doubled not making and is included in the training disaster.pbn are boards discarded because the score is to far from the PAR-score - some of these are grand going down even though it might be the correct contract, and I am working on a better selecting

So the most interesting is disaster.pbn

[ThePokerDude]

@ThorvaldAagaard I will create a github repo. Probably the weekend after next. Your help is highly appreciated. As soon as I have set it up, I'll come back at you.

@EdwardPiwowar @ThorvaldAagaard If you have any request for the regression tester, just let me know.