BEN Versioning

[ThorvaldAagaard]

Bjorn Hjalmarsson is referring current version of BEN as 11.25 - I can't see that version number anywhere, so with the updates, what would the new version of BEN be, and where is it maintained?

[lorserker]

ben has only one release (version 0.0.1) https://github.com/lorserker/ben/releases

i think it's about time to make a new one because there are quite a few new PRs merged since then. let's do it after we merge your PR

[ThorvaldAagaard]

I wonder where Bjorn has that version number from, but it can also be seen on his site: https://www.computerbridge.se/ben/

[lorserker]

i don't know. looks like a date. November 25th maybe :)

[ThorvaldAagaard]

Probably, so you will have to decide what the new Version should be :-)

[lorserker]

i think i'll go with 0.0.2 many zeros in front so people don't expect too much:)

[ThorvaldAagaard]

BEN is not that bad. 0.2 sounds better. On the other hand you have a release in the wild, so you could also use 1.1 :-)

Whatever you decide I think we should add it to the html and gameserver, so people will see it.

[eamongalligan62]

Ben1125 is a version of Ben with 11.25 million iterations using about 3 million hands trained by Eamon Galligan using all the hands since I started training Currently I have a new one using 2.5 million different hands but only conventions familiar to GIB and currently at 4.7 million iterations only bidding of course .. Current match sees Ben grimly holding onto a lead of 18 imps against WBridge5 level 3 after 94 boards I use samples like Thorvald with low acceptance in bots and sample.

[larrycai]

related with package release as well #74 and PR #75

Later we may have

• model release (could be in hugginface.co)
• code/package release (in github)

Then in competition, it will be stated like

• software - ben-v.0.2.0
• model bidding - ben-module-bidding-3M-v2023.1106 (3M is 3million hands/data, v2023.1106 is CalVer or we choose v0.1 semver)
• model playing - ben-module-playing-3M-v0.1

model version example: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1

[ThorvaldAagaard]

BEN is using 8 different models, and which models to use, is read from the configuration file.

When using huggingface is the idea then to load the models directly in the software from hugginface, or should the models be included in the build?

I have created the first of the models on: https://huggingface.co/AIBridgeEngine/single_dummy

[larrycai]

we can do it step by step

• Step 1: more stable release in hugging face, like #76 , needs time to master all, we can still manually copy them into ben git repo, no changes here
• Step 2: we can have extra step in github action to download from huggingface before building
• or Step 2: multistage build inside container build process, we download the models inside Dockerfile
• Step 3: Downloading from huggingface is also a good choice if the model goes bigger and ben engine can be pure engine without model. we can even release to version (ben:0.3.0-all-in-one, ben:0.3.0, it is common pattern as well)

[ThorvaldAagaard]

The models are pretty stable in size about 5M each, but all will probably be about performance, as there are many calls during a play. So a solution, where the model is downloaded either in the beginning or when building seems the best.

Using Huggingface.io thru API during play should only be used in examples of how to use the model

[ThorvaldAagaard]

I have decided, that when we get to UCBC 2024 the version will be 2.0.

At that time there has been model changes and we will be using PIMC for declaring. Compared to other software versions that seems OK for me.