[eFEAT] Data about new league

[akasolace]

See lot of users using an old tool that predict league for next season based on ranking. Maybe we could have this somehow in HO and/or maybe simply in a static webpage

@tychobrailleur I keep here for reference a list of implemented endpoint:

[GET]

• .../league/supported => returns list of all supported leagues

• .../league/{leagueID}/status => returns status of given leagueID

• .../league/{leagueID}/next-block?accept-job={true/false} => returns the next batch to be treated, if accept-job is set to false, the block is not set to in-progress on the server

• .../league/:leagueID/team/:teamID/pd-status => returns a tuple (pmStatus, [oppTeamIDs])

[POST]

• /league/:leagueID/block/:blockID/push/ => push one block to server via json

w/ json like {"username": "john Doe", "data": [{"teamId": 73868, "Score": 970050050000000}, {"teamId": 280009, "Score": 960050050000000}, .....

[mistermax80]

Which is it the tool?

[akasolace]

Which is it the tool?

hattrick promotion manager

[cyrill10]

Hey I'd like to contribute and get some Open Source Experience and doing that with a tool like HO would be great. Maybe this would be a great issue to start? I have 5+ years of Java experience.

[akasolace]

@cyrill10 Hello cyrill, welcome to the project. I already started to work on that issue. I see 3 streams of work on this one: 1) gather, compute and prepare data (json files). I started that part and I already made good progresses (let says it is half done) Python 2) have the json file stored on HOserver and create required request (PHP). help of @mistermax80 would be more than welcome on that one 3) prepare the new tool (panel) in HO that will allow user to fetch the required information (Java). I think this part will be fairly straightforward and could be a really good entry point in the project. If that is ok with you @cyrill10 I can leave you that part :-) Let me know what you think

[cyrill10]

Sure that sounds good. I'll start with the UI then. Let me know when the HOserver is ready for the request.

[akasolace]

@cyrill10 @mistermax80 I have terminated the prototype in python ... it works quite well. Now it needs to be ported from python to PHP and JAVA. At the moment, it works like this: 1) data are downloaded from HT and stored locally 2) data treatment / calculation is done 3) all requests are made on the data obtained from 2)

Hence, step 1) should ideally be done only once per league and per season. We do not want to saturate HO servers. At the moment my python code, download all the data it takes approx 5 minutes per league (for big league) => It requires all download to be done by a single person (that is not ideal). Alternatively, we could share the downloads among HO users i.e. if a user request data which are not available he is invited to download them first. Once the data are downloaded they are sent to the server and his request is sent again. (we can think of having the user download all the data for his country or only for the required division)

step 2) will be easy to port them from python to java

step 3) will be easy to port them from python to PHP

In summary, we will need I think the following: PHP
1) implements function to post data in the database ( 1 table with teamID, team name, calculated score, league name). potentially 1 row for each team in hattrick (~200k at the moment). 2) implements function to get data from the database

JAVA 1) download data from HT, prepare them and send them over to the database 2) make a simple GUI panel in tools modules (I guess)

The logic/calculation of the matching team ID can be done either in JAVA or in PHP. I have no preference, considering that it should be done only once per request, I think it would make more sense to have it server side (otherwise lots of request would be useless)

I stored the data for 3 leagues, hence we will be able to test code before next interseason (if we are ready before that :-) ).

The python code POC

to be discussed

[akasolace]

I checked Hattrick-PM, considering the performance and the overall config, I believe he used the same approach as I did. However, considering that we will cache HT result, HO experience will be much better

[mistermax80]

If I understand well, by java we can check if the data for that team id is already present, else it will do a request towards HT and download data and send to server. I don't know how to the prediction. Could you explain me?

If we do other requests to server, i think the code for connecting to server we should insert in a only point of the code, else we duplicate the code, it is not nice...

[akasolace]

If I understand well, by java we can check if the data for that team id is already present, else it will do a request towards HT and download data and send to server.

No the request will be made to server, a request like this (from my python code) where 1165439 is the team ID:

season = getSeason()
getStatus(season, 1165439)

if (season, teamID) not present in the server it will return a specific message (l 265 in my python code print(f"data for {leagueName}, season {season} are not available !")``) and HO will then offer the user the possibility to download the data

If we do other requests to server, i think the code for connecting to server we should insert in a only point of the code, else we duplicate the code, it is not nice...

agree

I don't know how to the prediction. Could you explain me? The prediction logic is quite simple, if you look at my code, it is all in function getStatus2(). Basically team have an hidden ranked (not the one displayed) and based on that rank there is a matching made strongest <-> weakest for qualification game and strongest <-> strongest for promotion

[cyrill10]

I looked at your example and would suggest we calculate it per country on user request, and store it on the HOserver once calculated. this way only one user per country needs to do the calc locally.

I started with the implementation in Java and think it shouldn't be that complicated, But I was wondering if there was a way to get all the existing leagues per country (or just the first for each level to get the others similar to your python code). Couldn't find anything in the CHPP API. Will we just build a Lookup Table or do you guys have any suggestions?

[akasolace]

I looked at your example and would suggest we calculate it per country on user request, and store it on the HOserver once calculated. this way only one user per country needs to do the calc locally.

yes this is what I had in mind. Only thing we need to consider is that it is more or less exponential. Hence, country with 5 divs would take 1 minute, with 6 divs 2 minutes and after that might double for extra league, so it can be fairly long for 8 divs country. I am not familiar with threads in Java, but could be nice if we do that as non blocking thread.

I was wondering if there was a way to get all the existing leagues per country (or just the first for each level to get the others similar to your python code). Couldn't find anything in the CHPP API

no, unfortunately, nothing available via CHPP. Also, if you looked at my code, you saw it works on the assumption that all league ID are incremental within a given country, league level. I found that it is not the case for Italy level 8.

Will we just build a Lookup Table or do you guys have any suggestions?

I see it 2 ways:

1. we store it on the server and only us are able to update it (for missing country, we can ask on the forum user to open a request on github while providing the information in format easy for us to paste server side)
   2. if data does not exit on server, we ask user to provide the information via HO, at request time

in both case I am not sure how to treat case like Italy level 8 (and I guess it will happen somewhere else as well). Instead of 1 number, I don't know if we will need 2 or one thousand. I did not check.

[cyrill10]

I did some digging for those IDs and found some helpful infos: There is currently 258'476 League ID they seam to be consecutive for all level below 8. Currently the only country that has a level 8 is italy. There the IDs are consecutive from VIII.1 to VIII.1024 and after a gap they are consecutive again from VIII.1025 to VIII.2048.

With those infos I'm writing a little Python script to create the lookup table for all leagues that we can put on the HOserver. I will upload the script once I'm done. I'll let the script create a SQL file that creates and fills those tables on the HOserver. Any preferences on how those Tables should look like?

I'd suggest that if the HOserver does not know the new league of the requested team he returns a list of all league IDs of the country so HO can actually determine the new league and send it back to HOServer.

[mistermax80]

The server is in php, so you can write directly code in php, and use the command line functionality, in additional the server is based on yii2 framework, it is useful for every needs.

Anyway, the server can expose the API rest for receive data from external, its dB is not directly accessible from external.

So if you want write code in python, you will send data to server by apirest.

Let me know what you think!

[mistermax80]

Furthermore, if you want write SQL code for initial infasament, it is ok the your way.

Could you send me the table structure? We should consider also the seasons.

[akasolace]

I'd suggest that if the HOserver does not know the new league of the requested team he returns a list of all league IDs of the country so HO can actually determine the new league and send it back to HOServer.

@cyrill10 not sure I understand this one ... but looks like you know what to do :-)

[cyrill10]

Furthermore, if you want write SQL code for initial infasament, it is ok the your way.

@mistermax80 As it would only be a one time import to the server I guess inserting the SQL would make sense. The list will also include the currently deactivated higher leagues so it should be relatively stable.

Could you send me the table structure? We should consider also the seasons.

I will send it to you once I'm done with the SQL. For now I'm creating a JSON File containing all the countries with their corresponding leagues. I will do some testing with it to see if the ID logic was the way we thought.

I will check-in the JSON File and the Python code I used to Github as it might be useful in the future.

@akasolace Could you add me to the devs of HO in Hattrick so I can access the Consumer Key and Secret there, without having to decipher it? My Hattrick username is Cyrill_10 (4655017)

[cyrill10]

I think the Table should be relatively simple on the server. All we really need is the leagueID the Level and the Country.

So something like:

Table LeagueUnits
int ID
int Level
int LeagueID or (CountryID)

We could make a Table for the Country, but that is not necessary in my opinion. If that is fine with you I will create a File containing the SQL Insert scripts and give you the link to it

[mistermax80]

Hi all, I don't understand if the data are static and after initial population if they will need update, if it is true, we can think add new table in ho dB, without connect external data. Could you help me to understand better?

[cyrill10]

@mistermax80 This feature contains two problems we need to solve:

1. (main task of the feature) Is to calculate the new leage of a team. This information will change for every season. And needs to be calculated newly. The data flow will look something like this:

User1 from CountryA in HO wants to find his new league. HO calls the HOServer to see if the HOServer knows the current Information for CountryA. If it does not HO will calculate the new Leagues for the whole CountryA and send the result back to HOserver.

2. To do the calculation of the new Leagues we need to know every league and its level and country. Unfortunalty those information is not provided by CHPP. Therefore we have to create our own lookup table on the HOserver. Those data should not change regularly (only when new countries get added or Hattrick starts to grow significatly again).

For 1. we will create a REST call on the server to fetch and populate those data.

For 2. I will provide the SQL script to initialize the DB. For now we do not need an interface for that.

I hope this makes more sense now.

[mistermax80]

OK, perfect! For point 2, for now or for future we can think the inizialize db we can do by new page, where we can insert the parameters as numer countryies, number divisions for country ecc... and submit these the server will populare the table in correct way. For time line, when do we release it? For this season we are in delay.

[akasolace]

For point 2, for now or for future we can think the inizialize db we can do by new page, where we can insert the parameters as numer countryies, number divisions for country ecc... and submit these the server will populare the table in correct way.

@mistermax80, those table are extremely stable, they hardly change at all over years. Creating a dynamic way to create them would be ano overkill. I think a sql script as mentioned by @cyrill10 is the way to go in that case

For time line, when do we release it? For this season we are in delay.

Could be nice to have it in place for next season even if it is only a DEV version. I stored sone league information of last interseason. That could help us to debug and validate the code.

[mistermax80]

Ok perfect! So just the table and its data are ready, I will create a new API service from server for exposing the data.

[akasolace]

@cyrill10 if you want you can join the dev team on slack channel

[mistermax80]

I didn't understand one thing, because do I want look the new league? What is it the scope?

[akasolace]

@cyrill10 are you still considering looking into it ?

[cyrill10]

I still am. Unfortunately I was offline the last few weeks but I should find some time to finish it in the next few weeks.

[akasolace]

@cyrill10 ok nice to know !

[akasolace]

@mistermax80 @cyrill10 in case you are still up for it ..... development cycle of 3.0 has started

[tychobrailleur]

See lot of users using an old tool that predict league for next season based on ranking. Maybe we could have this somehow in HO and/or maybe simply in a static webpage

@tychobrailleur I keep here for reference a list of implemented endpoint:

* `.../getStatus/{leagueID}`  => returns status and data availability [0-100%]

* `.../Take-Next-Missing-Block?leagueID={}&AcceptJob={true/false}`   => returns the next batch to be treated, if AcceptJob is set to false, it just simulate it but no action made on server side

Tomorrow I will try to create an endpoint for pushdata ? leagueID = xx & BlockID=xx as per the json file you shared with me.

Could we make these endpoints consistent? I suggest:

• GET /league/{leagueID}/status for the data status
• POST /league/{leagueID}/block/{blockID} for push data by block
• GET /league/{leagueID}/next-block?accept-job={true/false} for next block.

Right now it's a mish-mash of camel case / kebab case, URL param vs. query string, and initial lower case vs. upper case

[akasolace]

Could we make these endpoints consistent? I suggest: GET /league/{leagueID}/status for the data status POST /league/{leagueID}/block/{blockID} for push data by block GET /league/{leagueID}/next-block?accept-job={true/false} for next block.

yes, tomorrow I will also create table for Ireland, hence you will have server response other than error code

[akasolace]

@tychobrailleur hello, I created on server the "final" endpoint .../league/:leagueID/team/:teamID/pd-status I could not test it because dataTable are empty at the moment, so it is likely to be broken at this stage. However, on the server-side, I think logic-wise it is almost complete now.

[tychobrailleur]

@tychobrailleur hello, I created on server the "final" endpoint .../league/:leagueID/team/:teamID/pd-status

What is the output:

(pmStatus, [oppTeamIDs])

? For example, what are the possible values for pmStatus? Same for /league/{leagueID}/status — what are the possible values returned?

[tychobrailleur]

@akasolace A couple of changes to the league/{leagueID}/next-block endpoint would be needed:

• a code indicating if the block has been locked, or if all blocks are currently all locked by other processes. Currently it can only be inferred from Msg, a code would be more appropriate.
• BlockContent when returned is a string — can it be changed to an array of integers? Otherwise the string needs to be parsed again.

Thanks!

[akasolace]

@tychobrailleur hello, I created on server the "final" endpoint .../league/:leagueID/team/:teamID/pd-status

What is the output:

(pmStatus, [oppTeamIDs])

? For example, what are the possible values for pmStatus? Same for /league/{leagueID}/status — what are the possible values returned?

res = {"status_code": pdStatus.value, "status_desc": pdStatus.name, "oppTeamIDs": oppTeamIDs}

Pour le status:

class PM(Enum): Undefined = -1 DD = 0 MD = 1 S = 2 MP = 3 DP = 4

DD = direct demotion MD = Demotion Match barrage S = no change MP = Promotion Match barrage DP = Direct Promotion

oppTeamIDs: arrays of int of TeamIds in case of DD , MD, MP, DP most of the time it should be a single value but it can be more in case of doublons

[akasolace]

A couple of changes to the league/{leagueID}/next-block endpoint would be needed: a code indicating if the block has been locked, or if all blocks are currently all locked by other processes. Currently it can only be inferred from Msg, a code would be more appropriate.

@tychobrailleur pour l'instant c'est implémenté comme ca: dis moi si tu veux des changements

201 => "No more data required for league {leagueID}" 202 => "Currently, league {leagueID} has no block available for treatment" 200 => "Job awarded for league {leagueID} / Block ID {blockID}", "BlockID": 203 => Job NOT awarded for league {leagueID} / Block ID {blockID}", "BlockID"

BlockContent when returned is a string — can it be changed to an array of integers? Otherwise the string needs to be parsed again.

ok, je vais faire ca par contre ca va me demander des changements dans la logique côté server, et des changments de scripts, je ferais ca en début de semaine ....

[tychobrailleur]

Ok, thanks for the details! I'll let you know if I need any more change.

[akasolace]

BlockContent when returned is a string — can it be changed to an array of integers?

I think one way or another we have to store the list as string and parse it. I can change my code to parse it on the server side, or we can leave the table as is and parse it on HO side ... If you confirm you prefer this to happen on the server side, I will do the required changes ...

[tychobrailleur]

I think one way or another we have to store the list as string and parse it.

I am not sure why you have to store as a string, but in any case, I have already done the parsing work on my side: https://github.com/tychobrailleur/HO/commit/3edfd24b09e2f6bc943b790ad920af4a3f360c55#diff-ac5c285fe782cea4e2d197cdc9c28c56R144-R145

So if you feel it makes no difference either way, we can keep it that way, and document.

[akasolace]

ok, I will give it a try just in case with another endpoint and we will test ... but I don't think that performance will be an issue here anyway

[akasolace]

@tychobrailleur are you already at a point where you can push data for a given league? I would need all data fora league to be available (e.g. Ireland) in order to test the logic for promotion/demotion ...

[akasolace]

BlockContent when returned is a string — can it be changed to an array of integers?

@tychobrailleur ok I made the change on the server, you should now get directly a list of integers when calling next-block end point

[tychobrailleur]

@tychobrailleur are you already at a point where you can push data for a given league? I would need all data fora league to be available (e.g. Ireland) in order to test the logic for promotion/demotion ...

Yes, definitely. I am posting data but getting back a 502 (if I remember correctly). I'll try again this evening after work.

Right now, I am working on the UI part for handling the data availability. The next big task is displaying the actual results about the new league.

[tychobrailleur]

BlockContent when returned is a string — can it be changed to an array of integers?

@tychobrailleur ok I made the change on the server, you should now get directly a list of integers when calling next-block end point

Awesome — will give it a bash tonight.

[tychobrailleur]

The core logic is now in place in https://github.com/tychobrailleur/HO/tree/promotion-mgr. Here is a todo of the remaining tasks:

• [ ] Handle all possibilities of status of data availability
• [ ] Check that number of teams uploaded matches num of teams in the block.
• [x] Show / hide info depending on status
• [x] Hide download message once message complete
• [x] Feedback on progress of download
• [x] Display correct message on frontend.
• [x] Find a way to not block when switching to “League Details” tab.

@akasolace We should probably discuss how you'd like results displayed. So far:

Data unavailable, button to download (needs to be designed, obviously):

Dialog to confirm intent:

Result (text would be displayed based on status):

[akasolace]

@tychobrailleur I tried by curioisty your branch but I go this:

HttpDataSubmitter: Error fetching data for league supported: Failed to connect to /127.0.0.1:3000

[tychobrailleur]

@tychobrailleur I tried by curioisty your branch

Ah cool, exciting!!

but I go this:

HttpDataSubmitter: Error fetching data for league supported: Failed to connect to /127.0.0.1:3000

Locally I run mitmproxy on port 3000 for debugging. Obviously this will be removed when released.

You should be able to run the app by commenting out that line: https://github.com/tychobrailleur/HO/blob/promotion-mgr/src/main/java/module/series/promotion/HttpDataSubmitter.java#L240

[akasolace]

yes it solved the issue

[akasolace]

I just tested it. it started ok but then it got stuck ...

[Info] LeaguePromotionHandler: Status of league: 5 : {"status_code": 1, "status_desc": "NOT_AVAILABLE", "nbBlocks": 14, "nbBlocksReady": 4, "nbBlocksInProgress": 10} [Info] HttpDataSubmitter: Lock block for league 5...

I had to kill the process ...

[akasolace]

and on the server side, here what I see

[akasolace]

so looks like block 1 was pushed long time ao when I did some test blocks 2, 3 and 14 pushed ok other blocks were waiting for data but did not receive them (log file)