search

kghackers / kgparser

Statistics for https://klavogonki.ru/. Also contains code for legacy competitions parser.
https://www.klavostat.com
4 stars 3 forks source link
apachepoi assertj aws chart-js freemarker github-actions jackson java js json junit5 klavogonki lombok mapstruct maven mysql spring-boot zip4j

readme

Chat on Telegram GitHub issues Github PR

Bugs Code Smells Coverage Vulnerabilities Quality Gate Status

Related resources

Statistics generation import process

Json config files are used to set and transfer configuration values for main import classes.

How to execute PlayerDataDownloader to download data from Klavogonki to JSON files

PlayerDataDownloader takes 2 arguments: <inputConfigFilePath> and <outputConfigFilePath>. I will add download start and end dates from <inputConfigFilePath> and save the update json to <outputConfigFilePath>.

For example, to load players with playerId in [30000; 30100] (total 101 players) in 10 threads, set the input json file like this:

{
  "jsonFilesRootDir" : "D:/kg/json",
  "threadsCount" : 10,
  "minPlayerId" : 30000,
  "maxPlayerId" : 30100,
  "statisticsPagesRootDir" : "C:/java/kgparser/kgstats/src/main/webapp/"
}

Execution with overriding the log4j2.xml configuration file:

java -Dlog4j.configurationFile=log4j2.xml -jar kgstatsSrv/target/kgstats-srv-1.0.jar DOWNLOAD_PLAYER_DATA c:/java/config-input.json c:/java/config-output.json

Saved output config file will look like this:

{
  "jsonFilesRootDir" : "D:/kg/json",
  "threadsCount" : 100,
  "minPlayerId" : 30000,
  "maxPlayerId" : 30100,
  "dataDownloadStartDate" : "2021-01-17T21:39:19.0911347+01:00",
  "dataDownloadEndDate" : "2021-01-17T21:39:30.0225525+01:00",
  "statisticsPagesRootDir" : "C:/java/kgparser/kgstats/src/main/webapp/",
  "totalPlayers" : 101,
  "dataDownloadStartDateString" : "2021-01-17 21-39-19"
}

How to execute StatisticsApplication to import json files save by PlayerDataDownloader to the database

Database must already exist before the execution.

Pass the output json config file saved by PlayerDataDownloader as an <inputConfigFilePath> for StatisticsApplication.

Spring profile "database" must be set to turn on the JPA, Spring repositories etc.

This example passes the alternative Spring application properties file to be able to override the database name from the default application.properties.

log4j2.xml configures the logging.

java -Dlog4j.configurationFile=log4j2.xml -cp kgstats-srv-1.0.jar -Dspring.profiles.active=database -Dspring.config.name=application.actions.properties -Dspring.config.location=kgstatsSrv/src/main/resources/ IMPORT_JSON_TO_DATABASE c:/java/config-output.json

How to execute StatisticsApplication to generate statistics files from the database

Database must already exist before the execution.

Database must be filled with data on the previous step.

Pass the output json config file saved by PlayerDataDownloader as an <inputConfigFilePath> for StatisticsApplication.

Pass the statistics generation config file as a <statisticsGeneratorConfigFilePath> for StatisticsApplication.

Spring profile "database" must be set to turn on the JPA, Spring repositories etc.

This example passes the alternative Spring application properties file to be able to override the database name from the default application.properties.

log4j2.xml configures the logging.

:exclamation: If you have encoding problems in logs and saved files on Windows, also set a -Dfile.encoding=UTF8 option.

java -Dlog4j.configurationFile=log4j2.xml -cp kgstats-srv-1.0.jar -Dspring.profiles.active=database -Dspring.config.name=application.actions.properties -Dspring.config.location=kgstatsSrv/src/main/resources/ GENERATE_STATISTICS_FROM_DATABASE c:/java/config-output.json c:/java/generator-config.json

Какие графики и таблицы можно сделать на текущих данных

MySQL

Create a read-only user

See https://kodejava.org/how-to-create-a-read-only-mysql-user/

User name: report, user password secret:

CREATE USER 'report'@'%' IDENTIFIED BY 'secret';

GRANT SELECT ON kgparser.* TO 'report'@'%';

FLUSH PRIVILEGES;

Dump the database

See https://phoenixnap.com/kb/how-to-backup-restore-a-mysql-database

mysqldump -u root -p kgparser > c:\java\kg\kgparser.sql

Restore the database from dump

See https://alvinalexander.com/blog/post/mysql/how-restore-mysql-database-from-backup-file/ To restore the dump, we first need to create the database.

mysqladmin -u root -p create kgparser_from_dump
mysql -u root -p kgparser_from_dump < c:\java\kg\kgparser.sql

Drop the database

See https://linuxize.com/post/how-to-delete-a-mysql-database/ 

mysqladmin -u root -p drop kgparser-from-dump

Maven

Install main modules without tests and without javadoc generation

From root directory, run

mvn install -DskipTests=true -Dmaven.javadoc.skip=true

Execute MapStruct generation in kgstatsSrv

From kgstatsSrv directory, run

mvn compile

:exclamation: Now after making changes in kgparserSrv, you have to mvn install this module before running MapStruct in kgstatsSrv.