Open smoogipoo opened 4 years ago
Out of curiosity: what's the use-case of computing pp for all modes at once? Their pp calculations are so far detached from each other that I can't imagine ever tweaking all of them at the same time.
Also, since the modes operate on different tables, I don't think there's much saved effort in computing the difficulties and indices for all modes at once as opposed to per-mode.
Just to reduce the complexity of implementation. Right now I'm passing in two environment variables (MODE
and MODE_LITERAL
) to do exactly what you're saying.
If complexity is a non-issue then that's fine, I'm not against having env variables. Although for cases where data is being imported like this, it's generally the case that sr/pp calculations should take place.
Agreed... maybe then it would be a good idea to automate which modes are called by checking which types of data are imported? I.e. only run difficulty calculations for modes in which beatmap data is imported; only run pp calculations for modes in which score data is imported, etc.
An isolated docker container to do absolutely everything here would be perfect. Something like:
export IMPORT_DATA=./my-import-data/
export THREADS=2
docker-compose up data-import
I sometimes need to run full reprocesses of all users to generate updated star rating / pp ratings based on new algorithms that the community comes up with.
Until now, I've maintained a fork of this repository which adds the necessary functionality to do this. The changes required can be seen here: https://github.com/ppy/osu-web/compare/master...smoogipoo:pp-tester
The most important requirements are:
A way to import SQL data dumps found on https://data.ppy.sh.
In
dependencies/docker/start.sh
, I've had to work around some shortcomings of this data.sample_users
intophpbb_users
.sample_users
does not contain all the data ofphpbb_users
. Of particular importance isuser_lastvisit
andosu_playmode
, which I've had to manually update the data of post-import. This could be fixed by including these columns into the data dumps (or including a cut-downphpbb_users
instead ofsample_users
).osu_countries
,osu_genres
, andosu_languages
. The default deploy also doesn't contain default values for these tables which causes (/used to cause?) random JS errors on profile + beatmapset pages.osu_user_performance_rank
, which causes (/used to cause?) JS errors when displaying the graph in the profile, and would cause the profile page to crash.user_id
as amediumint
, but should be updated toint
to sufficiently support the sample data. (https://github.com/ppy/osu-web/issues/6001)Integration of the 4 core processes.
diffcalc
container).ppcalc
container).esindexer
container).This is currently done through setting
osu_counts.docker_db_step
at each step of the way. All other processes (ppcalc/diffcalc/es-indexer) wait for this to reach different stages before continuing.I'm open to other synchronisation solutions, such as potentially only running docker instances in order when the previous completes. In my investigations, docker-compose actually removed the ability for this in an update.
There should be automatic index creation after the data import finishes.
es:index-documents
andes:create-search-blacklist
to do this.Adjust the default environment variables.
.env.example
has the default ES configuration commented out. I'm appending the ES config.QUERY_DETECTOR_ENABLED=0
to remove errors that would pop up on beatmap set/profile pages. It would be helpful if there was a production environment read to use that didn't require manual adjustments.Have some way to allow non-logged-in access to beatmap listing search.
app/Libraries/Search/BeatmapsetSearchRequestParams.php
, but this doesn't work anymore. (https://github.com/ppy/osu-web/issues/6121)Other details
ppy/osu-difficulty-calculator
,ppy/osu-performance
, andppy/osu-elastic-indexer
and allow sr/pp calculations to take place.osu-server
as I'm doing (useosu-difficulty-calculator
instead).osu-difficulty-calculator
->osu-performance
->osu-elastic-indexer
.osu-difficulty-calculator
references a local beatmap directory. It could reference a URL instead via\"download_path\": \"https://localhost/osu/{0}\"
, but it would be nice if there was support for that at an nginx/laravel level (perhaps linking to some storage location in the .env or something).osu-difficulty-calculator
andosu-performance
should not require the mode to be passed in (https://github.com/ppy/osu-performance/issues/113 for osu-performance, and you can omit the option completely forosu-difficulty-calculator
).Some of the fat in my fork can be completely excluded, such as: