Change Epoch Timing

[iamentry]

Via discussions with @FO-nTTaX

https://discord.com/channels/93055209017729024/1077895919200112722/1197519869231505430

We should create a list of all modules and templates that use the current Epoch Time to allow for a swift and efficient change to a new Epoch Time (FO said 0000-00-00 is probably what it should be changed to).

[hjpalpha]

for git modules super easy, for non git modules a big pain in the a...

[FO-nTTaX]

Indeed, that's why we need a list. git is a somewhat simple find and replace. I assume the majority of modules relevant to this discussion are specifically excluding 1970-01-01, so we would just need to find them and replace these occurrences with the new default. A bot could possibly be used.

[iamentry]

I'm assuming that generally most of these modules are going to be squad and transfer-related? I'm going to start a spreadsheet that shows the location of the module and other stuff. Templates might be harder to track down.

[hjpalpha]

I'm assuming that generally most of these modules are going to be squad and transfer-related? I'm going to start a spreadsheet that shows the location of the module and other stuff. Templates might be harder to track down.

my guess there are way more match/placement related ones

[hjpalpha]

if you try to do such a list i suggest to use https://tools.liquipedia.space/?view=search for modules module: insource:1970 for the search for templates template: insource:1970 for the search

for modules it gives me several hundreds btw you can ignore all sandboxes and devs though i would say

[iamentry]

That is the most helpful thing I've ever seen. many thanks

[iamentry]

This is basically already a list of everything that I can change. I might be a bit limited on the module side by admin+ locked modules.

[iamentry]

for modules it gives me several hundreds btw you can ignore all sandboxes and devs though i would say

Should I also ignore archived modules?

[FO-nTTaX]

Ignoring archived modules is probably fine

[iamentry]

I have completed the list

https://docs.google.com/spreadsheets/d/1Hzup-iycCPfHGAjomjQGymNi8txQb3xYuc5p-UCBb88/edit?usp=sharing

All that is missing are the modules from Commons. I found this one and I know plenty others on Commons have to be changed, they just don't appear using module: insource:1970

[hjpalpha]

I have completed the list

https://docs.google.com/spreadsheets/d/1Hzup-iycCPfHGAjomjQGymNi8txQb3xYuc5p-UCBb88/edit?usp=sharing

All that is missing are the modules from Commons. I found this one and I know plenty others on Commons have to be changed, they just don't appear using module: insource:1970

shows plenty of commons for me:

[iamentry]

Awesome, mine must have not loaded initially. I will add them.

[iamentry]

I have checked all modules and templates for their edit protection levels and I can edit every non-git-maintained module. Very few of the modules that need to be changed are on the git, there tend to be 3 ≤ (4 for Sc2) for most main wikis and even less for others. I'm not aware of the server's ability to handle a bot run of this magnitude (397 modules & 52 templates) and I honestly think the purge runs will be more important for the bots to work on.

Templates are easier as there are only 3 that need to be changed by an admin+.

p.s. There are lots of git-maintained modules that need protection. I can link in admin-requests if needed.

[hjpalpha]

I have checked all modules and templates for their edit protection levels and I can edit every non-git-maintained module. Very few of the modules that need to be changed are on the git, there tend to be 3 ≤ (4 for Sc2) for most main wikis and even less for others. I'm not aware of the server's ability to handle a bot run of this magnitude (397 modules & 52 templates) and I honestly think the purge runs will be more important for the bots to work on.

Templates are easier as there are only 3 that need to be changed by an admin+.

p.s. There are lots of git-maintained modules that need protection. I can link in admin-requests if needed.

1. the number of moules changed isn't the issue the number of wikis where modules are being changed in relatively short time will be the issue, like a module change triggers tons of jobs but they are done one by one per wiki, so we only get issues if many wikis have jobs at the same time
2. most of the purges can probably be done via mass purge tool (faster than with bots)
3. for the protection stuff make a list and either open an issue for it here on the git or drop it in admin requests channel, we then can go through it in the next days

[hjpalpha]

additional note did you also check for epoch 0 stuff? for the search would use epochZero

[iamentry]

additional note did you also check for epoch 0 stuff?

I haven't. Thankfully there only seem to be a few new ones. I will get them added. Are there any other terms I should search with to make sure we have everything on the list?

[FO-nTTaX]

For the record: I'm looking to move to 0000-01-01 00:00:00 as the new default time

[iamentry]

Sounds like the best decision. The list is complete (I hope) with 415 modules and 52 templates. How do you think we should go about this? I can manually edit alongside a bot allowing us to clear wikis faster and with less strain on the server. The spreadsheet also allows us to know which wikis are done so either of you can use the mass purge tool after a wiki is finished. There should also probably be some sort of announcement on the discord explaining that this is going to happen and some things could be broken momentarily. And if something is broken they know who to ask.

[iamentry]

I could also prep a pr that's ready to go with all the changes on git-controlled modules.

[hjpalpha]

i don't think we can switch it on all wikis at the same time that would kill the server (all wikis running jobs would even kill the new one)

imo first adjust everything on commons (PR + bot run for the rest) after that select 15 wikis to start with

• pr to adjust the wiki specific git modules with 1970-01-01
• bot run to do the same on each of the 15 wikis
• after those are done purge jobs for all pages with that date in any lpdb data

once enough wikis have no jobs anymore start on the next 15 wikis and so on

imo go for biggest wiki to smallest wiki for the order of doing things

[hjpalpha]

also before we start with it we should:

• give a warning to (external) api consumers
• make announcements about some changes happening and that those can cause some temporary bugs on a few places

[hjpalpha]

wrote https://liquipedia.net/commons/Module:LpdbDateSwitchPurgeList to get a list of pages to purge after the module (and lpdb default date setting) adjustments usage is per wiki should probably keep the jobs lower compared to purging entire wikis completely (e.g. for sc2 it would be ~4.2k pages that need a purge compared to 45,462 content pages the wiki currently has)

[iamentry]

Diese Woche wird schwierig für mich, ich habe 3 Fußballspiele und Schule. Also, do we know if FO has gotten confirmation that we won't destroy the app with this change? We should also schedule when we want to begin the execution. These first 10 or so wikis will be super slow and then smooth sailing from there. I can prep wiki-specific prs like you suggested as well.

[hjpalpha]

Diese Woche wird schwierig für mich, ich habe 3 Fußballspiele und Schule. Also, do we know if FO has gotten confirmation that we won't destroy the app with this change? We should also schedule when we want to begin the execution. These first 10 or so wikis will be super slow and then smooth sailing from there. I can prep wiki-specific prs like you suggested as well.

hahaha ich feier das der erste satz in deutsch ist XD

for the batches going by the order on liquipedia.net landing page

first batch (15+ commons):

• commons
• dota
• val
• cs
• rl
• ml
• lol
• apex
• r6
• ow
• sc2
• pubgm
• aoe
• smash
• aov
• pubg

second batch:

• warcraft
• brawlstars
• bw
• wr
• fifa
• fighters
• heroes
• hearthstone
• artifact — main wikis end here
• f1 — sports wiki
• trackmania — alpha wikis start here
• cod
• freefire
• fortnite
• pokemon

third batch:

• tft
• halo
• clashofclans
• cr
• wow
• arenafps
• teamfortress
• tetris
• paladins
• sideswipe — alpha wikis end here
• stormgate
• smite
• worldoftanks
• osu
• crossfire

final batch (16):

• illuvium
• brawlhalla
• simracing
• zula
• splatoon
• omegastrikers
• naraka
• splitgate
• criticalops
• runeterra
• bataillon
• autochess
• magic
• underworlds
• squadrons
• battlerite

and yes it needs to be well coordinated devs need to adjust the lpdb default, preferably right before we merge the first PR and start the bot runs for the top 15 wikis and bot runs should be done by fast bots, i.e. fo or rath etc

[hjpalpha]

potential PS Skript to run pywikibot across several wikis to generate the purge list, purge the pages and delete the purge list again:

$wikis = @(
    'commons',
    'dota2',
    'valorant',
    'counterstrike',
    'rocketleague',
    'mobilelegends',
    'leagueoflegends',
    'apexlegends',
    'rainbowsix',
    'overwatch',
    'starcraft2',
    'pubgmobile',
    'ageofempires',
    'smash',
    'arenaofvalor',
    'pubg'
)

ForEach($wiki in $wikis){
    python pwb.py login -lang:$wiki

    python pwb.py pagefromfile -lang:$wiki -file:"PathToFile\pages.txt" -minor -title:"Liquipedia:PurgeListForDateSwitch" -summary:"Generate a list of pages to be purged for the date switch"

    python pwb.py touch -lang:$wiki -links:"Liquipedia:PurgeListForDateSwitch"

    python pwb.py delete -lang:$wiki -page:"Liquipedia:PurgeListForDateSwitch" -always -summary:"Delete purge list after purge"
}

with pages.txt being:

{{-start-}}
{{#invoke:LpdbDateSwitchPurgeList|run}}
{{-stop-}}

[FO-nTTaX]

Also, do we know if FO has gotten confirmation that we won't destroy the app with this change?

The app will be ready next week, there's a high chance we can commit to a Monday or Tuesday go

[FO-nTTaX]

#!/bin/bash

wikis=("commons dota2 valorant counterstrike rocketleague mobilelegends leagueoflegends apexlegends rainbowsix overwatch starcraft2 pubgmobile ageofempires smash arenaofvalor pubg")

for wiki in $wikis; do
        python pwb.py login -lang:$wiki
        python pwb.py pagefromfile -lang:$wiki -file:"/PathToFile/pages.txt" -minor -title:"Liquipedia:PurgeListForDateSwitch" -summary:"Generate a list of pages to be purged for the date switch"
        python pwb.py touch -lang:$wiki -links:"Liquipedia:PurgeListForDateSwitch"
        python pwb.py delete -lang:$wiki -page:"Liquipedia:PurgeListForDateSwitch" -always -summary:"Delete purge list after purge"
done

Bash version of hjpas script because I don't intend to install PowerShell on my Linux box :P

[FO-nTTaX]

Actually I realized this is stupid, I can just write an SQL script and change these directly on the database

[iamentry]

The app will be ready next week, there's a high chance we can commit to a Monday or Tuesday go

Montag ist für mich am besten. If wanted, I could write an announcement to be posted to the discord before we start.

[iamentry]

Is EpochZero being changed to anything in particular. Or are we changing things that say EpochZero = 0 to EpochZero = 0000-01-01

p.s. I'm working on the first batch

[FO-nTTaX]

we probably need to figure out where and how epochZero is used to make n informed decision on what to do with it

[hjpalpha]

Is EpochZero being changed to anything in particular. Or are we changing things that say EpochZero = 0 to EpochZero = 0000-01-01

p.s. I'm working on the first batch

in most places i think we can just use DateExt.minTimestamp instead

[FO-nTTaX]

I intend to start deploying for this today (Tuesday) if we think we are ready? @Verus-Vales @Rathoz @hjpalpha

[hjpalpha]

I intend to start deploying for this today (Tuesday) if we think we are ready? @Verus-Vales @Rathoz @hjpalpha

still open conversation in the PR apart from that we also need to adjust stuff outside the git when we start deploying imo first the wikis that get adjusted in the first PR and after those are done check how the server handles things

[hjpalpha]

i can adjust the non git modules after #3900 is merged

after the adjusts on the non git modules and #3892 and #3897 only templates should be left those we would need to adjust after the switch (imo manually since only a few per wiki and it allows double checking if the change is wanted)

[hjpalpha]

atm i do not have any further todos at work (might change tomorrow depending on what new stuff gets in) so i can probably hammer through a ton tomorrow if #3900 is merged by then

[hjpalpha]

progress on non git-modules:

• [x] lpcommons
• [x] ageofempires
• [x] apexlegends
• [x] arenafps
• [x] arenaofvalor
• [x] artifact
• [x] autochess
• [x] battalion
• [x] battlerite
• [x] brawlhalla
• [x] brawlstars
• [x] callofduty
• [x] clashofclans
• [x] clashroyale
• [x] commons
• [x] counterstrike
• [x] criticalops
• [x] crossfire
• [x] dota2
• [x] fifa
• [x] fighters
• [x] fortnite
• [x] formula1
• [x] freefire
• [x] halo
• [x] hearthstone
• [x] heroes
• [x] leagueoflegends
• [x] magic
• [x] mobilelegends
• [x] naraka
• [x] omegastrikers
• [x] osu
• [x] overwatch
• [x] paladins
• [x] pokemon
• [x] pubg
• [x] pubgmobile
• [x] rainbowsix
• [x] rocketleague
• [x] runeterra
• [x] sideswipe
• [x] simracing
• [x] smash
• [x] smite
• [x] splatoon
• [x] splitgate
• [x] squadrons
• [x] starcraft
• [x] starcraft2
• [x] stormgate
• [x] teamfortress
• [x] tetris
• [x] tft
• [x] trackmania
• [x] underlords
• [x] valorant
• [x] warcraft
• [x] wildrift
• [x] worldoftanks
• [x] worldofwarcraft
• [x] zula

[FO-nTTaX]

* [ ]  arenafps not listed at https://tools.liquipedia.space/?view=search

It should be?

[hjpalpha]

* [ ]  arenafps not listed at https://tools.liquipedia.space/?view=search

It should be?

yeah missed the space^^

[hjpalpha]

i think i got all modules now ignored any dev, sandbox, archive module also ignored any module that is/should be on the git since those should be covered via the PRs that got merged

atm https://tools.liquipedia.space/?view=search times out though so will have to check tomorrow

can do the same for templates tomorrow too via https://liquipedia.net/commons/Template:DefaultYear https://liquipedia.net/commons/Template:DefaultDate https://liquipedia.net/commons/Template:DefaultDateTime

should be rather easy as there are not as many templates to adjust

after that the switch will be super easy^^

[iamentry]

Fantastisch hjpa. Is there anything else within my capacity to change here or elsewhere (looking for more things to do or learn).

[hjpalpha]

if you want you can already look into the templates like this: https://liquipedia.net/commons/index.php?title=Template%3AFileInfo&type=revision&diff=548462&oldid=548102

[iamentry]

All templates have been changed except for this one Template:TeamCardImage that I can't edit.

[hjpalpha]

All templates have been changed except for this one Template:TeamCardImage that I can't edit.

found a handful more, those are done now too

modules double checked too

imo we are ready