search

azerothcore / azerothcore-wotlk

Complete Open Source and Modular solution for MMO
http://www.azerothcore.org
GNU Affero General Public License v3.0
6.51k stars 2.61k forks source link

[EPIC] AzerothCore progression system #5248

Closed FrancescoBorzi closed 1 year ago

FrancescoBorzi commented 3 years ago

We need a set of SQL files allowing us to switch from the first season of Classic, to the last one of WOTLK.

Every season will be then tested and played on ChromieCraft, so we will improve every single content. The progression phases are listed here: https://www.chromiecraft.com/progression/

This task is NOT about fixing stuff, but about enabling/disabling only the right contents per every phase.

Examples

An example of what we need has already been done by @Si1ker for the WOTLK phases, you can find it here: https://github.com/Si1ker/WoltkProgression

Some other minor tweaks were done by @JamesGent and are available here: https://github.com/azerothcore/classic-mode

The features

For every phase, we should limit the access only to the appropriate Instances and vendors (including PvP vendors) and we should make inaccessible to players all items that belong to a later phase.

This is mostly a job about researching and writing the correct SQL queries. No magic or complex C++ scripts required here.

The code

The code will be part of a module and should be a collection of SQL files that are easy to revert. See the example of up.sql and down.sql in the classic-mode repo.

The zero SQL assumes you are applying it on a clean AzerothCore and it has to disable everything from the later phases.

The first SQL is meant to be applied after zero, the second after the first SQL, and so on...

Phases

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/97764266-epic-azerothcore-progression-system?utm_campaign=plugin&utm_content=tracker%2F40032087&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F40032087&utm_medium=issues&utm_source=github).
yuanf225 commented 3 years ago

There is also a corresponding arena

Maurowin commented 3 years ago

BIG UP for this feature !

On the other hand, you should not forget that according to the extensions, there is the heroic mode which is unlocked according to the reputation or the quests.

jordigil commented 3 years ago

Pinned please.

FrancescoBorzi commented 3 years ago

Reminder: Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

noisiver commented 3 years ago

In case anyone is interested, I have actually been working towards just this. I am still learning SQL, but started doing work on WoW 1.1 using SQL-files. My work is located at https://github.com/tkn963/progression. Some content uses modules, like I scripted Lord Kazzak in Blasted Lands using a module.

I want to do this using the core itself, much like vMaNGOS does where you specify the patch in the config file and the core loads the content for that patch. I am however a little stuck at trying to figure out what tables need a minPatch/maxPatch. Basically tables that holds data that gets removed in later patches need min/max while other tables only need a column called patch.

I don’t know either if a feature like this would be considered custom or useful to have in the core.

FrancescoBorzi commented 3 years ago

@tkn963 it's great that you're working on it. About:

I want to do this using the core itself, much like vMaNGOS does where you specify the patch in the config file and the core loads the content for that patch. I am however a little stuck at trying to figure out what tables need a minPatch/maxPatch. Basically tables that holds data that gets removed in later patches need min/max while other tables only need a column called patch.

I don’t know either if a feature like this would be considered custom or useful to have in the core.

indeed that would be a nice feature to have, but it's more work than just having a bunch of SQL that can bring you from one version to another (which is what this EPIC is for).

Maurowin commented 3 years ago

In case anyone is interested, I have actually been working towards just this. I am still learning SQL, but started doing work on WoW 1.1 using SQL-files. My work is located at https://github.com/tkn963/progression. Some content uses modules, like I scripted Lord Kazzak in Blasted Lands using a module.

I want to do this using the core itself, much like vMaNGOS does where you specify the patch in the config file and the core loads the content for that patch. I am however a little stuck at trying to figure out what tables need a minPatch/maxPatch. Basically tables that holds data that gets removed in later patches need min/max while other tables only need a column called patch.

I don’t know either if a feature like this would be considered custom or useful to have in the core.

Maybe we should create a progressive module? Just to keep the CORE as breast as possible.

noisiver commented 3 years ago

In case anyone is interested, I have actually been working towards just this. I am still learning SQL, but started doing work on WoW 1.1 using SQL-files. My work is located at https://github.com/tkn963/progression. Some content uses modules, like I scripted Lord Kazzak in Blasted Lands using a module. I want to do this using the core itself, much like vMaNGOS does where you specify the patch in the config file and the core loads the content for that patch. I am however a little stuck at trying to figure out what tables need a minPatch/maxPatch. Basically tables that holds data that gets removed in later patches need min/max while other tables only need a column called patch. I don’t know either if a feature like this would be considered custom or useful to have in the core.

Maybe we should create a progressive module? Just to keep the CORE as breast as possible.

I looked into that, but I really couldn't find a reliable way of handling the database from a module. If there is one, I would love to look into it. Obviously the idea isn't to delete entries from the database but instead just load what is needed. Is there like a despawn creature/object feature to use in modules? If there is I would really dedicate a lot of time to developing such a module.

noisiver commented 3 years ago

I should also note that creatures change over time. Take city guards, they're 55 in vanilla but 75 in wrath. That's also something that would need to be changed through a module. I can't find anything like that though.

yuanf225 commented 3 years ago

https://github.com/Si1ker/WoltkProgression Please refer to

noisiver commented 3 years ago

https://github.com/Si1ker/WoltkProgression Please refer to

That, unfortunately, doesn't help. I have been developing my module, and got everything working except handling area triggers, disables and removing spells from npc trainers.

noisiver commented 3 years ago

Where do you want to store data for this progression system? Since I have been working on this for a while, of course there is always an unlimited amount of work to do, maybe it could help someone?

FrancescoBorzi commented 3 years ago

@tkn963 I just created a repo for it, feel free to send PRs there: https://github.com/azerothcore/progression-system

kaytotes commented 3 years ago

@FrancescoBorzi or anyone else who may be interested. Until a short while ago I was using Azeroth-Core but the needs of my current project have lead me elsewhere. While incomplete I didn't want what I'd been doing to completely die so I've opened the repository and adding details here just in case anyone else wants to use it as a base and finish it.

I'd been working on essentially bringing over the vMangos progression system to this project and had a majority of things working and tested, the last things on my list were related to quests.

The main work is here: https://github.com/Project-Epoch/mod-progressive-loading This branch is the most up to date with some half finished work on the quests: https://github.com/Project-Epoch/mod-progressive-loading/commits/feature/quests

I kept track of rough notes and what's been done in here: https://github.com/Project-Epoch/mod-progressive-loading/issues/1

The module relies on new hooks implemented in: https://github.com/kaytotes/azerothcore-wotlk/commits/feature/loading-hooks

A diff is available here: https://github.com/kaytotes/azerothcore-wotlk/compare/master...kaytotes:feature/loading-hooks

If you want this transferred over to the Azeroth-Core organisation I'd be happy too but otherwise I'll leave that repo up for about 6 months as of this comment before removing.

One piece that is unfinished but is important is this adding the client cache implementation based on my https://github.com/azerothcore/azerothcore-wotlk/pull/7136 It should be obvious based on the example I gave that this was originally to coincide with this module.

To make it easy to test and for player quality of life I recommend implementing that as it means users won't have to manually delete the Cache directory in the client when the server progresses to new phases.

This does not include any of the data as I hadn't really hit that point and will take many people to gather.

FrancescoBorzi commented 3 years ago

@kaytotes thanks for sharing! can you please send via PRs all the necessary hooks to make AC fully support your module?

Maurowin commented 2 years ago

Concerning the VANILLA part there is Vmangos which is still in development and which has a progressive system. https://github.com/vmangos/core

noisiver commented 2 years ago

Concerning the VANILLA part there is Vmangos which is still in development and which has a progressive system. https://github.com/vmangos/core

Yeah, I was thinking about implementing something similar but you would have to either modify the core or add handlers to be able to override the sql queries before they are run.

EDIT: Override them within a module I mean. I don't know how easy it would be to give that option to a module, but if you could it would be great because you wouldn't want a huge multiplication of entries in, for example, the creature templates but rather just use the same entry for creatures that simply change but set min_patch and max_patch to different values and only load those that are within the value set in the config. Just like vMaNGOS does, except fully optional and modular.

Maurowin commented 2 years ago

I wonder if we could use the same client that will be available for Wotlk classic. That would be huge!

noisiver commented 2 years ago

I can't imagine that working out, but I don't know much about the client. My guess is it would require such heavy modification that it would probably just be easier to get the 1.12.1 client to be able to connect to the world. No matter how you do it, the basic option would give users the ability to add, delete and edit content based on the patch (from 1.x to 3.3).

noisiver commented 2 years ago

I wonder if we could use the same client that will be available for Wotlk classic. That would be huge!

Wait, I just read your comment again and realized that's probably what you meant. My bad, sorry!

Maurowin commented 2 years ago

No, I meant, when WOTLK classic comes out, there will be a client that will be up to date with the new development tech. I hope that Azerothcore will be able to accept the client that will be released specifically for WOTLK classic.

kaytotes commented 2 years ago

@Maurowin I highly doubt that will happen, a lot of the opcodes and whatnot drastically changed, auth, you have to patch the exe to change realmlist in later editions etc. 3.3.5 is still perfectly suited for modern environments.

I am very confident that the links I provided in an earlier comment would get you very close to a vmangos like progression system, the hard part is then data. I still do not currently have time to finish it up but if someone wants to complete the Quests work (which shouldn't be too hard once you read the code and spot the pattern) then that would be great.

I apologise that I do not have to capacity to complete the initial work.

noisiver commented 2 years ago

If a way of altering the sql queries prior to actually running them was available I would pour a ton of work into a module that only loads specific content. My current work deletes creatures that are not within the patch range but that is bad for obvious reasons and I abandoned that idea.

Of course, you would need to be able to change almost all queries for the world database.

I am not skilled enough in C++ to work out some changes to the core to support this, unfortunately.

Maurowin commented 2 years ago

Maybe someone here could make hooks and give the modules the ability to modify the requests to do what you want.

kaytotes commented 2 years ago

@tkn963 That is exactly what my code does. I've added all but a handful of hooks

noisiver commented 2 years ago

@tkn963 That is exactly what my code does. I've added all but a handful of hooks

I know, but I don't have the ability to PR it over to the core. Well, I probably could but the standard in AC and everything would take me time to match.

My only suggestion on top of your great work is to have min_patch and max_patch instead of just patch. I might take a look at it, probably have to fork it so I'm not on a time limit, see if I can understand exactly how it works and if I can get it working on my end.

kaytotes commented 2 years ago

For the tables where it actually matters patch_min and patch_max already exist eg creatures, gobjects.

noisiver commented 2 years ago

As I'm working on this, my biggest issue so far is determining what tables need what columns.

I'm working through creatures first by adding min_patch and max_patch to creature_template and going through the errors and changing tables that print errors as I go.

There are certain tables I know for sure, like creature_template and creature since creatures change a lot over the course of the game and some even get removed etc. I'm stumped by other tables like creature_onkill_reputation.

All tables I'm looking at that could require a change: areatrigger (patch) areatrigger_involvedrelation (???) areatrigger_tavern (patch) battleground_template (patch) creature (min/max) creature_addon (???) creature_equip_template (???) creature_formations (???) creature_loot_template (min/max) creature_onkill_reputation (???) creature_questender (min/max?) creature_questitem (min/max?) creature_queststarter (min/max?) creature_summon_groups (???) creature_template (min/max) creature_template_resistance (???) creature_template_spell (???) creature_text (???) disables (min/max) disenchant_loot_template (min/max) dungeon_access_template (???) gameobject (min/max) gameobject_addon (???) gameobject_loot_template (min/max) gameobject_questender (min/max?) gameobject_questitem (min/max?) gameobject_queststarter (min/max?) gameobject_template (min/max) gameobject_template_addon (???) game_event (patch) game_graveyard (patch) instance_template (patch?) item_loot_template (min/max) item_template (min/max?) npc_trainer (min/max) npc_vendor (min/max) pool_creature (???) pool_gameobject (???) pool_pool (???) pool_template (???) quest_template (min/max) quest_template_addon (???) reference_loot_template (min/max) spell_loot_template (min/max) transports (patch) waypoints (???) waypoint_data (???)

The characters within parenthesis: ??? = I'm unsure if they need columns and if so which ones min = min_patch max = max_patch patch = added in specified patch and not removed later on

I would love some input on what I should and shouldn't work on as I'm going through it.

For example, the creature_onkill_reputation is only set to patch within vmangos but that's not multiple expansions and I can't remember if reputation gains changed between patches or not.

I might have forgotten some tables above.

noisiver commented 2 years ago

I have a serious question. As I'm working through all of this, been going pretty strong and I've added a total of 33 hooks so far but still a few remaining.

Some queries, like for instance the SmartAI, are called using prepared statements. That specific statement appears to only be used once, is it a horrible idea to convert it to a regular QueryResult or is there a way to alter the statement before it's actually called?

kaytotes commented 2 years ago

@tkn963 If you've implemented the custom condition hook it should be possible to create a condition on the smart script itself that checks the current patch meaning the existing behaviour isn't really messed with and makes it easier to test that you didn't break anything.

When I first started on this the goal was to hit the high value stuff first then drop down to the lesser things if proof can be found that it did change.

Eg I can't think of an example where an existing NPC had it's waypoints change and if that does happen you could create a new creature_template entry with a new waypoint ID which means that waypoints are kind of already covered.

I don't have time to actively work on this but feel free to ping me on discord and I can at the very least be of some help with the "why".

Ktotes#2737

noisiver commented 2 years ago

@kaytotes Amazing, thank you. I'm still fairly new when it comes to C++ and SQL so I'm learning as I go, but I feel like it could be done.

Currently I have things I am either unsure of or know will change over time to have min/max. The issue with this is that the scripts it's complaining about are rows that are not loaded from the database since they are not within the scope of what should be loaded. Would a condition actually trigger before it tries to load the smart_scripts table?

I also just went with, for lack of a better word, whatever when it comes to columns since the module I'm making is more or less just there to check for issues when loading data from the database while I'm adding hooks. When the hooks are finalized, the module can be developed in a better way. The min_patch or patch columns, depending on what I added, are set to max to prevent anything from loading so I can find any issues that pop up.

I've seen a few other people using phasemask to remove creatures, but for some reason I've always thought that if it isn't used, why load it?

I really want this feature added but, to be honest, I'm beginning to doubt that I'm skilled enough to pull it of.

kaytotes commented 2 years ago

The reason for using a phase mask for creatures / gobjects is basically specifically to avoid a shed load of errors on core start up. So many things are tied to the guids of those tables that deleting isn't wise, possible just harder.

By setting to phasemask 0 you eliminate that problem and no normal player (even GM's i think) can see them.

From my experience the underlying core is well optimised enough that the overhead of having these extra but invisible creatures is essentially negligible.

noisiver commented 2 years ago

I know of a workaround for that without having to add hooks to the core. You can alter creature_template data without changing the database itself but rather edit them just as the server finishes it's startup routine. You can also add, delete and modify data from the creature table and so on. I used to build my module on that before but was stopped by not being able to remove data from npc_trainer. You can add, but not remove. There are really very few hooks that needs to be added this way and the database doesn't even have to be altered. You would probably add your own tables though that contain that data that the module reads from.

I'm going to scrap the current project for now and build a new module based on my old plan and when I've reached a decent point I'll hit you up on discord and you can tell me what you think.

To give you an idea of what I'm talking about, have a look at these: https://github.com/azerothcore/azerothcore-wotlk/blob/d5509ceb375bf1b685285218b3363e578169a95a/src/server/game/Globals/ObjectMgr.h#L1167 https://github.com/azerothcore/azerothcore-wotlk/blob/d5509ceb375bf1b685285218b3363e578169a95a/src/server/game/Globals/ObjectMgr.h#L1315 https://github.com/azerothcore/azerothcore-wotlk/blob/d5509ceb375bf1b685285218b3363e578169a95a/src/server/game/Globals/ObjectMgr.h#L1064

noisiver commented 2 years ago

I can't, for the life of me, remember how I changed the values of the creature template. I know I have done it before by updating the data stored in memory after the server started. I need to find a way of getting the Creature using an entry from creature_template.

noisiver commented 2 years ago

@kaytotes Just figured I'd let you know that I'm having some issues no matter what choice I make. The new approach I'm using will work fairly well, except a lack of ability to modify disables, areatriggers and removing spells from trainers and possibly others, but the main problem is knowing what is actually needed. Personally I believe it would be best to start running through data to determine what needs to change through each patch. This would help a lot since you then know what tables you need to work with.

Do you know of a decent way of doing this? I used to compare the data between vMaNGOS and AzerothCore to check the difference as it's probably best to start with patch 1.1 to 1.12 before moving to TBC, but comparing those databases doesn't feel like the best way of doing it.

I'm guessing my plan was to find everything that was added in 2.x and above and set them to the patch above 1.12, then go through the 1.12 data and progress down in patches before doing the same for TBC. I just don't know if there's a way of knowing what content was added when, since there's no patch log specific enough. 😋

I was going to send you a PM on Discord about this but can't find your name in the list.

kaytotes commented 2 years ago

Getting all the data is generally the harder part. I've joined the Azeroth-Core discord so should be able to find me in there. I have some queries I used to grab some of this data.

In my opinion it's better to find simple cases that fit a specific use case.

A good example that covered multiple parts is the Wolves quest in Northshire Abbey.

Prior to wrath they weren't diseased, the quest text was different, drop item was different, model swapped, name swapped, auras added etc.

Just that one quest can cover a fair few tables.

Getting all the data is months if not years of effort to get accurate in my opinion so better to grab small chunks.

If you have a new idea for how to handle this then feel free to go down a different path.

Some of what I did around game events is basically muted by https://github.com/azerothcore/azerothcore-wotlk/pull/9099

If you added progressive disables too then maybe that's a good path to take idk.

noisiver commented 2 years ago

I decided to use the guild vault and meeting stones that were added in patches later than release. I also used creature levels for pre-TBC areas and changed their levels. It works fine.

I didn't mean getting all the data, but I do currently have enough to work with.

It's not about a different path, it's about alternate route that might be viable. It won't need hooks for the SQL-queries and no changes to the original tables.

The disables recently got support for game events so handling them separately is now pointless.

I would love your opinion, I've gone quite far but my knowledge of C++ is only so good. There are certain features that need to be added to the core, like handling disables, removing spells from trainers, removing (or disabling) transports, manage entries in the loot tables in the same way I'm handling the other stuff. It's all done in memory, so the server load is exactly the same, but at Startup() the module kicks in and changes what it's told to.

https://github.com/tkn963/mod-progression

I should mention that columns need to be added where required, I just used what I need for testing purposes.

By the way, is your discord name different? Because I even tried @kay and nothing showed up.

kaytotes commented 2 years ago

Shortened to ktotes on discord

Nyeriah commented 2 years ago

@FrancescoBorzi should those be moved to the progression repo?

FrancescoBorzi commented 2 years ago

@FrancescoBorzi should those be moved to the progression repo?

yes

Kitzunu commented 1 year ago

Closing after speaking with @Gultask

we agree that the progression module is far enough along that this issue won't be used, and tracking should be done on the module repo.

heyitsbench commented 1 year ago

Long time coming.