Read original data content instead of hardcoding it

[simonsan]

I see in the source files many hardcoded values of the original game mechanics instead of reading it from original data. To circumvent any legal issues you should try to use the .dat file as a data source or export it to a format usable by your engine (e.g. TOML, YAML, ... ).

You could for example use genieutils to extract the data and store it within a nyan container format - just as an idea. ;-)

Regarding e.g.: https://github.com/puzzlepaint/freeage/blob/master/src/FreeAge/common/type_stats_data.cpp

Cheers

[MaanooAk]

I start by stating: I'm have no experience or knowledge on the subject.

You can't reverse engineer our games to access the assets or otherwise do things that the games don't normally permit in order to create your Items.

Do names and "values" of the units count as assets? In any case, the values are not reverse engineered, they are provided by the game (in the tooltips) and random sites on the internet (which actually make money by having ads next to the "values").

These are some comments for discussion, I may be wrong.

@simonsan Is there a specific section of the document you are referring to?

[MaanooAk]

@simonsan Could you also tag (if you know) someone from the OpenAge (or other projects) project who probably has researched the matter.

[puzzlepaint]

Thanks for bringing this up, it is obviously very important to me that we are on the safe side regarding legal issues.

Like @MaanooAk, from reading over the Game Content Usage Rules quickly, I however can't see a clear statement regarding game logic values, maybe I overlooked it?

If anything, "You can't reverse engineer our games to access the assets", sounds to me like, the less we access the original game data, the better.

However, in general I suppose that the game logic values may be covered by copyright - unfortunately, I know little about this.

However, I suppose that the WK mod used on Voobly might have the same issue. I think that they distributed the HD edition's balance changes as a mod to the original game. So, it seems like it must be legally doable.

[simonsan]

@heinezen Could you elaborate on this matter please. ;-)

[puzzlepaint]

To also elaborate a bit on the following: "You can't reverse engineer our games to access the assets" actually sounds to me like accessing even just the graphics data (which is stored in a proprietary format) would be forbidden. However, in practice, there are lots of external programs that somehow access game assets, including this data.

My impression (which might be totally wrong) is that according to the above-mentioned content usage rules, Microsoft would have the right to take down any such program that it is not happy with. But I guess that given how such modifications have massively contributed to keeping AoE alive, they probably don't usually enforce the rules.

Regarding the general question of whether Microsoft might be unhappy with a project such as OpenAge or FreeAge, before I released the first version of FreeAge, I contacted @TheJJ from OpenAge, and one of his statements was that they have Microsoft employees in Matrix/IRC and watchers on GitHub, and they didn't hear any criticism yet.

[simonsan]

@puzzlepaint In my personal understanding the difference is, that openage for example doesn't ship with hardcoded values from the dat-file but rather uses the converted ones. Like this it doesn't distribute copyrighted material and makes indirectly sure that people actually own the game.

But I think heinezen is the person that had to think most of it, so I think he would be able to give a more precise judgement here.

[heinezen]

Hello :)

The problem I see here is not so much related to the game content usage, but to the content distribution. The .dat file is technically a database that is copyright protected. If you have large parts of the database in your code, then you are distributing content owned by Microsoft. This can be a problem as companies are very petty about the redistribution of original content by fan projects. Microsoft has also done this in the recent past (example).

The solution to this problem is to either always access the original assets or access them once for converting them to another format. By doing this you also somewhat ensure that users need to acquire the original version of the game first.

[simonsan]

I think an important point to value here as well is that a game engine reimplementation is actually a replacement for the original game and not something like a mod or anything in addition or something to the original like an online tech tree or something, which is actually in the interest of Microsoft to keep because of community being involved. The game content usage rules are on purpose spongy to use them the way it's needed.

We can revoke this limited-use license at any time and for any reason without liability to you (and if we think you are abusing this license or its intent, you may hear from us).

So it's better to not rely on their good will in the first place, which is why I opened this topic - with good intent to this project. 👍

[MaanooAk]

So then someone could just web-crawl the wiki? or parse this https://github.com/SFTtech/openage/commit/12ab282e61c5a78e6fe14c953302ad5269f5f06c and everything is fine?

[heinezen]

The easiest thing would be to just parse the .dat file and use the values directly from there. The wiki is not guaranteed to be up-to-date, neither are the openage docs.

[simonsan]

So then someone could just web-crawl the wiki? or parse this SFTtech/openage@12ab282 and everything is fine?

That would be funny, but no. You would still have it hardcoded and distribute it inside an replacement for the original engine. That's not the same like a document, that is basically useless for playing the game because it's not an engine at all - but a document.

Furthermore as something is hardcoded the license for using that hardcoded stuff could be just revoked - doesn't matter from where you have it if it's hardcoded into files in your project. That wouldn't be possible if you use the values directly or parse. Furthermore with parsing from the web you didn't find a solution to the ownership issue, that you need to make sure that someone isn't playing with an illegal copy and just uses your engine with the cracked material - which is the biggest issue to Microsoft i feel.

[puzzlepaint]

Personally I don't really see the difference between distributing the game data values in a text document or on pages like this, and distributing them in some source code (which is also a document), but since I'm unfortunately not knowledgeable in copyright matters, I will take your word for this :)

Furthermore as something is hardcoded the license for using that hardcoded stuff could be just revoked - doesn't matter from where you have it if it's hardcoded into files in your project. That wouldn't be possible if you use the values directly or parse.

But how does reading the values from the proprietary game databases work together with the condition to not reverse-engineer the game? If I understand that correctly, I feel like there's no way to use the data without violating these rules in some way or another. However, people commonly do that, and some of the projects even seem to be embraced by Microsoft (e.g., the current Capture Age overlay, which I guess reads the memory of the game process, is being used in official tournaments). Thus, perhaps the main thing to ensure is actually that the project is in Microsoft's interest and doesn't harm them in any way?

Furthermore with parsing from the web you didn't find a solution to the ownership issue, that you need to make sure that someone isn't playing with an illegal copy and just uses your engine with the cracked material - which is the biggest issue to Microsoft i feel.

The question of whether someone could play with an illegal copy of the game is not directly affected by this, I think. FreeAge already reads the graphics files from the original, so it does require the original asset files to be present already (regardless of whether it in addition reads the game data files or not). On the other hand, reading asset files is not a sufficient means of protection against using an illegal copy, since these files can be easily copied separately.

Unfortunately, I don't see any good way to ensure that games like OpenAge / FreeAge are only used with original copies of the game. Even if they implemented some kind of ownership check, malicious people could easily remove that from the source code again.

[simonsan]

Personally I don't really see the difference between distributing the game data values in a text document or on pages like this, and distributing them in some source code (which is also a document), but since I'm unfortunately not knowledgeable in copyright matters, I will take your word for this :)

It is one thing to compile knowledge into a Wiki. This is obviously tolerated because it aids the community overall around the official franchise. The other thing is to put values and gameplay elements into a new engine and then redistribute it while it could be easily seen to stand in concurrence to the official franchise. That might be less tolerated. For obvious reasons.

In plain language: This project makes itself vulnerable to DMCA takedowns without any need just by hardcoding the dat file values in the source code. By this you are depending on the good-will of Microsoft laywers that your project is not taken down or you are sued. I think this is as far as I can take it, other internals I'm not willing to share at this point. Hope you're doing well.

[puzzlepaint]

Yes, thanks for clarifying, from this perspective it makes a difference of course. I was only discussing the (layman's) perspective regarding copyright before, not additional other implications. Just to make this clear here, I don't plan to pursue the approach to hard-code units stats and similar game logic values any further, for the reasons presented above. But I would appreciate a clear discussion on what the right thing to do is here (and what must not be done), since it is vital to this project, and so that any potential similar pitfalls can be avoided in the future.

Since my other questions from above have not been answered yet, I however somewhat got the impression that both OpenAge and FreeAge purely depend on Microsoft's goodwill anyway, as even with the approach to access original game data / assets, they cannot do anything to avoid violating at least one end-user license term respectively game content usage rule? If that is true, then it seems that this alone is also a possible reason for Microsoft to take any of the projects down at any time?

(I will not develop this project any further until this is clarified (and if the concern above can be voided).)

[heinezen]

The most common approach to avoid these issues that most major engine reimplementation project choose a clean-room design for the implementation and not distribute any of the original data with the executable. openage does it this way, as do openmw, OpenRA, RPCS3, Dolphin and others. The implication is that the engine and game data have to be separate by design. So while these engines contain the functionality of the original games, they can work independently without the original content.

Since my other questions from above have not been answered yet, I however somewhat got the impression that both OpenAge and FreeAge purely depend on Microsoft's goodwill anyway. as even with the approach to access original game data / assets, they cannot do anything to avoid violating at least one end-user license term respectively game content usage rule? If that is true, then it seems that this alone is also a possible reason for Microsoft to take any of the projects down at any time?

Any of the community projects depends on Microsoft's goodwill, but most are tolerated because they do not decrease the value of the project they are selling. Having unit data displayed in the wiki is something very different than distributing the unit data with an engine (for free) to play the same game they are selling.

[simonsan]

But I would appreciate a clear discussion on what the right thing to do is here (and what must not be done), since it is vital to this project, and so that any potential similar pitfalls can be avoided in the future.

As heinezen was stating and as was my intention as well with opening this issue: In my eyes, one possible long-term solution is less hardcoding of game data by using other projects for example genieutils to extract/convert existing information.

Another approach might be to authenticate against Steam and check for the ownership of a game and depending on the result you download a module with the hardcoded values. For sure that would be not the most clean solution, because you still distribute the data somehow, but at least you can say: "Hey, at least I'm checking beforehand, that people are owners of the game." in the hope that this will be more tolerable than distributing without checking for ownership somehow.

But the more clean solution I think is really the first approach and if you want to be sure, implement a check on top of that with Steam.

EDIT:

Even if they implemented some kind of ownership check, malicious people could easily remove that from the source code again.

From one side that is right, from the other that is not the problem of the project itself, if someone else removes this protection. For playing online with each other you could implement a check for verifying the engine files if they are

• up-to-date, so if everyone is playing on the same release (if not update)
• untampered, so if the files that are negotiating transfers etc. are original and untouched

So that only your supported versions can play with each other and no assumptions can be made, that people use the original code for playing with illegal copies. If there are forks of your code doing it differently and remove those checks that shouldn't harm your project in any way.

[puzzlepaint]

Having unit data displayed in the wiki is something very different than distributing the unit data with an engine (for free) to play the same game they are selling.

I see, but just to emphasize this, please do not forget that FreeAge depends on other (graphics) files of the original to be present from the original game. Thus, regardless of whether some game logic numbers would be included in the code or not, the project would never make the game playable for free without owning the original, which the quoted statement above might seem to imply. ( I know you probably didn't want to imply that, but I have to be clear on this matter :) )

If there are forks of your code doing it differently and remove those checks that shouldn't harm your project in any way.

Are you sure about that? Especially since we apparently agree that:

Any of the community projects depends on Microsoft's goodwill, but most are tolerated because they do not decrease the value of the project they are selling.

Given the discussion here and the Game Content Usage Rules linked above, as soon as OpenAge or FreeAge would reach the state where they are playable without lacking any serious features of the original, and as soon as possible forks might remove the Steam ownership check discussed above, I think that Microsoft would reconsider whether they tolerate these projects or not. And I'd fear that the resulting decision would not be in our favor.

Therefore, unfortunately it seems to me that the best decision is to end the project now, rather than having it ended later or even risking to be sued. If there was any way to get an official statement from Microsoft whether, and under which conditions, they would tolerate projects like OpenAge and FreeAge, that would change things. But it seems unlikely to me that it is possible to obtain such a statement.

Maybe, in the hypothetical scenario that Microsoft would abandon AoE2:DE and shut down its servers, this might also change things again, but right now when they are even still actively developing it, it seems like a bad time for a re-implementation project.

Thanks again for bringing this issue up though. The existence of all the other community projects made me think that they are on a much safer basis than they apparently actually are.

[simonsan]

If there are forks of your code doing it differently and remove those checks that shouldn't harm your project in any way. Are you sure about that? Especially since we apparently agree that:

Yes. You can read that here: https://help.github.com/en/github/site-policy/dmca-takedown-policy#b-what-about-forks-or-whats-a-fork

Any of the community projects depends on Microsoft's goodwill, but most are tolerated because they do not decrease the value of the project they are selling. Given the discussion here and the Game Content Usage Rules linked above, as soon as OpenAge or FreeAge would reach the state where they are playable without lacking any serious features of the original, and as soon as possible forks might remove the Steam ownership check discussed above, I think that Microsoft would reconsider whether they tolerate these projects or not. And I'd fear that the resulting decision would not be in our favor.

I don't think so. There is not really something that would bring you to that conclusion. As long as you don't distribute the data, as we're trying to solve in this issue. Really, the only problem here is the distribution of the copyrighted material - and that should be easy to fix as stated above now that you know what could be done about it.

Therefore, unfortunately it seems to me that the best decision is to end the project now, rather than having it ended later or even risking to be sued. If there was any way to get an official statement from Microsoft whether, and under which conditions, they would tolerate projects like OpenAge and FreeAge, that would change things. But it seems unlikely to me that it is possible to obtain such a statement.

That is a pitty, because we were trying to solve a problem here and even gave relatively clear recommendations that come from somewhat knowledgable people in this regards.

Maybe, in the hypothetical scenario that Microsoft would abandon AoE2:DE and shut down its servers, this might also change things again, but right now when they are even still actively developing it, it seems like a bad time for a re-implementation project.

There is never a wrong time to reimplement something that is closed-source. Also given the fact how young FreeAge is. I would reconsider that in your point and just do what you like - but also consider the point that this issue brought up. :-D

[puzzlepaint]

That is a pitty, because we were trying to solve a problem here and even gave relatively clear recommendations that come from somewhat knowledgable people in this regards.

Well, while the suggestions solve the issue regarding game data distribution, the discussion uncovered an underlying problem that does not seem to have a good solution:

• Free/OpenAge, without wanting to, make it comparatively easy for malicious people to play with illegal copies of the game (Steam ownership checks can be removed, game data files can be copied)
• At the same time, they depend on Microsoft's goodwill, which Microsoft might not have given the point above.

I don't see a way to address the first point other than making the project closed-source, which somewhat defeats its purpose.

[simonsan]

That is a pitty, because we were trying to solve a problem here and even gave relatively clear recommendations that come from somewhat knowledgable people in this regards.

Well, while the suggestions solve the issue regarding game data distribution, the discussion uncovered an underlying problem that does not seem to have a good solution:

* Free/OpenAge, without wanting to, make it comparatively easy for malicious people to play with illegal copies of the game (Steam ownership checks can be removed, game data files can be copied)

* At the same time, they depend on Microsoft's goodwill, which Microsoft might not have given the point above.

I don't see a way to address the first point other than making the project closed-source, which somewhat defeats its purpose.

I don't know how and where in this discussion you came to this conclusion and mindset.

I also don't understand how you can equate FreeAge and openage like this, if openage takes completely different design decisions that make it not vulnerable to these spongy Usage rules, as we are not hardcoding but converting from the originals.

That the projects are depending on the goodwill of the company where the game originates from is absolutely normal in the game engine reimplementation-world if you take other examples that were mentioned in here as well. The issue and the recommendations in here are there to let you know how to lower the attack surface for possible legal actions. Furthermore you are living in the European Union with its' jurisdiction in these regards as well. You can ask the Free Software Foundation Europe if you are still unsure.

I repeat the solution steps to your problem:

• convert/parse from original files without hardcoding and distributing copyrighted material from DAT files (to be on the safe side)

(OPTIONAL) If you want to really make sure, that you took all the steps to fight piracy:

• check for hashes of files in use for multiplayer that they are not tampered from the originals. It's not your stuff what happens in forks if your code is clean. (you can't even sell knives anymore with the logic, because they are used to kill people. Or even ballpens. You are liable for what happens in your repository, everything outside is out of your control. Period.)

• use Steamworks API to check for the ownership of the original

[puzzlepaint]

I also don't understand how you can equate FreeAge and openage like this [...]

I did not mean to equate the two projects, I just wanted to say that I think that they are equally affected by this. In case this was unclear, I am sorry.

[...] design decisions that make it not vulnerable to these spongy Usage rules [...]

I think we may have a different understanding here. My impression is that the Game Content Usage Rules apply to both FreeAge and OpenAge, but maybe I am mistaken?

• One of the rules starts with "If the Item you create and distribute is a free app, [...]", which means that they clearly apply to applications in general.
• At the beginning, the document talks about a "[...] license for you to use and display Game Content and to create derivative works based upon Game Content, [...]". This makes me think that this would apply to OpenAge's converter as well, since it "uses" that Game Content as well and the OpenAge application would later "display" it. If that is a wrong understanding, please correct me.

In case these rules do apply, then we (and virtually all other community projects) violate them, since the first rule starts with "You can't reverse engineer our games to access the assets [...]", but accessing any kind of graphics data, game logic values, or similar, necessitates to read the game's proprietary data formats. Together with the other clause "We can revoke this limited-use license at any time and for any reason without liability to you [...]" this would mean that Microsoft could easily terminate any of these projects at any time.

The inability to implement strong measures against abusing such open source projects for piracy might be a strong reason for Microsoft to aim at terminating such a project. This seems like a very bad situation that I don't want to be in. Your suggestions do not seem like a convincing solution for me since they don't change this situation.

If there is anything that I am missing, such as for example if the Game Content Usage Rules actually do not apply to OpenAge for some reason, then please let me know. Otherwise, I think I have explained sufficiently why I now think that it is not a good idea to continue developing FreeAge (or OpenAge, but that is not my decision), and I think the discussion can end here.

[simonsan]

The inability to implement strong measures against abusing such open source projects for piracy might be a strong reason for Microsoft to aim at terminating such a project. This seems like a very bad situation that I don't want to be in. Your suggestions do not seem like a convincing solution for me since they don't change this situation.

Nah, it's fine. You will not have any problems with your repository if someone implements workarounds. That fork will get a DMCA takedown not yours, quote from the link I posted here:

GitHub will not automatically disable forks when disabling a parent repository. This is because forks belong to different users, may have been altered in significant ways, and may be licensed or used in a different way that is protected by the fair-use doctrine. GitHub does not conduct any independent investigation into forks. We expect copyright owners to conduct that investigation and, if they believe that the forks are also infringing, expressly include forks in their takedown notice.

So let's stay with the facts please.

If there is anything that I am missing, such as for example if the [Game Content Usage Rules](https://www.xbox.com/en-us/developers/rules) actually do not apply to OpenAge for some reason, then please let me know. Otherwise, I think I have explained sufficiently why I now think that it is not a good idea to continue developing FreeAge (or OpenAge, but that is not my decision), and I think the discussion can end here.

In my understanding you are using the assets (or parts of it) of the original game and ship them from your repository. openage for example uses a converter that is converting the assets on the client-side and they don't leave the users computer. Furthermore it can be run without the original assets as a standalone game engine used by other developers to create their games. This is the design decision it took what we are talking here, so that there is mostly no attack surface from the legal side (not that I'm aware of).

I tried to explain you several times what is the problem and what is a solution, that other engine developers took as well. If you are not fine with I implementing these solutions, that is for sure your own decision. Just wanted to help. ;-) Cheers.

[puzzlepaint]

Nah, it's fine. You will not have any problems with your repository if someone implements workarounds.

We are misunderstanding ourselves. I was not talking about the repository getting taken down due to forks of it doing piracy, but rather about my project getting taken down because of Microsoft deciding to enforce its Game Content Usage Rules.

If you are not fine with I implementing these solutions

I am fine with implementing these measures, but I don't think that they solve all problems. This is because:

openage for example uses a converter that is converting the assets

So OpenAge clearly "uses" the assets, which supposedly makes it fall under the Game Content Usage Rules, which it violates? If so, that is not a solution.

Furthermore it can be run without the original assets

But it seems to me that the fact that it offers additional functionality does not change anything about whether its functionality for interacting with the original game assets is okay or not.

In my understanding you are using the assets (or parts of it) of the original game and ship them from your repository

Please don't exaggerate. There are a few very incomplete numbers in the source code, that are used by the game logic and also widely known and described on various webpages. As soon as you told us about this being an issue, we stopped the development immediately.

---

Let's just agree to disagree at this point. I appreciate that you wanted to help, but there seem to be a lot of misunderstandings. After all, we have the same goal, so let's not fight :) Best of luck to OpenAge.