Set Feature packages policy

How should featured packages be chosen?

PR: #323

I think the purpose should be to promote content that demonstrates a high quality of what is possible in Minetest. The selection should be varied, and should vary over time. The featured content should be content that we are comfortable recommending to a first time player.

I think there should be slightly different policies and standards for different package types. Games should have the highest standards; they're the hardest to make but are the most important to get right.

I have read and agree with most of @Wuzzy2's proposed checklist for features games (issue). However, I do wonder how many games actually meet this standard, and if it is too strict?

Here are the rules that I do consider vital:

For all:

Must be 100% FOSS, with no legally questionable licenses such as WTFPL
Must have a high res cover image screenshot (1110x624)
Must work out of the box
Must support the latest Minetest version
Must be well maintained and reasonably stable (no game-breaking or major bugs)
Bonus if the package is novel or underappreciated
Should generally be high quality, but at least good quality
should have at least 5 reviews, and be >90% positive.

Games only:

Always at least 3 featured games (we will be displaying these in the mainmenu eventually)
They should meet higher standards for quality

Yeah, I think adopting Wuzzy's checklist for games is a good idea. But a smaller list for mods and texture packs with more specific rules would be a good idea as well

Re: Your proposed rules

but at most good quality

Should be "at least"

Must have at least 5 reviews, and be >90% positive.

This conflicts with

Bonus if the package is novel or underappreciated

maintained (ideally by a group)

I don't think this is worth mentioning, most games aren't maintained by a group.

Re: @Wuzzy2's proposed checklist; MUSTs are in bold, as they are the most important Stability:

MUST: No crashes

Should be no frequent or obvious crashes, for specificity's sake. In fact, the "No game-breaking bugs" clause covers this already.

MUST: No catastrophic map breakages in the past 2 months

Every part of this is completely arbitrary. Map breakages are part of development, though in fairness development should have a separate branch. This does not prevent unforeseen bugs from occurring, however. The arbitrary 2-month point applies to the latter two map breakage SHOULD requirements as well.

MUST: Does not screw up Minetest

What does this even mean?

Usability:

SHOULD: Has a crafting guide or something similar (unless not required)

This depends entirely on the game. Having a game with plenty of recipes but no guide is perfectly valid and may be a part of desired gameplay.

^ MUST: If a crafting guide is used, it shows the correct output count

This only applies to recipes with a count or a conventional output in the first place. Frankly this entire clause should be removed.

Gameplay:

CAN: Passes the Six Hour Test (only applies to sandbox games)

I don't see anything wrong with this, just make sure it stays as CAN. There are plenty of cases where a sandbox game would not need to pass the test yet still be notable.

CAN: Players don't feel that something in the game is “lacking”

This is arbitrary, undefined, and subject to opinion. Should be removed.

Graphics:

MUST: Textures do not induce eye cancer or vomiting

I get the point but proper terminology should really be considered. Programmer art is perfectly valid, but may fall under "eye cancer" in some cases. Something like "seizure" or "nausea"-inducing may be more appropriate. Note that this is a fairly case-dependent thing.

Quality Assurance:

SHOULD: No legacy API calls

This should be "Not dependent on legacy API calls". Having them for compatibility's sake is perfectly fine.

Writing:

MUST: All items that can be obtained in normal gameplay have description set

This should only apply to games where items are expected to have a description. A game may lack descriptions on purpose.

SHOULD: Texts do not look like as if a cat walked on the keyboard

Again, proper terminology would be appreciated. Also, gibberish text can be a part of game design. The clause should instead be "text is not pointless gibberish" or similar.

SHOULD: Descriptions of things convey useful and meaningful information

Again, only applies to games that even have a description. And there may also be games with purposefully misleading or useless information. This only aplies where appropriate.

Yay, thank you for getting the discussion rolling! I really appreciate this. Let me first say that all my suggested rules are just suggestions and I am very open to being challenged. I am not sure about everything either. I am more interested in having a discussion in the first place, resulting in SOME criteria (which is better than "anything goes").

I do not think these rules need to be set in stone. They can always be changed depending on needs or new knowledge (i.e. when a rule turned out to be dumb). I think we will probably never going to catch every possible edge case anyway.

Also, when a game fails the featured game criteria, it doesn't mean it has lost its chance forever. If a newer improved version appears, a re-review might be possible. On the other hand, I think games should be featured even if we personally dislike them if they match quality criteria and many other players like them. We should be careful to NOT gatekeep games for our own personal dislike. That's easier said than done, tho.

Rubenwardy

The featured content should be content that we are comfortable recommending to a first time player.

I absolutely agree. That's the main gist of my philospohy behind all this as well. But I would be careful not be too strict here. Difficult but good games could be weeded out too harshly by that.

I think there should be slightly different policies and standards for different package types. Games should have the highest standards; they're the hardest to make but are the most important to get right.

Indeed. Honestly, I don't even care about featured mods and texture packs for now. Let's focus only on games for now, those are indeed the most important. Games are the bread and butter of Minetest.

I have read and agree with most of @Wuzzy2's proposed checklist for features games (issue). However, I do wonder how many games actually meet this standard, and if it is too strict?

I also think that probably all games currently fail under my criteria, sadly. But frankly, this is because all our current games do have quality issues. Some games come closer than others tho. Inside the Box would EASILY pass all my criteria if it would be a proper singleplayer game. Unfortunately, ITB has awkward external dependencies, highly awkward installation instructions and the code release doesn't include any levels. It's a real shame that ITB isn't a proper singleplayer game (yet).

Must be 100% FOSS, with no legally questionable licenses such as WTFPL

This needs to be clarified, "legally questionable" is a weasel word. Maybe "FSF-approved or OSI-approved license". Also clarify that 100% includes media files (sounds, graphics, etc.).

Must be well maintained and reasonably stable (no game-breaking or major bugs)

While stability is very important, asking for "well maintained" is kind of stupid. What if the game is so stable, it doesn't need maintenance? ;) I feel this is kinda redundant with "must work with latest MT release". Ideally, future Minetest releases are still fully compatible with games from like 3 version numbers ago. IMO it's also somewhat Minetest's job to ensure game stability and not "pull the rug under game devs", as hecktest likes to say. ;)

Maybe "maintenance" should be reworded to be more specific: I.e. if users find important bugs or FOSS issues, they are going to be fixed. Maybe my bug rules were too strict and could be changed. The question is not IF your game has bugs (it does, lol), but how you are going to deal with them.

should have at least 5 reviews, and be >90% positive.

Ooff, that is a hard NO from me. 90% positive is way too high, especially with our relatively low number of games in the first place, so we should be much less strict here for now. While I agree that positive user reception CAN be a good indicator for being featured, it should not be a MUST-have, especially with such a high percentage. Also, if reviews are taken into account, the review content should also be taken into account. Some reviews can be written in bad faith, like "This game sucks, it doesn't run on my Intel i386. Very disappointing!". And they are always subject to manipulation, although Minetest doesn't seem to have that problem yet. Additionally, reviews can become outdated. It's not fair to disqualify a game for poor reviews from its early alpha stages from 2 years ago.

Must have a high res cover image screenshot (1110x624)

You mean the thing that's uploaded on ContentDB or screenshot.png? Because I agree with the first but not neccessarily with the second one. The resolution is awkwardly specific but yeah, it's a good minimum. ;-) Is the aspect ratio important?

In general, I agree with all your other points.

I am disappointed you did not include the author approval. If the author themselves considers the game to be highly incomplete, you should listen to them. They probably know what they're doing, and you don't have to waste time with a full review while a game is still in alpha stage. On the other hand, this might be downgraded be a SHOULD criteria in case an author abuses this rule.

GreenXenith

MUST: Does not screw up Minetest

This is deliberately left vague. Screwing up Minetest can mean many things. Basically, no featured game should make Minetest unusable. The main idea is that when the game somehow interferes with Minetest's settings or behavior that somehow makes it unusable or interferes with any the ability to use Minetest correctly or play other games. Like a game which manages to set the Minetest font size to 1pt even after leaving the game, or completely destroys the main menu somehow. If something like this happens, it will be hard for newbies to recover. This is VERY important and should not happen in any featured game.

Yeah, I admit the crafting guide stuff is rather specific. It really depends on the type of game.

This should be "Not dependent on legacy API calls". Having them for compatibility's sake is perfectly fine.

OK.

Again, proper terminology would be appreciated. Also, gibberish text can be a part of game design. The clause should instead be "text is not pointless gibberish" or similar.

Yeah, I meant with that that we should not feature games that are riddled with unintentional typos or very poor grammar. This can really ruin an otherwise fine game.

MUST: No catastrophic map breakages in the past 2 months Every part of this is completely arbitrary. Map breakages are part of development, though in fairness development should have a separate branch. This does not prevent unforeseen bugs from occurring, however. The arbitrary 2-month point applies to the latter two map breakage SHOULD requirements as well.

Catastrophic map breakages are map breakages like that when you update the game, every node turns into an unknown node. Or worse, the mapgen overrides already built structures, or empties inventories, and it's the game's fault (not Minetest's). What I do NOT mean is stuff like continuity errors (those are still annoying, but not catastrophic because the map's still playable).

This stuff must obviously be avoided, it can be really, really frustrating for players to lose their map after an update although the game is feature. If every update causes map breakage, then the game does not deserve "featured" status yet because it's too heavy in flux and players can't trust compability. But players must trust that their world will still work after an update.

Yes, 2 months are highly arbitrary, can be turned into something vague in that it should ideally never happen but when it happens, only in exceptional cases (like major releases), and definitely not part of the "normal" release process (i.e. EVERY update breaks your map). If catastrophic map breakages are happening frequently, that's a sign the game's just quite not ready yet. Longterm stability IS important.

Hades Revisited is the perfect example that would be currently fail this criterion. Many updates in the past caused heavy map breakage because of heavly alpha-stage development. This rule will easily filter out many unstable games, which is a good thing, actually.

SHOULD: Descriptions of things convey useful and meaningful information Again, only applies to games that even have a description. And there may also be games with purposefully misleading or useless information. This only aplies where appropriate.

Hmm, yeah, I agree this is kind of arbitrary. It depends a lot on the genre. I added this because I often was frustrated by lots of low-quality item descriptions that are confusing for no reason. Or when they literally just show the itemstring. I am aware that misleading names can be intentional, which is why it's only a SHOULD. I wouldn't mind downgrading it down to a CAN.

CAN: Players don't feel that something in the game is “lacking”

This is arbitrary, undefined, and subject to opinion. Should be removed.

A lot of CAN criteria are subjective, and this is intentional. At the end of the day, if we only look at purely technical criteria, we fail to weed out games that formally match quality criteria but are just super boring to many. The idea basically is related to the Big Empty Sandbox problem.

CAN: Passes the Six Hour Test (only applies to sandbox games)

I don't see anything wrong with this, just make sure it stays as CAN. There are plenty of cases where a sandbox game would not need to pass the test yet still be notable.

Yeah, I agree it should always be only a CAN at most. Six Hours is an arbitrary line. My opinion is that if a sandbox game runs out of interesting content way before 6 hours, it's probably not very interesting and gets boring too fast. "content" can be interpreted broadly: Blocks, items, biomes, things you can do, whatever. Anything that is interesting. Minetest Game currently fails the Six Hour Test badly and this is the main reason why many newbies are so disappointed. The Jimquisition talked about the problem of Big Empty Sandboxes i.e. when sandbox games advertise themselves how big their worlds are but turn out to be shallow of empty because it looks mostly the same everywhere or you can't do a lot of things in the big world. This is what the Six Hour Test tries to avoid. If a game fails the Six Hour Test, it's very likely also a Big Empty Sandbox. We should avoid featuring Big Empty Sandboxes, even if they formally meet all the other criteria.

MUST: Textures do not induce eye cancer or vomiting

I get the point but proper terminology should really be considered.

Yeah it's mostly a joke. Obviously the official guidelines use proper wording. It's mostly if the graphics are OBVIOUSLY horrible and there's wide agreement about that. I'm thinking of very-low-effort, 20 seconds MS Paint imagery.

(crafting guide stuff)

I agree, it's all too specific and really depends on the game. I agree with removal.

Indeed. Honestly, I don't even care about featured mods and texture packs for now. Let's focus only on games for now, those are indeed the most important. Games are the bread and butter of Minetest.

I agree that games are the most important, certainly

Inside the Box would EASILY pass all my criteria if it would be a proper singleplayer game. Unfortunately, ITB has awkward external dependencies, highly awkward installation instructions and the code release doesn't include any levels. It's a real shame that ITB isn't a proper singleplayer game (yet).

Yeah, I really wish ITB was a singleplayer game. It could use the HTTP API to download/upload boxes for that functionality, and also the leaderboard.

Must be 100% FOSS, with no legally questionable licenses such as WTFPL

This needs to be clarified, "legally questionable" is a weasel word. Maybe "FSF-approved or OSI-approved license". Also clarify that 100% includes media files (sounds, graphics, etc.).

CC BY-SA 3.0 isn't a FSF or OSI approved license though - they only reviewed and approved 4.0. I think 3.0 should be allowed

Must be well maintained and reasonably stable (no game-breaking or major bugs)

While stability is very important, asking for "well maintained" is kind of stupid. What if the game is so stable, it doesn't need maintenance? ;) I feel this is kinda redundant with "must work with latest MT release". Ideally, future Minetest releases are still fully compatible with games from like 3 version numbers ago. IMO it's also somewhat Minetest's job to ensure game stability and not "pull the rug under game devs", as hecktest likes to say. ;)

Maybe "maintenance" should be reworded to be more specific: I.e. if users find important bugs or FOSS issues, they are going to be fixed. Maybe my bug rules were too strict and could be changed. The question is not IF your game has bugs (it does, lol), but how you are going to deal with them.

In my local draft, I have changed this to the following two points:

MUST: Is well maintained (author is present and active).
MUST: Be reasonably stable, with no game-breaking or major bugs.

Perhaps "well" can be removed, or replaced with "sufficiently". The idea is that the author (or team) is still present to merge PRs and fix bugs, even if they're not actively making features

should have at least 5 reviews, and be >90% positive.

Ooff, that is a hard NO from me. 90% positive is way too high, especially with our relatively low number of games in the first place, so we should be much less strict here for now. While I agree that positive user reception CAN be a good indicator for being featured, it should not be a MUST-have, especially with such a high percentage. Also, if reviews are taken into account, the review content should also be taken into account. Some reviews can be written in bad faith, like "This game sucks, it doesn't run on my Intel i386. Very disappointing!". And they are always subject to manipulation, although Minetest doesn't seem to have that problem yet. Additionally, reviews can become outdated. It's not fair to disqualify a game for poor reviews from its early alpha stages from 2 years ago.

This is a SHOULD, rather than a MUST. I added it as it's a good metric for "at least 1 player enjoys this".

Must have a high res cover image screenshot (1110x624)

You mean the thing that's uploaded on ContentDB or screenshot.png? Because I agree with the first but not neccessarily with the second one. The resolution is awkwardly specific but yeah, it's a good minimum. ;-) Is the aspect ratio important?

The cover image is the image that displays on the homepage if you're featured, and on the package page's hero banner (the big full width image at the top). It can be set on the screenshots page, and doesn't need to match screenshot.png/the thumbnail.

I've changed the min resolution to 1920x1080 in my draft, this is much more standard and matches a typical screen resolution. The aspect ratio doesn't matter - it'll be cropped to 16:9 on the homepage and shorter on the package page. I have been thinking that it could be worth splitting these, I guess, as these are two different uses

I am disappointed you did not include the author approval. If the author themselves considers the game to be highly incomplete, you should listen to them. They probably know what they're doing, and you don't have to waste time with a full review while a game is still in alpha stage. On the other hand, this might be downgraded be a SHOULD criteria in case an author abuses this rule.

Sorry, I considered this obvious and forgot to include it

Yeah, I admit the crafting guide stuff is rather specific. It really depends on the type of game.

I think it's perfectly fine to have criteria that depends on the type of game

CAN: Players don't feel that something in the game is “lacking”

This is arbitrary, undefined, and subject to opinion. Should be removed.

A lot of CAN criteria are subjective, and this is intentional. At the end of the day, if we only look at purely technical criteria, we fail to weed out games that formally match quality criteria but are just super boring to many. The idea basically is related to the Big Empty Sandbox problem.

My problem with this criterium is that players will always say that something is lacking in the game.

I've opened a PR that is Wuzzy's list with some modifications - I've merged somethings, and added other criteria

Heh, nice. :D

One important part is still missing: What is the process to actually turn a game into a featured game? Like, who makes the review, who adds the tag, etc. Is there some discussion or something? And when are games going to get un-featured?

Like going through the checklist is easy, of course, but there still needs to be the final decision whether to actually feature the game.

Also note I am an editor AND a gamedev. That might be a problem. ;-)

I suggest that once the policy goes through to start a discussion of the first games to feature.

One important part is still missing: What is the process to actually turn a game into a featured game? Like, who makes the review, who adds the tag, etc. Is there some discussion or something? And when are games going to get un-featured?

Like going through the checklist is easy, of course, but there still needs to be the final decision whether to actually feature the game.

It should not fall to one person - should be done based on a board of editors (or another group)

Not sure how to do it though, I'd rather not use an issue on this repository because it 1) clutters the repo with non-dev stuff and 2) require use of GitHub, a proprietary platform.

Perhaps the author could create a thread on the repo, I can add a simple UI for this that pre-populates a thread with the criteria for them to fill out and justify. Editors can then approve/reject. It should be possible to submit for featured review multiple times

Also note I am an editor AND a gamedev. That might be a problem. ;-)

Being a game dev is fine, however there should be measures to prevent conflict of interest

I suggest that once the policy goes through to start a discussion of the first games to feature.

Certainly

It may also make sense to establish a periodic review process, e.g. once a month or so, editors will meet (or at least make comments available to the rest of the editor team by some deadline) to:

Review existing Featured packages and decide if they still rise to the level necessary, or whether requirements have changed (e.g. the policy has changed to raise the bar), or if the package has suffered quality issues lately, stagnated, changed design/direction, not updated to the latest engine version, or received new negative feedback due to discovered issues.
Determine if we have the correct number of Featured packages. If there are too many, consider whether we should cull some, and/or tighten the standards. If too few, try to identify candidates and invite their authors to apply.

Discussion about the content of the policy seems to have quiesced, so I think we should move onto an approval/ratification phase. I suggest that we:

Prune out anything still controversial from this PR and move it to a separate PR, and continue discussion there.
Get what we agree on so far approved, merged, and start implementing it.

I don't know what is still controversial in the PR, as I've not received any further comments after changes

minetest / contentdb

Set Feature packages policy #321

Rubenwardy

GreenXenith