[Tracking Issue/Discussion] The Ecosystem Guide

[Lokathor]

I know that there's an open PR for the start of an ecosystem guide, but since this is one of our goals before the next meeting I wanted to have a general discussion and get people's thoughts about the subject without totally clogging up the review of kvark's specific PR.

The general subject of this issue is anything related to the gamedev-wg doing an ecosystem guide.

Assuming that we do make an ecosystem guide, I feel compelled to ask: What format should such a guide take?

1) Obvious first point: AreWeGameYet is already an ecosystem guide.

• We don't maintain it, though this is not itself a problem because it does get active attention.
• There's a lot of open issues but not many open PRs. It seems like once someone does the PR work that PR gets merged fairly promptly, but the owner isn't constantly putting in the new entries themselves.
• AreWeGameYet is fairly un-opinionated regarding one library or another, it just sorta throws everything into the pile with minimal descriptions and lets readers figure things out for themselves. 2) The Charter of this WG fairly specifically says that we should not be promoting one library over another.
• AreWeGameYet fills the space of an un-opinionated list of everything.
• The only good space after that is to have an opinionated list of everything (and perhaps some things don't even make it onto the list, which is itself an opinion).

Combined, these two items make me want to back up and ask: Do we make an ecosystem guide at all?

• I think that there should be an ecosystem guide that's says a lot more about each particular library in the guide. Like the Guide To Rust Graphics in 2019 that @icefoxen wrote. That article is really good, and frank opinions about the fitness for use of major crates is really valuable advice for the community.
• However, as soon as you say anything more than basic facts about each crate you start having to have an opinion about things, which is pretty close to "promoting one library over another", which the charter says for the WG to not do.
• I do recall from seeing some early discussions before the WG was formed that some folks wanted the WG to rally the community around one specific library for each particular programming element. Getting everyone to pour all effort into Amethyst or Nalgebra or whatever library. That clause in the charter seems to be a reaction to that idea, and a lesser statement of just saying "these libraries are actually fit for use for developing games right now [list here]" might be acceptable in the long term.

My conclusion from these facts: Either we have to change the charter at least a bit (not an action to be taken lightly!!) or we can't have an ecosystem guide that is usefully different from AreWeGameYet, and we probably shouldn't do a separate guide at all. Instead, we should just direct people to work on AreWeGameYet.

Thoughts on this? Or thoughts on anything else about the subject of an Ecosystem Guide?

[17cupsofcoffee]

I'm a bit torn on this!

On the one hand, I feel like there's a massive need for a semi-official 'state of the ecosystem' post; something we can point new gamedevs to as a starting point that's not quite as overwhelming as AreWeGameYet. On the other, I think the WG not being too biased towards any one set of tools is really important.

For example, Icefoxen's guide to Rust game engines was pretty critical of Piston. Now, I think those criticisms were extremely fair, but there's a big difference between "this one guy says you shouldn't use Piston" and "this group that is somewhat officially linked with Rust says you shouldn't use Piston", and I'm not sure if I'd be comfortable with the latter. So that's the tightrope that needs to be walked, I guess.

[icefoxen]

I sure as hell wouldn't want to proselytize my graphics library guide as an official position, and I heckin' wrote it. To write something like that, which IS baldly critical, you have to be unencumbered by other concerns and know that you are only speaking for yourself. It's much easier to say the things that you feel must be said when you are saying them from the outside, without any official status.

Easy option for the matter of opinion: have the ecosystem guide be a collection of other people's guides, without endorsing or criticizing particular ones. Just organize them by topic and note that some of them are opinionated. The WG could serve a useful role by having this index, and keeping it up to date, since guides go stale and get updated and so on.

Also for any collection of information there need to be clear rules for removing things as well; one of my biggest annoyances with arewegameyet is there's lots of stuff like mold2d and corange-rs which never went anywhere but they're still included alongside mature and actively developed stuff that's actually usable.

[17cupsofcoffee]

Easy option for the matter of opinion: have the ecosystem guide be a collection of other people's guides, without endorsing or criticizing particular ones. Just organize them by topic and note that some of them are opinionated. The WG could serve a useful role by having this index, and keeping it up to date, since guides go stale and get updated and so on.

I'm super on board with this idea, both for ecosystem guides and just guides/blog posts in general.

[Lokathor]

Yeah, if we end up with one guide we certainly don't want to go full Icefox spicy curry level of opinion with it. Something closer to a plain teriyaki level of opinion would potentially be appropriate: a useful guide for people who aren't sure where to start and who want someone to just suggest something to start with and where they might want to look, but not too strongly worded so that people don't get totally scared off of going their own route.

But I do strongly support the "list of guides" approach, now that I'm thinking about it.

[aclysma]

• I agree that this workgroup should not promote one library over another in a general sense. (i.e. "X" is broadly better than "Y")
• If we want to have a list of crates somewhere, positive statements-in-fact, if used carefully, might be a way to help people find the best crates for them without actually endorsing them. For example, "abc has many regular contributors and is sponsored by A, B, and C." This way we can help someone make their own choice of where to start without feeling unfair or biased to anyone. (If we go this route, we need to do it carefully!)
• Endorsement by individuals (i.e. someone's personal blog) could fill the need for more opinionated advice. Maybe we should encourage people to write more. (I'm planning writing a quick run-down of what crates I would use for a handful of common situations.) Ideally, there would be more than one. :)
  • Blog posts, etc. that are helpful could be linked to in WG communications like a monthly newsletter (as long as it's clear the link goes to a non-WG resource that does not represent the WG as a whole.)
• As long as AreWeGameYet continues to be alive and well, and occupies the space of "unopinionated inventory of gamedev-relevant crates", I think it's fine for the WG to recommend that as a resource and help maintain it. (i.e. put in PRs). If it's status ever changes (it dies, it is no longer unopinionated, etc.), there's no reason the WG can't start providing an unopinionated list of its own at that point.

[icefoxen]

Now that I think of it, can we get the owner of arewegameyet, @doppioslash , in on this conversation to tell us more about their current status and what they want arewegameyet to be? I think that it can use lots of work but it also seems very much a passive maintenance effort, and there's stuff there that hasn't been updated since pre-Rust-1.0 days that I'd really like to clean out. (Hopefully it's not an opinionated statement to say that having corange-rs in the rust-gamedev/wg#3 spot under "game engines" isn't really helping anyone.)

Could a semi-concerted effort by the WG to either rework the site or just make a bunch of PR's to bring it up to snuff be something people would consider?

[doppioslash]

Personally, I'd want arewegameyet to stay an un-opinionated repository of everything that may be useful. If you're going to make pull requests updating obsolete parts, they will be merged. I don't like to remove libraries, but we could put on them an "obsolete" tag if they haven't been updated for a long time. Some parts of arewegameyet work better than others. I think the list of libraries works well. We ended up having a list of games and articles too. That felt like a good idea at the time, but they feel less useful and harder to keep updated now that the community is bigger.

[doppioslash]

The lists could also be reordered to put the most useful libraries on top, though I'm not keen on the piles of discussion that may entail. If the wg manages to decide on an order, then we can reorder them.

[icefoxen]

Yeah there was some discussion of that on the unofficial Discord. The acceptable options seemed to be "do nothing", "write code to order them randomly on each page load", or "write code to order them by how lately they have been updated and assume nobody's enough of a dick to try to game it". There's probably other good solutions; I'd be fine just by ordering them alphabetically or by some other obvious-and-arbitrary criteria as long as there was an "obsolete" tag or subsection to make it easier to search.

[aclysma]

• crates.io has a metric "recent downloads" - but it has its own problems. A project with a popular downstream project will see a lot of incidental downloads.
• Showing time (days/weeks/months) sounds useful even if the list wasn't sorted by that.

[Lokathor]

crates.io can show reverse-dependencies, which seems like a fair starting point.

[HeroesGrave]

Just throwing in my 2 cents:

My ecs crate is currently # 2 on arewegameyet despite its (relatively) terrible performance in benchmarks and the fact that I haven't been updating it for years other than merging the rare pull request from people using it. I have no interest in updating it because in order to make it "good" I had to rewrite it from scratch and now it's a completely different library (also specs is probably better).

I think a possible solution (which would require some effort) would be to run a regular "audit" of crates in the ecosystem to check for inactive projects and (only if they've been inactive for a while) contact the owner(s) to see what they think of the state of their project and in particular, whether people should consider using it for new projects. Crates that aren't intended to be used for new projects or whose owners have gone AWOL can be marked as such and put in a less prominent position in any sort of list/ranking (whether that's arewegameyet or some other resource).

While you could "game" such a system, there aren't really any benefits to doing so, and I think this would at least filter out a lot of projects that the owners are "done" with (which I think is the primary problem here). The main question is whether this approach would be worth the effort.

[doppioslash]

One idea I had was to add a bit of frontend code, and make the list sortable by various criteria.

[kvark]

Thank you for filing this @Lokathor and not derailing my PR :) :heart:

There is an obvious conflict: having a guide would have a great value for the community but we the WG can't be opinionated. Listing all the things like AreWeGamesYet isn't helpful enough to navigate the ecosystem, it's just better than doing a "crates.io" search. Having an opinionated guide is not helpful, since it will be changing often and there is no way to make all the participants happy.... But I think, there is a middle ground we could take to get the best of both sides.

We could list the existing libraries with some extra information that is:

1. objective (!): meaning that it's in scope of the WG
2. concisely useful: meaning that the reader would be able to make an informed judgement based on the facts we present. We can state the status, major pieces of the tech stack and features.
3. not found on crates.io: we'll need to do some digging and maintenance, to a reasonable degree. For example, games and apps using a library typically don't upload themselves to crates, so they don't show up in reverse dependencies.

Here is an example of such an entry:

Three-rs:

• crate and repository
• description: "Three.js inspired 3D engine in Rust"
• status: passively maintained by the author
• users: MicViz, transit, Bimensal, plane-splitter
• priorities: ease of use, pure Rust (edit: could be tags?)
• features: 3D graphics, animations, full glTF support, sounds
• tech: graphics = gfx pre-ll, ECS = froggy, math = cgmath

Note that some filtering needs to take place when talking about priorities, features, and (most importantly) users, e.g. I didn't include ones that don't have a screenshot in the Readme (with an exception of "plane-splitter" which I have screenshots for).

[AlexEne]

Anyone against me moving this on the website repo?

I think that's a reasonable place for the ecosystem guide to have a page on once it's done. T1he PR has already started, but it's markdown so once that gets merged it's a matter of just opening a PR with adding it to the site too.

I'm on an issue cleaning spree :)

[17cupsofcoffee]

I think given that this guide is presumably going to take the form of a page or a blog post on the site, and given that discussion of the newsletter/intro post are already on that repo, moving it there would make sense.

That said, if we're going to start discussing all site-related stuff on that repo, it might be worth drawing attention to that fact on the Discord/maybe in the README of this repo, so that people know to look there as well as here!

[Wodann]

During our biweekly meeting we coined multiple ways of improving AreWeGameYet organisation:

1. Indicate the "liveness": if something is outdated versus recently updated
2. Classify with features that we (the community and the working group) all agree on. Modifying the feature labels would need to be peer-reviewed so that the classification stays objective at large.
3. Give a clue about the tech stack ("main" dependencies) as well as compatibility (e.g. "this works with winit").

Complementary to that we'll create a mapping of the ecosystem similar to Jason Gregory's runtime game engine architecture map. This will allow us to identify missing pieces.

How this will actually take shape is to be decided, but we'd really appreciate AreWeGameYet's input (@doppioslash) in the discussion. Let me coin a first proposal, inspired by the above thread. Feel free to propose additions, improvements, or provide arguments as to why these are terrible ideas.

1. Liveness

• Add a flag to indicate whether the crate is actively/passively/not maintained
• Add minimum Rust version to crate overview
• Add date of last update to crate overview
• Add git statistics, such as number of forks, stars, and latest commit (to give an indication of developer activity)
• Allow sorting of crates based on downloads, minimum Rust version, date of last update, and git stats

Caveat: Depending on how much information is added, the UI could become too crowded. As such this might require a redesign.

2. Feature comparison

AreWeGameYet already subdivides the ecosystem into 19 different categories. After reviewing the categories to reflect all aspects of a game engine, we'll try to specify a set of feature tags for each of the categories. For every category we will then:

• tag the supported features of each crate
• tag the supported operating systems of each crate
• show a comparison table of the crates' features

The tricky part is that merely "having" a ton of features might not be the best criteria. E.g. for an ECS you might be interested in the performance of a lot of entities vs a lot of components. Does anyone have any good ideas on how to reflect these objectively? Maybe

• Add some kind of quantitative measures in the overview
• Add an Additional Resources section that links to in-depth comparisons/blogs

3. Compatibility

• Add nightly/stable to crate overview

• Show tech stack by mentioning dependencies one layer down in the ecosystem; e.g. according to Jason Gregory's runtime game engine architecture map. An example would be:

• Animation Crate A, with ragdoll physics (depends on Physics Crate B)

• Physics Crate B (depends on Math Crate C)

This way one can jump through the ecosystem from A to B and from B to C

[aclysma]

Personally, I like a lot of the ideas you've proposed. (Although it may be a lot of work!)

Add a flag to indicate whether the crate is actively/passively/not maintained

I think this would be useful, but it sounds difficult to do in an automated way, and I think it would be liable to rot. Would be really nice to have this if possible though.

Add an Additional Resources section that links to in-depth comparisons/blogs

Linking to other resources could be a practical way to be helpful while staying unopinionated as a WG. (For example, if there's a good "what are people using for physics" reddit thread and a there are a lot of good responses, that might be handy for someone taking a look around the ecosystem.) There may not be that many resources we can point to just yet, but I could see this improving as the ecosystem matures.

[Wodann]

@aclysma I agree that it is a lot of work. I was hoping to prioritise the list based on feedback and implement accordingly.

The actively/passively maintained flags would be set upon insertion of a new crate, as proposed in @kvark 's format. Ideally, maintainers would change the status once they stop maintaining their repo - similar to how they sometimes add a notification in their README. I agree with you that in reality it will probably mean some maintenance work for the WG though. I.e. review the status of inactive repos every month or so.

@doppioslash I realise that there are a lot of proposed ideas. Could you give some insight into what aligns with AreWeGameYet's vision, and what is achievable?

[icefoxen]

Crates.io already has a "maintenance status" tag, we could just display that if we trust the owners of the crates to put meaningful stuff there.

[Lokathor]

I didn't know such a thing exists, so I'm not sure we should trust people to use it right until we publicize it a lot more.

[Wodann]

@icefoxen That's a good one!

For completeness:

# Optional specification of badges to be displayed on crates.io.
#
# - The badges pertaining to build status that are currently available are
#   Appveyor, CircleCI, GitLab, Azure DevOps and TravisCI.
# - Available badges pertaining to code test coverage are Codecov and
#   Coveralls.
# - There are also maintenance-related badges based on isitmaintained.com
#   which state the issue resolution time, percent of open issues, and future
#   maintenance intentions.
#
# If a `repository` key is required, this refers to a repository in
# `user/repo` format.
[badges]