search

a327ex / blog

gamedev blog
3.27k stars 143 forks source link

Programming lessons learned from making my first game and why I'm writing my own engine in 2018 #31

Open a327ex opened 6 years ago

a327ex commented 6 years ago

I just released my first game, BYTEPATH, and I thought it would be useful to write down my thoughts on what I learned from making it. I'll divide these lessons into soft and hard: soft meaning more software engineering related ideas and guidelines, hard meaning more technical programming stuff. And then I'll also talk about why I'm gonna write my own engine.


Soft Lessons

For context, I've been trying to make my own games for about 5-6 years now and I have 3 "serious" projects that I worked on before this one. Two of those projects are dead and failed completely, and the last one I put on hold temporarily to work on this game instead. Here are some gifs of those projects:

The first two projects failed for various reasons, but in the context of programming they ultimately failed (at least from my perspective) because I tried to be too clever too many times and prematurely generalized things way too much. Most of the soft lessons I've learned have to do with this, so it's important to set this context correctly first.


Premature Generalization

By far the most important lesson I took out of this game is that whenever there's behavior that needs to be repeated around to multiple types of entities, it's better to default to copypasting it than to abstracting/generalizing it too early.

This is a very very hard thing to do in practice. As programmers we're sort of wired to see repetition and want to get rid of it as fast as possible, but I've found that that impulse generally creates more problems than it solves. The main problem it creates is that early generalizations are often wrong, and when a generalization is wrong it ossifies the structure of the code around it in a way that is harder to fix and change than if it wasn't there in the first place.

Consider the example of an entity that does ABC. At first you just code ABC directly in the entity because there's no reason to do otherwise. But then comes around another type of entity that does ABD. At this point you look at everything and say "let's take AB out of these two entities and then each one will only handle C and D by itself", which is a logical thing to say, since you abstract out AB and you start reusing it everywhere else. As long as the next entities that come along use AB in the way that it is defined this isn't a problem. So you can have ABE, ABF, and so on...

But eventually (and generally this happens sooner rather than later) there will come an entity that wants AB*, which is almost exactly like AB, but with a small and incompatible difference. Here you can either modify AB to accommodate for AB*, or you can create a new thing entirely that will contain the AB* behavior. If we repeat this exercise multiple times, in the first option we end up with an AB that is very complex and has all sorts of switches and flags for its different behaviors, and if we go for the second option then we're back to square one, since all the slightly different versions of AB will still have tons of repeated code among each other.

At the core of this problem is the fact that each time we add something new or we change how something behaves, we now have to do these things AGAINST the existing structures. To add or change something we have to always "does it go in AB or AB*?", and this question which seems simple, is the source of all issues. This is because we're trying to fit something into an existing structure, rather than just adding it and making it work. I can't overstate how big of a difference it is to just do the thing you wanna do vs. having to get it to play along with the current code.

So the realization I've had is that it's much easier to, at first, default to copypasting code around. In the example above, we have ABC, and to add ABD we just copypaste ABC and remove the C part to do D instead. The same applies to ABE and ABF, and then when we want to add AB*, we just copypaste AB again and change it to do AB* instead. Whenever we add something new in this setup all we have to do is copy code from somewhere that does the thing already and change it, without worrying about how it fits in with the existing code. This turned out to be, in general, a much better way of doing things that lead to less problems, however unintuitive it might sound.


Most advice is bad for solo developers

There's a context mismatch between most programming advice I read on the Internet vs. what I actually have learned is the better thing to do as a solo developer. The reason for this is two fold: firstly, most programmers are working in a team environment with other people, and so the general advice that people give each other has that assumption baked in it; secondly, most software that people are building needs to live for a very long time, but this isn't the case for an indie game. This means that most programming advice is very useless for the domain of solo indie game development and that I can do lots of things that other people can't because of this.

For instance, I can use globals because a lot of the times they're useful and as long as I can keep them straight in my head they don't become a problem (more about this in article #24 of the BYTEPATH tutorial). I can also not comment my code that much because I can keep most of it in my head, since it's not that large of a codebase. I can have build scripts that will only work on my machine, because no one else needs to build the game, which means that the complexity of this step can be greatly diminished and I don't have to use special tools to do the job. I can have functions that are huge and I can have classes that are huge, since I built them from scratch and I know exactly how they work the fact that they're huge monsters isn't really a problem. And I can do all those things because, as it turns out, most of the problems associated with them will only manifest themselves on team environments or on software that needs to live for long.

So one thing I learned from this project is that nothing went terribly wrong when I did all those "wrong" things. In the back of my head I always had the notion that you didn't really need super good code to make indie games, given the fact that so many developers have made great games that had extremely poor code practices in them, like this:

And so this project only served to further verify this notion for me. Note that this doesn't mean that you should go out of your way to write trash code, but that instead, in the context of indie game development, it's probably useful to fight that impulse that most programmers have, that sort of autistic need to make everything right and tidy, because it's an enemy that goes against just getting things done in a timely manner.


ECS

Entity Component Systems are a good real world example that go against everything I just mentioned in the previous two sections. The way indie developers talk about them, if you read most articles, usually starts by saying how inheritance sucks, and how we can use components to build out entities like legos, and how that totally makes reusable behavior a lot easier to reuse and how it makes literally everything about your game easier.

By definition this view that people push forward of ECS is talking about premature generalization, since if you're looking at things like legos and how they can come together to create new things, you're thinking in terms of reusable pieces that should be put together in some useful way. And for all the reasons I mentioned in the premature generalization section I deeply think that this is VERY WRONG!!!!!!!! This very scientific graph I drew explains my position on this appropriately:

As you can see, what I'm defending, "yolo coding", starts out easier and progressively gets harder, because as the complexity of the project increases the yolo techniques start showing their problems. ECS on the other hand has a harder start because you have to start building out reusable components, and that by definition is harder than just building out things that just work. But as time goes on the usefulness of ECS starts showing itself more and eventually it beats out yolo coding. My main notion on this is that in the context of MOST indie games, the point where ECS becomes a better investment is never reached.

As for the context mismatch part of this, if this article gets any traction anywhere, there will surely be some AAA developer in the comments who goes like "I have 2️⃣ 0️⃣ years of 👨‍🏫experience👨‍🏫 in the industry 🎲🎮👾 and what this 👶 kid is saying is 👇😡TRASH!!😡👇 ECS is 💯very💯 useful and I've shipped 🚢☁️🌧 multiple 🥇AAA games that made 🤑 💵 millions of dollars 💸💰 and are played by 🌎billions of people🌏 all over the world!! 🗺 Stop ✋👮 this 🚓 nonsense at once!!🚓😠💢"

And while the AAA developer has a point that ECS was useful for him, it's not necessarily true that it will be useful for me or for other indie developers, since because of the context mismatch, both groups are solving very different problems at some level.

Anyway, I feel like I've made my point here as well as I could. Does this mean I think anyone who uses ECS is stupid and dumb? No 🙂 I think that if you're used to using ECS already and it works for you then it's a no-brainer to keep using it. But I do think that indie developers in general should think more critically about these solutions and what their drawbacks are. I feel like this point made by Jonathan Blow is very relevant (in no way do I think he agrees with my points on ECS or I am saying or implying that he does, though):


Avoiding separating behavior into multiple objects

One of the patterns that I can't seem to break out of is the one of splitting what is a single behavior into multiple objects. In BYTEPATH this manifested itself mostly in how I built the "Console" part of the game, but in Frogfaller (the game I was making before this one) this is more visible in this way:

This object is made up of the main jellyfish body, each jellyfish leg part, and then a logical object that ties everything together and coordinates the body's and the leg's behaviors. This is a very very awkward way of coding this entity because the behavior is split between 3 different types of objects and coordination becomes really hard, but whenever I have to code an entity like this (and there are plenty of multi-body entities in this game), I naturally default to doing it this way.

One of the reasons I naturally default to separating things like this is because each physics object feels like it should be contained in a single object in code, which means that whenever I need to create a new physics object, I also need to create a new object instance. There isn't really a hard rule or limitation that this 100% has to be this way, but it's just something that feels very comfortable to do because of the way I've architectured my physics API.

I actually have thought for a long time on how I could solve this problem but I could never reach a good solution. Just coding everything in the same object feels awkward because you have to do a lot of coordination between different physics objects, but separating the physics objects into proper objects and then coordinating between them with a logical one also feels awkward and wrong. I don't know how other people solve this problem, so any tips are welcome!!


Hard Lessons

The context for these is that I made my game in Lua using LÖVE. I wrote 0 code in C or C++ and everything in Lua. So a lot of these lessons have to do with Lua itself, although most of them are globally applicable.

nil

90% of the bugs in my game that came back from players had to do with access to nil variables. I didn't keep numbers on which types of access were more/less common, but most of the time they have to do with objects dying, another object holding a reference to the object that died and then trying to do something with it. I guess this would fall into a "lifetime" issue.

Solving this problem on each instance it happens is generally very easy, all you have to do is check if the object exists and continue to do the thing:

if self.other_object then
    doThing(self.other_object)
end

However, the problem is that coding this way everywhere I reference another object is way overly defensive, and since Lua is an interpreted language, on uncommon code paths bugs like these will just happen. But I can't think of any other way to solve this problem, and given that it's a huge source of bugs it seems to be worth it to have a strategy for handling this properly.

So what I'm thinking of doing in the future is to never directly reference objects between each other, but to only reference them by their ids instead. In this situation, whenever I want to do something with another object I'll have to first get it by its id, and then do something with it:

local other_object = getObjectByID(self.other_id)
if other_object then
    doThing(other_object)
end

The benefit of this is that it forces me to fetch the object any time I want to do anything with it. Combined with me not doing anything like this ever:

self.other_object = getObjectByID(self.other_id)

It means that I'll never hold a permanent reference to another object in the current one, which means that no mistakes can happen from the other object dying. This doesn't seem like a super desirable solution to me because it adds a ton of overhead any time I want to do something. Languages like MoonScript help a little with this since there you can do this:

if object = getObjectByID(self.other_id) 
    doThing(object)

But since I'm not gonna use MoonScript I guess I'll just have to deal with it :rage:


More control over memory allocations

While saying that garbage collection is bad is an inflammatory statement, especially considering that I'll keep using Lua for my next games, I really really dislike some aspects of it. In a language like C whenever you have a leak it's annoying but you can generally tell what's happening with some precision. In a language like Lua, however, the GC feels like a blackbox. You can sort of peek into it to get some ideas of what's going on but it's really not an ideal way of going about things. So whenever you have a leak in Lua it's a way bigger pain to deal with than in C. This is compounded by the fact that I'm using a C++ codebase that I don't own, which is LÖVE's codebase. I don't know how they set up memory allocations on their end so this makes it a lot harder for me to have predictable memory behavior on the Lua end of things.

It's worth noting that I have no problems with Lua's GC in terms of its speed. You can control it to behave under certain constraints (like, don't run for over n ms) very easily so this isn't a problem. It's only a problem if you tell it to not run for more than n ms, but it can't collect as much garbage as you're generating per frame, which is why having the most control over how much memory is being allocated is desirable. There's a very nice article on this here http://bitsquid.blogspot.com.br/2011/08/fixing-memory-issues-in-lua.html and I'll go into more details on this on the engine part of this article.


Timers, Input and Camera

These are 3 areas in which I'm really happy with how my solutions turned out. I wrote 3 libraries for those general tasks:

And all of them have APIs that feel really intuitive to me and that really make my life a lot easier. The one that's by far the most useful is the Timer one, since it let's me to all sorts of things in an easy way, for instance:

timer:after(2, function() self.dead = true end)

This will kill the current object (self) after 2 seconds. The library also lets me tween stuff really easily:

timer:tween(2, self, {alpha = 0}, 'in-out-cubic', function() self.dead = true end)

This will tween the object's alpha attribute to 0 over 2 seconds using the in-out-cubic tween mode, and then kill the object. This can create the effect of something fading out and disappearing. It can also be used to make things blink when they get hit, for instance:

timer:every(0.05, function() self.visible = not self.visible, 10)

This will change self.visible between true and false every 0.05 seconds for a number of 10 times. Which means that this creates a blinking effect for 0.5 seconds. Anyway, as you can see, the uses are limitless and made possible because of the way Lua works with its anonymous functions.

The other libraries have similarly trivial APIs that are very powerful and useful. The camera one is the only one that is a bit too low level and that can be improved substantially in the future. The idea with it was to create something that enabled what can be seen in this video:

But in the end I created a sort of middle layer between having the very basics of a camera module and what can be seen in the video. Because I wanted the library to be used by people using LÖVE, I had to make less assumptions about what kinds of attributes the objects that are being followed and/or approached would have, which means that it's impossible to add some of the features seen in the video. In the future when I make my own engine I can assume anything I want about my game objects, which means that I can implement a proper version of this library that achieves everything that can be seen in that video!

Rooms and Areas

A way to manage objects that really worked out for me is the notion of Rooms and Areas. Rooms are equivalent to a "level" or a "scene", they're where everything happens and you can have multiple of them and switch between them. An Area is a game object manager type of object that can go inside Rooms. These Area objects are also called "spaces" by some people. The way an Area and a Room works is something like this (the real version of those classes would have way more functions, like the Area would have addGameObject, queryGameObjectsInCircle, etc):

Area = Class()

function Area:new()
    self.game_objects = {}
end

function Area:update(dt)
    -- update all game objects
end
Room = Class()

function Room:new()
    self.area = Area()
end

function Room:update(dt)
    self.area:update(dt)
end

The benefits of having this difference between both ideas is that rooms don't necessarily need to have Areas, which means that the way in which objects are managed inside a Room isn't fixed. For one Room I can just decide that I want the objects in it to be handled in some other way and then I'm able to just code that directly instead of trying to bend my Area code to do this new thing instead.

One of the disadvantages of doing this, however, is that it's easy to mix local object management logic with the object management logic of an Area, having a room that has both. This gets really confusing really easily and was a big source of bugs in the development of BYTEPATH. So in the future one thing that I'll try to enforce is that a Room should either use an Area or it's own object management routines, but never both at the same time.

snake_case over camelCase

Right now I use snake_case for variable names and camelCase for function names. In the future I'll change to snake_case everywhere except for class/module names, which will still be in CamelCase. The reason for this is very simple: it's very hard to read long function names in camelCase. The possible confusion between variables and function names if everything is in snake_case is generally not a problem because of the context around the name, so it's all okay 🤗


Engine

The main reason why I'll write my own engine this year after finishing this game has to do with control. LÖVE is a great framework but it's rough around the edges when it comes to releasing a game. Things like Steamworks support, HTTPS support, trying out a different physics engine like Chipmunk, C/C++ library usage, packaging your game up for distribution on Linux, and a bunch of other stuff that I'll mention soon are all harder than they should be.

They're certainly not impossible, but they're possible if I'm willing to go down to the C/C++ level of things and do some work there. I'm a C programmer by default so I have no issue with doing that, but the reason why I started using the framework initially was to just use Lua and not have to worry about that, so it kind of defeats the purpose. And so if I'm going to have to handle things at a lower level no matter what then I'd rather own that part of the codebase by building it myself.

However, I'd like to make a more general point about engines here and for that I have to switch to trashing on Unity instead of LÖVE. There's a game that I really enjoyed playing for a while called Throne of Lies:

It's a mafia clone and it had (probably still has) a very healthy and good community. I found out about it from a streamer I watch and so there were a lot of like-minded people in the game which was very fun. Overall I had a super positive experience with it. So one day I found a postmortem of the game on /r/gamedev by one of the developers of the game. This guy happened to be one of the programmers and he wrote one comment that caught my attention:

So here is a guy who made a game I really had a good time with saying all these horrible things about Unity, how it's all very unstable, and how they chase new features and never finish anything properly, and on and on. I was kinda surprised that someone disliked Unity so much to write this so I decided to soft stalk him a little to see what else he said about Unity:

And then he also said this: 😱

And also this: 😱 😱

And you know, I've never used Unity so I don't know if all he's saying is true, but he has finished a game with it and I don't see why he would lie. His argument on all those posts is pretty much the same too: Unity focuses on adding new features instead of polishing existing ones and Unity has trouble with keeping a number of their existing features stable across versions.

Out of all those posts the most compelling argument that he makes, in my view, which is also an argument that applies to other engines and not just Unity, is that the developers of the engine don't make games themselves with it. One big thing that I've noticed with LÖVE at least, is that a lot of the problems it has would be solved if the developers were actively making indie games with it. Because if they were doing that those problems would be obvious to them and so they'd be super high priority and would be fixed very quickly. xblade724 has found the same is true about Unity. And many other people I know have found this to be true of other engines they use as well.

There are very very few frameworks/engines out there where its developers actively make games with it. The ones I can think off the top of my head are: Unreal, since Epic has tons of super successful games they make with their engine, the latest one being Fortnite; Monogame, since the main developer seems to port games to various platforms using it; and GameMaker, since YoYo Games seems to make their own mobile games with their engine.

Out of all the other engines I know this doesn't hold, which means that all those other engines out there have very obvious problems and hurdles to actually finishing games with it that are unlikely to get fixed at all. Because there's no incentive, right? If some kinds of problems only affect 5% of your users because they only happen at the end of the development cycle of a game, why would you fix them at all unless you're making games with your own engine and having to go through those problems yourself?

And all this means is that if I'm interested in making games in a robust and proven way and not encountering tons of unexpected problems when I get closer to finishing my game, I don't want to use an engine that will make my life harder, and so I don't want to use any engine other than one of those 3 above. For my particular case Unreal doesn't work because I'm mainly interested in 2D games and Unreal is too much for those, Monogame doesn't work because I hate C#, and GameMaker doesn't work because I don't like the idea visual coding or interface-based coding. Which leaves me with the option to make my own. 🙂

So with all that reasoning out of the way, let's get down to the specifics:

C/Lua interfacing and memory

C/Lua binding can happen in 2 fundamental ways (at least from my limited experience with it so far): with full userdata and with light userdata. With full userdata the approach is that whenever Lua code asks for something to be allocated in C, like say, a physics object, you create a reference to that object in Lua and use that instead. In this way you can create a full object with metatables and all sorts of goodies that represents the C object faithfully. One of the problems with this is that it creates a lot of garbage on the Lua side of things, and as I mentioned in an earlier section, I want to avoid memory allocations as much as possible, or at least I want to have full control over when it happens.

So the approach that seems to make the most sense here is to use light userdata. Light userdata is just a a normal C pointer. This means that we can't have much information about the object we're pointing to, but it's the option that provides the most control over things on the Lua side of things. In this setup creating and destroying objects has to be done manually and things don't magically get collected, which is exactly what I need. There's a very nice talk on this entire subject by the guy who made the Stingray Engine here:

You can also see how some of what he's talking about happens in his engine in the documentation.

The point of writing my own engine when it comes to this is that I get full control over how C/Lua binding happens and what the tradeoffs that have to happen will be. If I'm using someone else's Lua engine they've made those choices for me and they might have been choices that I'm not entirely happy with, such as is the case with LÖVE. So this is the main way in which I can gain more control over memory and have more performant and robust games.

External integrations

Things like Steamworks, Twitch, Discord and other sites all have their own APIs that you need to integrate against if you want to do cool things and not owning the C/C++ codebase makes this harder. It's not impossible to do the work necessary to get these integrations working with LÖVE, for instance, but it's more work than I'm willing to do and if I'll have to do it anyway I might as well just do it for my own engine.

If you're using engines like Unity or Unreal which are extremely popular and which already have integrations for most of these services done by other people then this isn't an issue, but if you're using any engine that has a smaller userbase than those two you're probably either having to integrate these things on your own, or you're using someone's half implemented code that barely works, which isn't a good solution.

So again, owning the C/C++ part of the codebase makes these sorts of integrations a lot easier to deal with, since you can just implement what you'll need and it will work for sure.

Other platforms

This is one of the few advantages that I see in engines like Unity or Unreal over writing my own: they support every platform available. I don't know if that support is solid at all, but the fact that they do it is pretty impressive and it's something that it's hard for someone to do alone. While I'm not a super-nerdlord who lives and breathes assembly, I don't think I would have tons of issues with porting my future engine to consoles or other platforms, but it's not something that I can recommend to just anyone because it's likely a lot of busywork.

One of the platforms that I really want to support from the get-go is the web. I've played a game once called Bloons TD5 in the browser and after playing for a while the game asked me to go buy it on Steam for $10. And I did. So I think supporting a version of your game on the browser with less features and then asking people to get it on Steam is a very good strategy that I want to be able to do as well. And from my preliminary investigations into what is needed to make an engine in C, SDL seems to work fine with Emscripten and I can draw something on the screen of a browser, so this seems good.

Replays, trailers

Making the trailer for this game was a very very bad experience. I didn't like it at all. I'm very good at thinking up movies/trailers/stories in my head (for some reason I do it all the time when listening to music) and so I had a very good idea of the trailer I wanted to make for this game. But that was totally not the result I got because I didn't know how to use the tools I needed to use enough (like the video editor) and I didn't have as much control over footage capturing as I wanted to have.

One of the things I'm hoping to do with my engine is to have a replay system and a trailer system built into it. The replay system will enable me to collect gameplay clips more easily because I won't need to use an external program to record gameplay. I think I can also make it so that gameplay is recorded at all times during development, and then I can programmatically go over all replays and look for certain events or sequence of events to use in the trailer. If I manage to do this then the process of getting the footage I want will become a lot easier.

Additionally, once I have this replay system I can also have a trailer system built into the engine that will enable me to piece different parts of different replays together. I don't see any technical reason on why this shouldn't work so it really is just a matter of building it.

And the reason why I need to write an engine is that to build a replay system like I 100% need to work at the byte level, since much of making replays work in a manageable way is making them take less space. I already built a replay system in Lua in this article #8, but a mere 10 seconds of recording resulted in a 10MB file. There are more optimizations available that I could have done but at the end of the day Lua has its limits and its much easier to optimize stuff like this in C.

Design coherence

Finally, the last reason why I want to build my own engine is design coherence. One of the things that I love/hate about LÖVE, Lua, and I'd also say that of the Linux philosophy as well is how decentralized it is. With Lua and LÖVE there are no real standard way of doing things, people just do them in the way that they see fit, and if you want to build a library that other people want to use you can't assume much. All the libraries I built for LÖVE (you can see this in my repository) follow this idea because otherwise no one would use them.

The benefits of this decentralization is that I can easily take someone's library, use it in my game, change it to suit my needs and generally it will work out. The drawbacks of this decentralization is that the amount of time that each library can save me is lower compared to if things were more centralized around some set of standards. I already mentioned on example of this with my camera library in a previous section. This goes against just getting things done in a timely manner.

So one of the things that I'm really looking forward to with my engine is just being able to centralize everything exactly around how I want it to be and being able to make lots of assumptions about things, which will be refreshing change of pace (and hopefully a big boost in productivity)!

Anyway, I rambled a lot in this article but I hope it was useful. Also, buy my game (click below to buy 🙂)

vrld commented 6 years ago

Good writeup. The first two soft lessons feel especially relevant. As for the first hard lesson: maybe half of the nil-issues can be solved by using strict.lua. It won't catch locals or undefined fields in tables, but spelling mistakes will be easily detected. The lua-users wiki also contains a page on that topic.

idchlife commented 6 years ago

Dear god with generalization of everything and making you know even sub-libraries for your game before you even have working prototype - it's so me. I worked very hard for 3 months for game (browser based), backend and frontend. Everything looked like polished commercial company game. Separate parts, modules, comments, tests etc. Well, I really tried hard, very hard to maintain motivation but at the end I burnt out. It happens. It happens to everyone. In the future games I tend to work with smaller ideas and create tinier and simpler codebases for first prototype to see the light. I hope it will help to finish them.

Looking forward to new posts and engine news from you! ❤️

Nikaoto commented 6 years ago

Good read, always great to read the experiences of fellow lövers :^)

dbousamra commented 6 years ago

Great read :) I also struggled with Unity. I am using libgdx now, and having a better time. I am using your BYTEPATH series as a loose guide to learning gamedev (just implemented in libgdx now)

ffiarpg commented 6 years ago

I didn't see the godot engine mentioned. It might be worth considering before writing your own.

tonetheman commented 6 years ago

Excellent information and good writeup!

joonazan commented 6 years ago

The part about screen capture in-engine makes no sense. You can use NVIDIA Shadowplay to capture all your gameplay. You might have to buy a hard drive for the footage if you don't have a hundred gigs to spare, but that should not matter. A replay system which records game state / player inputs would be a slight improvement over screen capture, but then you run into problems like replay compatibility between game versions.

And please do not build a "trailer system" into your engine. It is called a video editor.

The Jellyfish example is no longer puzzling once you realize that modeling domain objects with programming language objects is a trap. A server is a black box that sends and receives messages, just like an object. But let's say you have an object representing a carrot, and I break it in half. Should the "break" method return one half and transform the object into the other?

If you think about just the data, it is pretty obvious to represent the jellyfish as: jellyfish = { head = {...}, tentacles = [{..}, {..}, {..}] }. Then you can make a function that draws the jellyfish and you can separate drawing the tentacles into their own function if you wish.

You do not give a concrete example about bad abstraction here, but you did give one in chapter 10. The duplicated constructor problem illustrated there could be solved by writing functions that return data instead of thinking in terms of constructors. For example, you can call the function for spawning one kind of enemy and modify the output so that it suits your needs. It's even faster to type out than copying! And Lua should be excellent for doing prototype-based programming.

You can prevent null dereferences by either having a good type system or doing a lot of manual work. For example Haskell has Maybe a = Just a | Nothing for things that may or may not exist. You could do the same in Lua, and it would prevent errors, but it would be frustrating.

It turns out that even with a modest type system, you can apply the Maybe treatment everywhere. Elm, which is basically a subset of Haskell doesn't have runtime errors apart from infinite loops.

a327ex commented 6 years ago

@vrld Thanks for the tip!

@idchlife @Nikaoto @dbousamra @tonetheman Thanks!

@ffiarpg Godot wasn't mentioned because as far as I've researched its developers aren't actively making their own games with it. The main developer seems to have a lot of games in the past, which is good, but if no games are being made currently then it likely suffers from the problems I mentioned in the post in regards to this.

@joonazan

A replay system which records game state / player inputs would be a slight improvement over screen capture, but then you run into problems like replay compatibility between game versions.

And please do not build a "trailer system" into your engine. It is called a video editor.`

My replay system will not be input based and will just record everything, so compatibility problems shouldn't be an issue. And I very much disliked using video editors and I'd rather code my trailers, since it gives me more control. I'll build the trailer system in my engine for this reason.

If you think about just the data, it is pretty obvious to represent the jellyfish as: jellyfish = { head = {...}, tentacles = [{..}, {..}, {..}] }. Then you can make a function that draws the jellyfish and you can separate drawing the tentacles into their own function if you wish.

I was actually thinking about this but I talked myself out of it since the objects from the physics engine are big blobs of code and data, which I thought would make it harder for me to just have everything as data and then a bunch of functions that operate on it like you mentioned. I haven't thought much about if this issue is real or imagined, so this is a way of doing things that I'll definitely explore in the future.

You do not give a concrete example about bad abstraction here, but you did give one in chapter 10.

I don't think you understood my argument regarding abstractions here. The argument applies to every type of abstraction there is, from functions to ECS. The solution you just mentioned falls into the left side of the AB image, the one where AB box is inside a transparent box.

For example Haskell has Maybe a = Just a | Nothing for things that may or may not exist. You could do the same in Lua, and it would prevent errors, but it would be frustrating.

Don't you need to handle the different cases (if the thing is there or if nothing is there) still in your code? If you do then this isn't really that useful since I can just do that on my own. The issue is a way to fix this problem that isn't overly defensive.

lisardo commented 6 years ago

@SSYGEN Some type systems have nice ways to handle null objects, e. g.: https://kotlinlang.org/docs/reference/null-safety.html#safe-calls

Nicktho commented 6 years ago

Thanks again @SSYGEN, this repo is becoming a gem of information. I originally started programming with game development in my early teens, although now I work exclusively in the web environment. I was looking around for node/javascript based engines I could use to follow along to your bytepath guide but wasn't satisfied with any of the offerings out there at the moment. I'm so comfortable in JS that i'm not too interested in switching languages just for testing the game development waters again, but nonetheless reading through your content has really benefitted me day to day.

a327ex commented 6 years ago

@lisardo Ah, thanks. That syntax is really nice and would be way less intrusive to use. I think that would solve my problems with this almost completely.

@Nicktho Thanks!

chartinger commented 6 years ago

If you have the time i would recommend writing your own engine anyway - even if you do not use it, you learn a lot of things and get a better understanding why and how existing engines are the way they are and what they tried to solve.

I still prefer libGDX for my projects (as a hobby, game jams) with a custom mini "framework / engine" on top.

vadi2 commented 6 years ago

Did you use Luacheck during development? It is a godsend for proofing errors.

ddouglas87 commented 6 years ago

@SSYGEN Are you making an argument for procedural programming over OOP for small games?

Over generalization is a real issue, and over abstraction can make reading code difficult. Also, you probably already know this, but procedural programming is much better as a 'get up and go' paradigm. However, I wonder if this difficulty with generalization comes from a lack of bottom up programming with passive abstractions.

Rigid abstraction can be a common difficulty in top down programming (usually leaving code bases decoupled from their original architected abstraction), where bottom up programming tends to not suffer from this anywhere as much. For example, instead of making objects from the get go, making a handful of units (usually functions), and then down the road bundling them together into a common abstraction can work out better. This way an abstraction is only created when it is beneficial or necessary, building up, not down.

mratsim commented 6 years ago

Great write-up !

Since you're familiar with C, you might want to look at the Nim language. It has the syntax of Python, the speed of C and the metaprogramming power of Lisp. Since it compiles to C, C++ or Javascript you can call any C, C++ or JS library in your code.

It's very easy to make a template or a macro that copy-paste part of AB, ABC, ABD, AB* without needing OOP inheritance, generalization, etc.

It is also quite popular among game programmers because you can completely control memory allocation, the GC is optional you can choose to deactivate it globally or for certain objects or choose your implementation or even the max pause allowed.

Here are some game related projects in Nim:

Edit: forgot about

joonazan commented 6 years ago

@SSYGEN I share your hatred of video editors. AE / Premiere are ok, but expensive. Blender is one option that I'll try the next time I want to edit.

FP error handling

Don't you need to handle the different cases (if the thing is there or if nothing is there) still in your code? If you do then this isn't really that useful since I can just do that on my own. The issue is a way to fix this problem that isn't overly defensive.

There are three kinds of error handling: unhandled errors, badly handled errors and gracefully handled errors. Badly handled errors are arguably worse than a crash, because you don't necessarily notice them.

The functional programming approach is to handle all errors. Madness, right? The upside is that if some piece of code doesn't return an error type, it will do what it is supposed to every time. That is huge for maintainability, as it raises the likelihood that you'll never have to touch a piece of code again.

Handling all the cases by hand would not give you the same guarantee about not crashing. You also usually never unwrap Maybes. Instead, you propagate the error or give a default value. Some examples:

getName : Form -> Maybe Name
getName form =
  Maybe.map2 Name (getField "firstname" form) (getField "lastName" form)
Dict.get n neighbors
|> Maybe.withDefault Set.empty

As you can see, Elm is slightly verbose here. In Haskell or Idris you can avoid the Maybe.map -style functions, but that's not relevant enough to further describe.

From bad to graceful

Actually, there are two kinds of errors. Those that aren't exceptional and those that should never happen. When a player tries to equip an item, but his stats are too low, the error must be handled gracefully. When you index out of bounds, there is no way to recover.

Or so I thought before I tried handling those. I just put some minimal valid values as default values. Now, when a user encountered a bug, things related to the bug usually disintegrated, but that's ok. During development, crashes can be good. User-facing crashes are never good.

That's the baseline. And I found that in many cases I could do way better with very little effort. The next step towards graceful would be to cancel whatever action caused the bug.

I have a lot of time to spend on the functions that can fail, because often when I see a function that can fail I go: "That function should never fail. Can I alter the surroundings so that it doesn't?" and it turns out I can. Doing that is worthwhile, because making one thing never fail usually gets rid of error handling in multiple places.

The abstraction problem

The solution you just mentioned falls into the left side of the AB image, the one where AB box is inside a transparent box.

At the core of this problem is the fact that each time we add something new or we change how something behaves, we now have to do these things AGAINST the existing structures. To add or change something we have to always "does it go in AB or AB*?"

I suspect that your problem is that AB and AB* are objects that do not have clear responsibilities. This is pretty much true for any game object, as they mix spawning, behaviour, event handling and drawing. I'm not saying that making that kind of object is wrong, but depending on it with some other code definitely is.

If AB was a function called "uniformRandomLocation", it would be pretty clear that you are not supposed to change it unless it has a bug or your coordinate system somehow changes.

in the first option we end up with an AB that is very complex and has all sorts of switches and flags for its different behaviors

This is almost always bad. Some even teach that all booleans are bad because of this.

I'd say instead of adding flags, split it into pieces or don't use it if it doesn't help as-is. In the example, build a "spawning kit" of useful functions. Similarly, you want to have some common primitives for drawing, so you can for example adjust the line drawing style to taste.

ThaisRobba commented 6 years ago

@SSYGEN really nice write up - and congratulations on the game release!

You might want to take a look at Amulet (http://www.amulet.xyz) - it is a small, pragmatic game engine with built-in support for exporting to multiple targets (including HTML) and the creator, Ian McLarty, develops his games with it.

It is very "batteries included" and warrants a look - even if only for inspiration :-)

jaythomas commented 6 years ago

Premature Generalization is a case of making your code DRY before you finish fleshing out the pieces and it's not an issue specific to just game development. Donald Knuth said premature optimization is the root of all evil.

mattiasgustavsson commented 6 years ago

I agree with a lot of what you're saying. I also think that having your own engine can make the process more enjoyable too - no fighting a third party tool you don't know or control. If you've made your own engine, you know exactly what everything is.

The "timer" system you describe - I have one very much like it (but in C++ so using lambdas and a bit of templates), and it really is a very very powerful tool - makes many things so much easier.

When I wrote my engine, I decided to build many of the low level systems in stand-alone C or C++ modules, and then just have a high level layer to tie everything together. My intention was that the low level modules would be useful for others making their own engine - maybe as a starting point or intermediate solution.

a327ex commented 6 years ago

@vadi2 I didn't, but thanks for the tip, I'll check it out.

@ddouglas87 I've been thinking about that a lot and I wanted to try coding that way more but I seem to always default to objects when it comes to entities because it seems to make more sense.

@mratsim Thanks for the post. I've checked Nim multiple times before and I'm really interested in it but it's probably something I'll leave for the future to seriously consider. I don't like using tech that hasn't been around for a while at least with a number of proven projects.

@mattiasgustavsson I've seen your single file libs repo before. I really like that way of doing things in the C/C++ gamedev community and I'll also use some of those libraries in my engine when possible. Thanks! ^^

ddouglas87 commented 6 years ago

@SSYGEN You may or may not already know this, but if you abstract procedural paradigm code as one of the later steps (maybe even last step) then you will see and learn new ways of creating abstractions that will not step on your toes.

Refactoring is a game with a different goal.

And inheritance: it's not taught properly in any books I've read. Inheritance is about creating subtypes. It's type theory: you're creating a type of a thing, instead of a thing. It's not about reducing code duplication but instead creating new types of things, or using someone else's types.

So for that, I gotta ask, "Have you tried making any types before, instead of just objects?"

a327ex commented 6 years ago

@ddouglas87

So for that, I gotta ask, "Have you tried making any types before, instead of just objects?"

While this seems interesting philosophically I don't understand how it applies to my problems. Could you expand more?

You may or may not already know this, but if you abstract procedural paradigm code as one of the later steps (maybe even last step) then you will see and learn new ways of creating abstractions that will not step on your toes.

Could you give me examples on this too? I don't understand what you mean exactly.

ddouglas87 commented 6 years ago

@SSYGEN

While this seems interesting philosophically I don't understand how it applies to my problems. Could you expand more?

This comes down to the domain of architecture. If you're got an hour checkout this fluff piece that explains the difficulty others face regarding abstraction and generalization: https://www.youtube.com/watch?v=GAFZcYlO5S0 (And if you don't care to watch it, I don't blame you.)

A problem architects have is they draw up the "perfect code base" (or perfect program). Then they give it to a team of engineers (or one engineer, I've been that one), and the engineers look it over and start writing the code. Big deal right?

Well, as time goes on, the code base drifts from the original architected abstraction. This is because when you're down in the trenches you realize there is a better or easier or quicker way to do a thing, so you go do it. This change diverges from the original grand plan.

In many ways this problem is similar to yours, except from a different vantage point. The problem: Creating abstractions before creating the code.

When possible, the code is created first, and then it is abstracted afterwords. This abstraction then seems like an unnecessary step, because then why do it when everything works right? For that, there are multiple good answers to this question. I could rant at you for hours about it, so I'm going to omit all of the reasons but one: If you abstract at the end and take that extra time, every time you do, you will become better at writing and reading code, especially reading and writing libraries and frameworks.

Could you give me examples on this too? I don't understand what you mean exactly.

Sure. Lets say you're in finance (I know, I know, but it's an easy example), and you need to work with currency. You can't use a IEEE floating point type, because under the hood it stores the value in scientific notation, which does have enough rounding errors to make it not a valid use case, especially if this is for a bank or the stock market or similar.

The first solution is using a bignum math library. These have perfect accuracy and have what you call infinite precision. So instead of double current_usd; you've got bignum current_usd;. It's a drop in replacement, so you don't have to change any of your code (hopefully), and it just works. Yay!

But then you notice bignum math is slow. You're doing crazy stock market stuff and it needs to be FAST. So then you end up making a decimal type, which stores numbers with decimals in them like 1.23 in binary under the hood. Now replacing bignum current_usd; with decimal current_usd; and wa-la! Little to no changes to the code base and it is 4 times faster. Nice!

You'll notice here bignum and decimal (as well as float, double, int, ...) are not objects, they're types. Now, you could argue a type is an object, but you could argue everything is an object, so that side steps the point. The point is the way of thinking about it. Are you creating a type of a thing, or are you creating a thing? A thing is just a function / functions with its own state. Basically, globals with functions, then given a class name.

You're already writing objects, even if you're using globals and functions. It's the same thing. The only difference is an object increases code readability, because once you give it a name, you have a single word (and hopefully not a ObjectOfFourWords) that represents it in your head. All of a sudden there is less for you to think about, because the conscious mind relies on words. You're probably already boiling things down in your head to a finite number of words, why not document it in code by turning it into an object? It's a little bit of work to do it at the end but the benefits are enormous.

The big problem is, you don't know what it is going to be called until it is done. You bump into the architecture problem in that video of premature generalization, by trying to name a thing first, not after.

This is why white board problems during interviews are so hard. When you're writing on a white board you usually want to come up with the name first. On an IDE it is easy to change the name later by refactoring.

edit: Oh man, I totally went on a tangent. lol woops. Back to types: Inheritance isn't like object abstraction. If you have 10+ objects that are all similar, the next level of abstraction is creating a module (the name depends on the programming language). There shouldn't be a reason to do this, unless you've got a million similar objects, which you don't, so I wouldn't even consider looking into it. But if you did, you'd still want to do it at the end, after all these objects are already made and solidified.

A type is the type of abstraction for inheritance. Without understanding how to make types and really diving into it, inheritance is always going to come off like a piece of shit. So, imho you should seriously consider creating a type, even as a side project just to explore that process. Once the mentality solidifies, then try creating a 'subtype' or an 'abstract type' (same thing) which is inheritance. This will take some play too. And no, I'm not saying, this is a good idea for your current code base, but instead, once you bump into inheritance in a framework you want to use, or you need to write a framework, or something similar, you now have a path of understanding that will let you get to where you want to go. When the time comes then hopefully it will not be so painful to explore inheritance.

I wouldn't blindly try to create an object, nor a package, nor do inheritance, without conceptually understanding the mental prerequisites behind it away from the act of doing so.

joakim-noah commented 6 years ago

Don't just stick with C/C++, compare newer languages like the D programming language, someone mentioned Nim earlier. D even has some Lua wrappers you can use.

gotnull commented 6 years ago

@ddouglas87 You must be fun at parties.

ddouglas87 commented 6 years ago

@gotnull Yah, I was rambling a bit much. Passion will do that. I hope I haven't offended.

dbriemann commented 6 years ago

Great read. I am currently undecided between using a framework/engine/library or writing my own. I have worked with Love2d and encountered similar problems as you: building and deploying to different platforms, adding in-app purchase or ads to e.g. an android game etc.

Since you seem to be a C coder have you had a look at raylib(C library, also has Lua bindings)?

You have complained about the myriad of ways that Love/Lua allow you to structure code / write the game. Have you thought about using Go? It enforces very clear code. There is also the ebiten game framework which is very promising but so far does not support custom shaders (which is holding me back). The developer also has created and released at least two mobile games with it AFAIK.

paxer commented 6 years ago

@SSYGEN what’s your thoughts about Defold https://www.defold.com ? It is Lua, King (the engine developer) use it for their own games and they have a really nice core concept of “message passing” https://www.defold.com/manuals/message-passing/ between objects similar to Erlang which eliminate many null issue cases and decouple objects from each other.

flamendless commented 6 years ago

@paxer alot of devs dont want to use it since it's not open source, and it's made by king (which is like people not liking windows because of, well, Microsoft). Another most hated feature of it is that you can't work on it if youre offline

paxer commented 6 years ago

@flamendless with Defold, you have to be online when you create a new project, but after that you can work offline. They are planning to remove this requirement in the future release. Actually in the new editor you can work offline, but with the 1.x it is still an issue

a327ex commented 6 years ago

@paxer I've looked into it and I really like a lot of what they're doing, but not having the code is a deal breaker unfortunately. I'm going to steal a few of their ideas on how to do certain things because they're really good, however.

As for message passing I don't think that's a good idea for general gameplay coding. It just seems very bureaucratic and while it would help to solve the nil issues I don't think I wanna solve them in a way that makes coding the game more painful than it needs to be.

flamendless commented 6 years ago

@paxer really? Then that's awesome!

What i like about defold is that it's very easy to export for android, also, a lot of useful plugins!

flamendless commented 6 years ago

@paxer actually, can you give me the link about their plan to remove the online requirement for creating a project?

paxer commented 6 years ago

@flamendless checkout this thread https://forum.defold.com/t/how-can-i-work-offline-as-defold-new-project-has-to-be-created-online/1937/9

a327ex commented 6 years ago

An example of the kind of code that I'm talking about in the "Soft Lessons" part of the article https://github.com/NoelFB/Celeste/blob/master/Source/Player.cs

volumia commented 6 years ago

Sweet article!

I still do disagree with generalization though.
Although I do agree that organically coding is better for early projects or small teams, I still see the need for reusuablity.

I feel that the biggest hurdle is the jump in paradigm - ECS, for example is notoriously inconsistent in implementation and is a lot harder to learn than general OOP.

Once you get past the original hurdle, it's a lot easier than you'd be lead to believe to code, I think.
Something I've been doing in some projects is that the player is just a variable container and all the actual state is in separate state machines (which is pretty common).

Not everything needs to be generalized, though.
Although the Celeste code you linked isn't ideal (far from it), it's still usable and functional, which is a lot more important than clean code, like you said.

Also, side note about Unity from someone who uses it a lot:

Although I would agree that it seems like the developers have a huge disconnect from the community a lot of times, I feel like it's gotten better in the past year.

Lots of not-broken features have been introduced, and open-sourced, like the SRP, Shader Graph, ECS, and Input System for active feedback and contribution, and acquisition of community features to integrate into the full engine, like ProBuilder.

It only appealing to newcomers part is simply not true - people from Nintendo to Blizzard or the developer of Subnautica use it, so that's gotta mean something, right?

Anyway, rant over, awesome article, and good luck on your engine!

ChengCat commented 6 years ago

Thanks for sharing your experiences. I am also in progress of making a switch away from LOVE. I have been introducing too many C library components into my game, and at one point it seems to me that I only need LOVE to receive input events, which could be easily replaced by a Lua binding to GLFW or SDL.

I have read LOVE's source code, and it can be seen as a set of Lua modules written in C++. The whole game is actually driven from Lua side. It's not hard to build your own engine with the help of a set of third-party C libraries and LuaJIT FFI.

I am deciding to take a step further, and try a Lisp language in place of Lua. I am concerned with the status of LuaJIT, and I am feeling that I am more productive with a Lisp language. My biggest concern so far is the GC pause, and I think I can properly evaluate the situation only after making some progress.

meghprkh commented 6 years ago

Just an aside, I just realized Github issues is a really good way to blog, with comments and reactions :sweat_smile: Only lacking thing IMO is analytics

notverymoe commented 6 years ago

Really interesting write-up, thanks for putting it out there! I definitely understand a lot of what you're talking about, particularly regarding overgeneralization.

Anyway, hope your engine development is speedy!

P.S. Never thought about using github for blog entries, might look at doing that. The formatting and reacts are really useful!

kravchik commented 6 years ago

It's not just that the contexts of indies and teams are different. For the team - a complex structured and abstruse-hierarchical code - the same evil as for indies. The fact is that the role of inheritance is hypertrophied in virtually all textbooks, and dislodges the composition and copy-paste. So I'd focus on this priority: first copy-paste, then re-use via function or composition, and only in extreme, reinforced-concrete-proven case - use a hierarchy. I can also say that in fact, there is not much difference between an indie and a team. Indie has sections of code that will be reused many times and will be polished "like by adults" - all sorts of core-components. And the teams have sections of code that need to be "just written" and not rewritten - because tomorrow the client can ask something different. Just the ratio of one and the other in different projects may differ.

Yes, with Unity, sadness comes out. They have already created a gold mine, their own market, from which they collect their rabid interest. So they should focus on this - a stable engine (and not with features). Ie, IMHO, it's a bad practice - being the masters of the market - and trying to compete with sellers (create services, include Assets in the main engine, add features), instead of improving the conditions of the market itself (increasing engine stability, stimulating sellers/buyers of Assets). And they should add more features to the market itself, different licenses, subscriptions - so that users could not only buy one-time but also sponsor, as on Patreon. This I proceed from my experience. So I released the Asset, it is not needed by everyone, but for whom it is needed - it saves hundreds of hours of animators or is generally key for the game. And, it turns out that sales do not cover my work on the asset, but those for whom it is important - are willing to pay for updates / monthly, but can not.

About trailers. I suffered until I understood two things. The first is to find a convenient editor. Set a task - and try to solve in different editors. The majority rejected at the stage of loading, with two or three in principle, something can be done, and the only one you fall in love with, and, finally, you can do business :) (I stopped with kdenlive). The second. Just like you. I listen to a track and imagine what will happen on the screen. And then everything is simple - cut under the track, under the beat, under the change of the melody. By the way, in my engine (in Java) - I made a recording of sound and video to get smooth videos. So I can compose that kind of videos where you can see the smoothness and those 60 frames. In Unity, this could be done only without sound and with a non-real-time rendering.

Apollo144 commented 6 years ago

@SSYGEN take a look at Gideros SDK, it truly is a hidden treasure. It's open source, no marketing blah blah blah, you can join the team if you want. It has:

egordorichev commented 6 years ago

@dbousamra the issue with LibGDX, is sadly, no consoles support what so ever. In anything else, its an awesome engine!

srodrigo commented 6 years ago

Don't just stick with C/C++, compare newer languages like the D programming language, someone mentioned Nim earlier. D even has some Lua wrappers you can use.

What about Rust? I know very little about Nim, but I haven't seen any big projects using it, and no-one uses D (its creator doesn't count). Rust is used for major projects such as Firefox and NPM. Whereas it's still not very used in game dev, I would rather have a look at it instead of marginal languages.

srodrigo commented 6 years ago

I'm going to read the links provided about problems with Unity. I was almost decided to use MonoGame (I don't mind C# :) and actually probably makes you more productive than C++), but the Unity roadmap for next year looks quite good. I'm afraid of hitting roadblocks with Unity though, that's why I'd prefer something less intrusive, and MonoGame is the only serious framework, baked with game hits.

Honestly, even if I would love to build my own tech for the complete control it would give, that would mean not focusing on making games. This is a trap, and you will end up spending all your energy and time on the engine, and probably not ship any games. Also, I don't like C++, and Rust (the only serious alternative, someday) is still too young for game dev. So unfortunately (or not), I'll 99% sure go for MonoGame.

JoaoBaptMG commented 5 years ago

The three soft lessons were like a blow to my chest. I do not have a lot to add, but it seems like everything I took years to learn standalone and while in university simply lost all their value. I could argue with you as long as I wanted, but I saw these consequences becoming real in one of the projects I am still struggling to continue. Anyway, I will take that to heart when I (try to) make my next games (sadly, that will leave me with not a lot of "decent" code I could show to business to hire me).

suaviant commented 5 years ago

I stumbled upon this just now. Great write-up. Thanks for this. I agree with a lot of it and you have said much that goes unsaid (so far as I know). As a game dev, getting it done and done well is the primary concern.

And yes, I found a problem with a Unity module and took it to the forums and the resulting exchange was downright meme-able. In the end, no solution.

I will be recommending to my company we move on from Unity one or two projects down the line (we do small scale) and onto something open source. Personally, I never want to touch it again.

anthonyk1225 commented 4 years ago

Since reading this last week, I've thought of what you wrote in the Premature Generalization section at least 10 times. I feel like a weight has been lifted off my shoulders. Being DRY is good, but getting sh*t done is better. Thanks for the thought-provoking post.