search

adventuregamestudio / ags

AGS editor and engine source code
Other
708 stars 159 forks source link

AGS4 and the death of legacy #403

Closed AlanDrake closed 4 years ago

AlanDrake commented 7 years ago

So, to make coding AGS less aggravating, I'll be starting a branch whose purpose is to wipe out legacy stuff. Whether or not another engine will steal its throne in the far future, this can be done now. Deleting all that stuff is certainly more pleasing than adding features wrestling with all these chains littering the source. And should make things easier to understand which in turns could also help the other AGS-inspired engines.

So far I'm experimenting and taking notes, since my knowledge of the inner workings is not as advanced as Crimson's.

I can say though that I've succesfully blasted away those pesky pixel_size/multiply_up/divide_down conversions, which had almost 400 occurences all around, and after compiling it all worked the first try. I feel greatly encouraged.

I'll create a proper branch on my repo as soon as the VS2015 branch is officially merged with master.

Things to cut away:

fdslk1 commented 7 years ago

I have found encloseJS that seems to compile to machine code. And then I also found that Emscripten(LLVM to Javascript) can convert C++ Projects to Javascript. Of course, as with anything, who knows how effective it is and how easy it is to use.

I tried to use Emscripten in personal projects is kind of a mess, but I'm not a pro coder so I'm not sure how good I am. Still a lot of game engines are using it to port to HTML5. Unity and UE4 use it.

The good thing about the builds made with Emscripten are how optimized they work. If anyone is willing to give it a shot or check it out. I think the main issue of people wanting to port and distribute their games with AGS will be solved...

Lets be honest folks, you need a lot more than just windows and linux to actually make a profitable game in the indie market.

If AGS gets a better way to port the game games, that will be the game changer. Or, if you want to avoid doing that, someone that actually knows how to do it, can write a proper documentation or tutorial.

rocifier commented 7 years ago

Performance with JS isn't an issue as long as you have access to WebGL and V8. Then it would be overpowered for any AGS game. 200MB memory is nothing considering most phones coming out now have 4GB RAM built in. I've not seen any on the market which have below 2GB, I went and checked the other day. No point in designing a new engine for legacy devices.

SuperDre commented 7 years ago

Except you really underestimate the ammount of phones that are out there which still haven't got 4GB ram (that may be common with new phones, but certainly not phones/tablets which are older than 6 months). These days a lot of people just buy their phone and use it for several years. Also don't underestimate the ammount of computers/laptops that are still in use.

Maybe you buy a new phone/tablet/computer every year, but most people don't.

To be honest, I don't see the need for another new engine when you've already got enough other engines that are excellent at also doing adventure games (both 2D as 3D).

So it's rather 'no point in designing a new engine' period.

Better spend time at developing a better designer which works with already available engines.

Also JS isn't really a great language, yes it's much used, but that doesn't make it good.

From: rocifier notifications@github.com Sent: Monday, October 2, 2017 09:28 To: adventuregamestudio/ags Cc: Subscribed Subject: Re: [adventuregamestudio/ags] AGS4 and the death of legacy (#403)

Performance with JS isn't an issue as long as you have access to WebGL and V8. Then it would be overpowered for any AGS game. 200MB memory is nothing considering most phones coming out now have 4GB RAM built in. I've not seen any on the market which have below 2GB, I went and checked the other day. No point in designing a new engine for legacy devices.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/adventuregamestudio/ags/issues/403#issuecomment-333459382, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABl1t0JoaTBdXS5qnNhXL1fUGASaGHmrks5soJCFgaJpZM4NFcZG.

AlanDrake commented 7 years ago

This is going off topic. The current AGS is feature complete and has a working IDE. Making another engine or a full rewrite doesn't seem feasible at the moment (we don't have enough experienced developers, and it should probably be talked about somewhere else). Besides there are already tzachs' MonoAGS C# and scotch's JS AGS spinoffs that could use a helping hand.

The question is what do we do with the current codebase ? If it needs to be mantained I still think there's value in cleaning it up. If we stripped the code to the bone, it would be possible to finally organize everything into proper namespaces. From there we might rewrite components and refactor.

I don't know when I'm actively resuming my work on this branch, my coding sprees come and go, but if 1-2 people join in the effort we can slowly grind forward sharing our knowledge and ideas.

We can discuss and coordinate in the AGS discord.

tzachshabtay commented 7 years ago

@SuperDre I think you underestimate the amount of time a rewrite is going to take. I'm working on my c# rewrite for over 2 years now, I think (https://github.com/tzachshabtay/MonoAGS), and realistically, if nobody joins me, I won't have a stable release in another 2 years. Also XAGE (http://clarvalon.blogspot.co.il/) has been in development from at least 2008(!), and still not released. By the time a new engine is released and actually used, low powered devices will probably be a non-issue.

I feel Javascript (especially if it's typed- i.e either TypeScript or Flow) is a very good choice for a rewrite, not because it's a great tech (it's really not, and LOL about it being faster than c++) but because of its very wide cross platform support, and up until recently the only reasonable choice for web. Like CW briefly mentioned, Scotch is already working on a javascript rewrite, and iirc he said he had more than 70% of the APIs already implemented. He also said he'll publish his work on Github at some point, though so far I can't see it. I'll try poking him on Discord.

Edit: oops, missed AlanDrake's post, sorry for the off-topic, so on-topic, I agree that there's value in cleanup, though would probably be more effective to do the cleanup gradually instead of removing everything at once, which kind of leaves the current AGS codebase in limbo.

ghost commented 7 years ago

@AlanDrake

The question is what do we do with the current codebase ? If it needs to be mantained I still think there's value in cleaning it up. If we stripped the code to the bone, it would be possible to finally organize everything into proper namespaces. From there we might rewrite components and refactor.

@tzachshabtay

I agree that there's value in cleanup, though would probably be more effective to do the cleanup gradually instead of removing everything at once, which kind of leaves the current AGS codebase in limbo.

Tzach, honestly, I've been following the advice you gave me about refactoring in "baby steps" for few years, and it caused me lots and lots of pain.

I wanted to mention this before, but probably now it is better time. In my experience I found there are two refactoring cases. First when the code is generally okay but some parts are bad, and you want them better (this is often the case during development of a new code). In that case doing "baby steps" works well. It also works well when you are just getting acquianted with the code, so you just refactor a bit here and a bit there, and that works fine.

But there is a second case, when the code is so bad and entagled so much, that it takes additional effort to keep it working after refactoring even a small bit. This is what happened to me somewhere during refactoring of AGS, and also one of the reasons why I ceased doing big refactoring in the past: I had to spend more time trying to invent the way new refactored code may seamlessly work with the parts that were still unrefactored. I also had to artificially restrict the changes I made, and spend extra time and effort designing "intermediate" states of code (Not to mention I had to redo numerous tests after each such step).

Indeed, doing gradual refactoring will also keep codebase in working state at most times, but this will also increase the total time of work, and risk of fast burnout. Meanwhile, the number of useful changes to the engine will be kept at minimum.

This is why I'd rather propose different approach (used it alot in another project): 1) First, come with the final design, on paper. What parts would you like to see in it? How will these parts connect with each other. 2) Locate where the analogues of your components are located in AGS, just to know where they are and how they are interconnected. 3) Code one full component of the final design, even though as a first prototype. 4) Cut out old component from program; not necessarily deleting it, e.g. just replace function calls with calls to your new component. 5) Make sure new component works new way, then, if wanted, bring back old behavior, with perhaps a "legacy mode" switch (there are different ways to do this codewise). 6) Repeat with next component.

Then again, only to clarify, why I keep suggesting rewrite from scratch: that's a logical development of this refactoring idea: instead of replacing component by component, write engine from barebones, using modern libraries like Allegro5 or SDL2, then put old behavior back. (I never suggested to get rid of existing IDE, by the way)

But yeah, I fully realize that's a big work. So it all depends on how many people are there and how many time they may invest. Oh, and what the final goal is, of course. One must keep it in mind all times... I seem to loose mine over these years...

tzachshabtay commented 7 years ago

All fair points, but I don't think it contradicts my suggestion for doing it gradually, i.e, like you said, one component after another (it's still "baby steps", just the baby is maybe a little bigger). From what I understand, @AlanDrake simply went "Leroy Jenkins!!" and started cutting everything (i.e removed stuff horizontally across the board), which might be fine if there's a quick test & merge back, but if work has halted and nobody else is helping and people might want to merge other stuff into master, it will just get more and more difficult to merge it in. It might be worth it to focus on this and freeze other development until something can be merged back.

AlanDrake commented 7 years ago

Yeah, sorry I lost steam there, but the cutting was proceeding quite well, if someone wants to join in I'll free up some time to merge the latest master changes, make a list of what has been done and still needs to, and move the 4.0 branch to the main repository, so it's more visible.

Just please stop changing the game format, I thought we had agreed that 3.4 only needed band aids to keep it working.

AlanDrake commented 7 years ago

Sorry, I didn't mean to anger you. My memory isn't worth much, but when you said you were planning to wrap up the RC version and be done with it, I'm pretty sure a feature freeze was discussed. Altough, yes, I cannot say that anyone expressly agreed to anything, except that you'd keep a 3.4 branch all for you.

Since I've butchered various places that handle the game project data, but I'd like to keep compatibility with your final release, the less the game format changes the better. I mean, merging is going to kill me, and I know... it's a grave I've dug myself!

Slack was the place to discuss internally for this stuff, but when you left, it kinda died down. I'm hoping to bring wannabe devs on discord, since it's one of the few things beside slack that work under this evil corporate proxy and I can use to be present.

What if someone asks me to add another thing into engine, should I tell them I cannot do that because Alan Drake told me not to?

I'm not sure how to answer this, since you said yourself you were allegedly finishing the release candidate before abandoning the project (a day we all dread of), what are your actual plans ?

ghost commented 7 years ago

NOTE: several unrelated comments removed.

fdslk1 commented 7 years ago

@ivan-mogilko @jdiperla

I know I barely made two comments on this stuff and I know I'm no one to speak, but I think you're taking stuff far too personal and you should both stop typing for now and chill the heck out.

On-topic:

Now: Quick question, have you all considered making a poll about the features you want out and what people want in the engine inside the forums? I think you should ask your user base what they expect and need from the engine, most will say they need better porting (which is the only downside of AGS) others maybe remove legacy and others maybe want nothing new.

Sometimes the best course of action is doing nothing and just gather intel. And this is coming from a long-term AGS user.

AlanDrake commented 7 years ago

I already got the input that matters from chat and discord, I have even taken out 8bit mode from the kill list for the time being. Except few things all should mostly work as it does now, until the cleanup is complete. What comes next should be decided in the future, when and if this part is done.

There's no value in making plans for the users now, it's the developers the current concern. There's not a single one happy of working on the code. Any cool stuff has been postponed, be it because the code kept changing making a merge difficult or that side effects kept appearing out of nowhere, this has discouraged me and many others.

Cleaning the code from the old stuff is the only path left. Coding for AGS must become easier or else devs who can't invest much time will keep running away.

ghost commented 7 years ago

Sorry, but could I delete the unrelated comments posted above from this thread? Since that's me who brought them here...

Only shortly speaking on topic: 1) The most critical issues users were noting over last couple of years, that I remember:

Some of these could be fixed locally, I believe, e.g. sprite manager is mostly a separate component that could've be just rewritten without affecting rest of program. Others, on other hand, may require a big change in code and game format (e.g. room object limits).

2) In regards to how cleaning the code relates to ports: that really depends on what has to be done in order to make the port. I think this could deserve a separate discussion. For example, @sonneveld 's work on porting AGS to SDL2 (which supposedly could make it run better on some systems) was mostly done by writing an intermediate layer "under" the engine, where he substituted Allegro4 functions with calls to SDL2, meaning that the engine code (dirty or not) was largely untouched. If the porting is done by completely rewriting code, like with the javascript port, cleaning the original code may not be required at all, except maybe for studying reasons.

But that's true that you need first to find a developer who wants to do any of this.

I would also like to point to an unfinished rewrite made by ScummVM member made 5+ years ago. It is not 100% compatible, but has much cleaner codebase: https://github.com/fuzzie/scummvm/tree/ags/engines/ags The reasons we did not use it back then are... still confusing.

PS. Sorry again for the drama...

AlanDrake commented 7 years ago

Nevermind, I understand your despair. And yes this issue could use some cleaning, I'll remove some as well.

SuperDre commented 7 years ago

In regards to ports, I would suggest to pickup the rewrite for ScummVM as that is THE most popular adventuregame hub available and it already has many ports.

And no Ivan, you don't have to appologize for the drama, you did one heck of a great job, and that's not easy.

From: Ivan Mogilko notifications@github.com Sent: Tuesday, October 3, 2017 10:13 To: adventuregamestudio/ags Cc: SuperDre; Mention Subject: Re: [adventuregamestudio/ags] AGS4 and the death of legacy (#403)

Sorry, but could I delete the unrelated comments posted above from this thread? Since that's me who brought them here...

Only shortly speaking on topic:

  1. The most critical issues users were noting over last couple of years, that I remember:

    • ports (already mentioned)
    • unicode text support (believed to be not a big issue if you make game on English, but as soon as it is another language, esp. non-latin one, one has to rely on hacks)
    • various annoying engine restrictions: e.g. max number and size of sprites.

  2. In regards to how cleaning the code relates to ports: that really depends on what has to be done in order to make the port. I think this could deserve a separate discussion. For example, @sonneveldhttps://github.com/sonneveld 's work on porting AGS to SDL2 (which supposedly could make it run better on some systems) was mostly done by writing an intermediate layer "under" the engine, where he substituted Allegro4 functions with calls to SDL2, meaning that the engine code (dirty or not) was largely untouched. If the porting is done by completely rewriting code, like with the javascript port, cleaning the original code may not be required at all, except maybe for studying reasons.

I would also like to point to an unfinished rewrite made by ScummVM member made 5+ years ago. It is not 100% compatible, but has much cleaner codebase: https://github.com/fuzzie/scummvm/tree/ags/engines/ags

PS. Sorry again for the drama...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/adventuregamestudio/ags/issues/403#issuecomment-333771487, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABl1t3tkhM6firWS-c0x9LNLzrd_Ambcks5soeyOgaJpZM4NFcZG.

rofl0r commented 7 years ago

as far as @fuzzie's scummvm work is concerned, it is already very usable. one can play most games with it. the major issues it has is that 1) save/restore is not implemented (which makes it a bit hard to debug issues that appear further inside a game...), and 2) the pathfinder algorithm is buggy (it probably needs backports of the fixes that recently went into AGS itself). i contributed to her port in the past, but couldnt get the pathfinding issues fixed back then - that's basically what drove me away.

btw, if someone really does a rewrite of the engine, i think it would be wise to consult experienced team members about new features, so mistakes made in the past won't be repeated - example: the plugin mode for windows DLLs. (shudder)

jdiperla commented 7 years ago

Disclaimer: Just for the record before everyone thinks I am a complete Jerk here... We worked this out behind the scenes and we both apologized for our behavior. At least I think we are good anyway.

Back on track to the legacy stuff... Out of curiousity, how far backwards compatible is the engine able to run older games? And how far back can it import game projects from AGS? Just curious about that.

On Tue, Oct 3, 2017 at 7:16 AM, rofl0r notifications@github.com wrote:

as far as @fuzzie https://github.com/fuzzie's scummvm work is concerned, it is already very usable. one can play most games with it. the major issues it has is that 1) save/restore is not implemented (which makes it a bit hard to debug issues that appear further inside a game...), and 2) the pathfinder algorithm is buggy (it probably needs backports of the fixes that recently went into AGS itself). i contributed to her port in the past, but couldnt get the pathfinding issues fixed backed then - that's basically what drove me away.

btw, if someone really does a rewrite of the engine, i think it would be wise to consult experienced team members about new features, so mistakes made in the past won't be repeated - example: the plugin mode for windows DLLs. (shudder)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/adventuregamestudio/ags/issues/403#issuecomment-333811687, or mute the thread https://github.com/notifications/unsubscribe-auth/AB1wh00MjrV2G3pVVUlonXuG-6EhCvptks5sohd0gaJpZM4NFcZG .

ghost commented 7 years ago

Out of curiousity, how far backwards compatible is the engine able to run older games? And how far back can it import game projects from AGS?

Engine: loads games from AGS 2.5 and higher, but 100% compatibility is not guaranteed (there were no bug reports for a while though).

Editor: v3.4.1 in theory can load everything since AGS 2.5 too (earlier versions could only load 3.* projects). But in practice this has never been thoroughly tested.

jdiperla commented 7 years ago

So if development of going back further was dropped and attention was given to other matters, would it be a big deal to the community? I mean, if memory serves, AGS below 2.5 compiled everything in DOS anyway. It could easily be played on any system at that point using Dosbox on any OS actually, including mobile and some consoles. I mean, if work still continued on supporting older engines, great I suppose. But maybe it shouldn't be a priority? As far as importing projects from 2.5... That is good work! But the same thing goes for that... Not many projects will be around from earlier editions and projects can probably be imported into 2.5 and then imported into 3.4, right? Just a thought. This was probably discussed, but there were 70 pages to sift through and I have read every post, but don't remember this being specifically discussed at length.

SuperDre commented 7 years ago

Well I don't know about the current regular engine, but it would ofcourse be great if it just supported everything.

But I guess that's more the job of ScummVM.. Having to deal with dosbox has always been an annoyance.

From: jdiperla notifications@github.com Sent: Tuesday, October 3, 2017 21:51 To: adventuregamestudio/ags Cc: SuperDre; Mention Subject: Re: [adventuregamestudio/ags] AGS4 and the death of legacy (#403)

So if development of going back further was dropped and attention was given to other matters, would it be a big deal to the community? I mean, if memory serves, AGS below 2.5 compiled everything in DOS anyway. It could easily be played on any system at that point using Dosbox on any OS actually, including mobile and some consoles. I mean, if work still continued on supporting older engines, great I suppose. But maybe it shouldn't be a priority? As far as importing projects from 2.5... That is good work! But the same thing goes for that... Not many projects will be around from earlier editions and projects can probably be imported into 2.5 and then imported into 3.4, right? Just a thought. This was probably discussed, but there were 70 pages to sift through and I have read every post, but don't remember this being specifically discussed at length.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/adventuregamestudio/ags/issues/403#issuecomment-333958117, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABl1t9_3823a08Yk-yklbtWkQHmemReRks5sopBBgaJpZM4NFcZG.

jdiperla commented 7 years ago

Don't get me wrong, I would love that too. Especially if I can get Rob Blanc back to working natively and even on my mobile devices. But with a work around possible, maybe it would ease pressure and give more time to the devs to work on other needed features or fixes to the engine. I mean it's up to them. But just thinking out loud about that. I am not saying that it's off the table completely, but maybe something that is more of a side project or less of a priority.

AlanDrake commented 7 years ago

Okay then, I finished merging master to the AGS4 branch. Later I'll write down what is the current situation and move the branch to the official repository.

I think the only big gotchas so far are that a project must be first upgraded through classic 3.4.1 and that the blank room template is broken (due to its extreme vintage format), I'll replace it with another until AGS can generate it on the spot. (replaced until it can be generated)

EDIT: The AGS4 branch has been moved to the main repository EDIT2: battle plan

morganwillcock commented 6 years ago

In terms of getting off Allegro 4, are you thinking to replace existing library functions with something back-end agnostic or are the way that game events are handled too specific to Allegro 4? I think you'd probably pick up more developers if the whole thing was stripped back to a new event loop and stub functions to support the existing scripting language (would also give the opportunity to evaluate the language at that point). Personally I've only dabbled in C++, but would be willing to contribute if the bar to entry was a bit lower and there was a obvious place/person to go to when guidance was needed.

ghost commented 6 years ago

In terms of getting off Allegro 4, are you thinking to replace existing library functions with something back-end agnostic or are the way that game events are handled too specific to Allegro 4?

Coincidently, I sent a message to AlanDrake earlier today. Recently I was considering trying to approach this problem from another angle: write a simple rendering engine based on Allegro 5, and then copy only game-related logic from AGS on top of it, piece by piece. Naturally, even in case of success there will be compatibility issues for a while until all features are "restored", most notably it will take some effort to bring 8-bit games back to work. But I believe that may be easier, or at least cleaner, than trying to shove another library into current code. Especially considering there is an attempt to clean and restructure it under way.

That's just an experiment, so I cannot give any promises. I am little busy now, and also somewhat discouraged by the state of the project (to which I am perhaps one of the causes). But that does not prevent from giving this a try. If that won't work, I guess that will at least be a practice programming with Allegro 5.

PS. Why Allegro 5? No serious reason really, A5 and SDL2 capabilities look similar at the first glance (maybe A5 is a little more ambitious with number of addons); Allegro 5 has floating-point API for drawing which perhaps gives a potential to support smoother movements in the game world (people were suggesting subpixel rendering years ago when all this AGS upgrade talk started).

morganwillcock commented 6 years ago

I've always like the idea that the core engine be basic with optional features on top, so games declare any optional features they need. This also means it someone wants to reimplement the basic core in some other language, they have a spec to work to and games that don't use extra features are then compatible. So starting from basic engine does sound appealing for someone new, plus if any major changes like independence of engine speed and game speed need to be made, I can imagine this would be problematic to implement as is. I do have a couple of work related projects I'm working on for the moment, but if this approach has interest then I would like to assist once these are finished. Perhaps just a design document in the interim, but in terms of backend library, it seems most options are event driven and so equally different to Allegro 4.

AlanDrake commented 6 years ago

Ok, I couldn't do a clean rebase, but the "ags4" branch has been updated to master. I fixed some whitespaces and reverted one of those useless variable name changes you mentioned. I tested it against my two 3.4.1 test projects and they work as expected, so there good hopes nothing went amiss.

@ivan-mogilko It's time to decide if it's good enough to merge into master for future development. Let me know if you have any doubts.

The current issues are mostly down to data format, as mentioned, a project has first to be upgraded with 3.4.1+ to be opened. Also the current game templates use outdated formats, so they should be upgraded (scripts too). I bundled an upgraded blank room, although I think it's something that should be generated on the spot.

sonneveld commented 6 years ago

Hi guys, I know I'm late to the party, but I'm in the slow and gradual refactor boat. I don't think big rewrites or removing large caches of code is a good idea. It would take too long for a rewrite to be on feature parity, and removing lots of code would require a long time to iron out bugs.

I haven't had a good look at your AGS4 branch, but I really do think it's a shame we're losing backwards compatibility here. Surely we could just keep different implementations (old and new) within the same branch? (i.e, instead of having lots of branching, we just have replaceable modules) Also there's enough major changes here that I suggest renaming it to 'ags-ng' and porting changes piece by piece.

I think writing some more test game data would be helpful here. Just to make it easier to test the behaviour of features (Current and new)

My SDL2 port is working quite well at the moment. That might be a candidate for AGS 3.5.

edit: added further comments about ags4 branch

AlanDrake commented 6 years ago

If anything, removing old chunk of unused and backward compat code is only going to make it easier to iron out bugs.

Besides we're not losing backwards compatibility, it's going to remain on it's own branch. AGS main purpose is to make games, so it's not a bad thing to make a clear cut once a while. Implementing new features while not breaking old behaviours is harder than it should be and discourages further development.

It's also pretty confusing from the user side having all those options and old script commands, how does a newcomer know what's the proper way to code for the last version and what's deprecated ? A newbie might be using a game template with old script APIs, then he goes on the forums and sees different things, confusing options, etc. Experienced AGS users may know every nook and cranny, but we also want new people to join in.

Some game developers might also want to customize the engine to suit particular needs, so it would be nice not to make it harder than it should be.

Have you already tried merging your sdl2-port and ags4 ? What kind of impact would it have ? I hope it wouldn't make it too hard since it's something I'd like to see coming to AGS.

sonneveld commented 6 years ago

quick thoughts:

Leaving legacy code in its own branch will mean it will wither and die. The only alternative to run old games on new hardware is to hope that scummvm supports it.

In regards for ease of use for users, I'm not sure that's an argument for removing code. That's an argument for marking functions as deprecated and allowing running in a forward compatible mode that doesn't expose deprecated functions.

Have you considering modularising the code? Take old legacy code and wrap it in a module, allow switching modules to enable newer behaviour. It could even be a compile time option for developers who want a custom engine.

edit: Having modules would encourage actual rewrites of code as well. It looks like the old code just has all the version checks removed, but it still mimics the original structure of the code. Rewriting individual modules could allow completely different arrangements

I haven't tried merging sdl2-port into ags4 branch but I would prefer to merge it into 3.4 to allow a 3.5 release first.

edit: I know you put a lot of time in this legacy refactoring but I really don't think this is quite the way to go.

AlanDrake commented 6 years ago

But still, you can't easily modularize while old and new are jumbled together. I'm not invested enough to be able to modularize legacy from current, moreover an attempt like that involving a lot of code being moved around, would make it difficult to merge again if there's further development on master in the meantime. Even harder than merging my cleanup branch.

Wouldn't it be better to first clean up to current spec and then reimplement legacy engine stuff and loading routines directly on a completely separate module ?

ghost commented 6 years ago

I wish this cleanup happened years ago when I still had enough enthusiasm to work a lot, and there were more contributors around. Today I have a constant feeling that this project is simply dead, because no one has interest in actually working on the engine itself.

Both paths mentioned here have pros and cons, and I would be glad to give my support to either.... but the problem is, defeating the cons in either of these ways is possible only when there is a dedicated and active team.

The more I think about it, there is no a good way out of this situation. No matter what path is chosen, whether modularizing keeping the old code, or cleaning up, possibly breaking program, then fixing it and trying to get more updates... Who is going to do all the work in any case? Seems like no matter what it will lead to a program that does not evolve and has no proper maintenance.

sonneveld commented 6 years ago

Module development could be merged in piece by piece, all modules wouldn't necessarily need to be complete before being merged into default.

That doesn't mean we can't replace or remove some code. Like the record/playback, it never was finished so we could remove that completely.

ghost commented 6 years ago

Sorry for the above negative comment, but I could not resist saying that, because that's really how I feel right now.

After posting that I realized that probably should post something more positive too, so here it is. Sorry for the long comment below, but please read it all...

@AlanDrake , you are saying that it is hard implementing new features because of legacy code, and while I agree in general sense, it does not mean that it is exactly the code that you want to remove that makes it hard. It may be because the code in AGS is generally bad. It is also may be because of other things in the engine that cannot be removed right away. And removal of the legacy code won't necessarily make it easier. Perhaps there is much more work to be done, refactoring the code, before you could implement your feature, and it may appear that legacy code was the least of your problems?

I'm not invested enough to be able to modularize legacy from current, moreover an attempt like that involving a lot of code being moved around, would make it difficult to merge again if there's further development on master in the meantime.

This is same problem as with any change or refactoring. It is usually solved in following way:

For example, in the past I tried to investigate that resolution/scaling issue you were talking about. From my findings, it seems that AGS is doing conversion between script and game coordinates. The problem is that it does so in erratic and inconsistent way: some of the game variables keep coordinates in "game" values, and others keep them in "script" values.

The solution that comes to mind is following: move and restrict all coordinate conversion to the place where "script" connects with the "game". Luckily, we already have such place, because of refactoring we did years ago. This is the special functions registered in script systems, the ones that look like:

RuntimeScriptValue Sc_DrawingSurface_DrawImage(void *self, const RuntimeScriptValue *params, int32_t param_count)

This is the best place to put all these of conversions. Basically, if a script function receives coordinates in parameters, you convert these parameters into game coord system before passing into actual game function. And if a script function returns coordinate, you convert it back after receiving it from game function. Changing weird names from multiply_up_coordinate to e.g. convert_script_to_game_coord would also improve things, making it easier to understand what is going on.

I am giving this example just to point out that such things are doable.

Wouldn't it be better to first clean up to current spec and then reimplement legacy engine stuff and loading routines directly on a completely separate module ?

Take loading routines for instance, they are already modularized with legacy stuff separated (except for loading rooms, I think, but that's a minor matter compared to savegames and game loading imho).

Now, finally about the main thing, that I was going to say...

@AlanDrake , you idea was to make the code better and suitable for further development. And your idea was to simpy remove all the legacy functionality (or rather the obvious part of it) at once.

So, as you see in my previous comment, my biggest fear is that the program will be left in an uncertain state with little people to actually maintain or develop it.

A year ago, when you just started this discussion, I was not in a very good state to participate or help, mostly because there was too much on my hands (version 3.4.0 had to be finished, and also we were developing 3.4.1 at the same time), and frankly my emotional condition was not good either.

What has changed since is, that the actual development has currently slowed down, and I do not have any big plans about it anymore. There are certain bugs that still need to be fixed, but that's a minor thing.

So, I've been thinking ... trying to find a way out of this really hard... And I would like to suggest an alternative way.

What if, instead of just deleting all the old code, we would start a big refactoring project.

The rules of this "project" are:

1) We create "classic" or "stable" branch at current master, this will be the branch to complete AGS 3.4.2, and maybe create 3.5.0 with SDL support from @sonneveld .

2) Master branch becomes the place of a grand refactoring project.

3) It is made absolutely clear, that master branch is not guaranteed to be safe to use until we say so (come to certain stage in refactoring). We allow it to break things from time to time, and don't pursue 100% functionality until later stages.

4) No new features are allowed, maybe with few exceptions that don't counteract refactoring.

5) We work altogether, refactoring part by part. The less "part" is, the better (when possible).

6) We discuss refactoring, opening an issue ticket for each separate task. This way everyone knows the problem in question, and can help and give advices. Probably there should be a "master" ticket with the list of the biggest issues which strangle development, for the reference, to pick from.

What do you think about this?

AlanDrake commented 6 years ago

I don't care for refactoring legacy code, I think it's a waste of time. The only reason I ever picked up AGS in the first place was for its ability to make adventure games. The thought it could run older games never once crossed my mind. I don't want to sound harsh, but 99% of the potential game developers out there, only cares about making new games.

You know I started this whole cleanup for making it easier to refactor and evolve, so I really don't see why you're now suggesting a big refactor on master. It's more work and more frustrating. Did anything change to make you think it's possible now, when it wasn't possible three years ago ?

I feel it would be a mistake. We just don't have the manpower and I don't want to do it. I've made it clear various times, I don't want to deal with legacy code. So either we cut away the dead weight and work together to clean, refactor and improve AGS, or I'm throwing the towel.

ghost commented 6 years ago

Alright, I see your point. Well, maybe it is too late for this refactoring. This is something that should have been done many years ago. Too much time wasted. Unfortunately, I seem to still living part in that dream of ideal project.

I also want to be honest. Even if we go the way you want, I do not have any faith in AGS anymore. It's too outdated, and too much has to be changed to make it a bit more modern.

I maybe will help with some refactoring in the cleaned code, but cannot promise much.

ghost commented 6 years ago

So, what should be the plan then? We declare that master is under big rewrite and should not be trusted.

There is a lot of things that probably should be scrapped along with the legacy code. Everything that imposes constraints on data structures, making it difficult to modify them. That is: parts of the plugin API, the memory variables directly exposed into script API.

We need a list of changes from your branch. You've mentioned that there are some additional changes, like different mask sizes in rooms, this all has to be documented.

ghost commented 6 years ago

@sonneveld ,

Leaving legacy code in its own branch will mean it will wither and die. The only alternative to run old games on new hardware is to hope that scummvm supports it.

I agree that there is this danger, but as you see, this project was from the start suffering from unclear goals and mixed expectations. When I joined, actually I wanted to do about same thing as @AlanDrake is doing now. That was also the reason for me to join. I knew that Alyssa Milburn from scummVM team is working on a port, so I figured that such port will make old games covered.

There were bad decisions made, and we started a new development over JJS's ports, which goals, on contrary, was to support all the old games. ScummVM port was never finished, and we got stuck in what we have. To tell the truth, I always felt terrible about this, because I too had a lot of plans regarding AGS, which I could never bring to life, and I also never intended to be maintainer of a program to run legacy games. If I could return back in time, certainly I would choose another path.

Ideally we probably should be doing what you say, but it's true that there is not enough people involved, and there is also the fact that too much time was lost. I am not even sure if AGS will be used by many people in the future.

It is still probably possible to kind of mix Alan's solution with yours. For example, when the new cleaned branch is set and tidied more or less, we may begin bringing old support piece by piece, in such way that won't trouble current development.

AlanDrake commented 6 years ago

Yes, we declare that we dropped legacy support and are in the process of a rewrite. Baseline for importing projects will be minimum the latest 3.4.1 until a new game format is decided. I'll write a few notes about the quirks related to importing projects.

At the moment the current ags4 branch should be able to run modern ags games without issues, except in cases where sprites were flagged as lowres and the game was hires, or a deprecated functionality was used.

I've been adding "// CLNUP" comments to potential places where to intervene. I'm counting on your expertise to figure out other points needing to be addressed. Some code has been commented and could be removed, if not needed for reference.

As how to move toward the future I'm not sure.

Here's the list of changes:

Other stuff removed was unused variables and dead code.

necros2k7 commented 5 years ago

Ok, deleting old games support, but how can people run old ac2games/DOS games? Calsoon for example, Crime time, Book of spells? Isnt it a good idea to support all engine versions in one runtime exe? (like for example ScummVM supporting wide versions of SCI , AGI engines for example). Im ok with engine cleaning but if there is a standalone win compatible legacy DOS AGS engine games runner

AlanDrake commented 5 years ago

Ideally, the legacy game engine itself should go in a separate module. That way it would make possible to support both old and new development. The problem is untangling the engine framework from the game engine logic, so that they become indepentent from each other. Once 3.5 is finalized and merged, I'm planning to attempt this for AGS4 in order to see how it could be done. If succeeded, it may be used as a blueprint to perform the same for AGS Classic.

In the end what we should be aiming for would be to get the following components:

The pros:

The cons:

How to get there though? How to design the framework API? My main focus is making a game not making game engines, we'll need to work together to make this happen.

fernewelten commented 5 years ago

Ok, deleting old games support, but how can people run old ac2games/DOS games?

One obvious solution would be to continue using the existing engines that the games were designed for and bundled with. The old executables don't drop off the ends of the world just because new versions are being released.

like for example ScummVM supporting wide versions of SCI , AGI engines for example

I don't think it is quite fair to compare the AGS to the ScummVM project.

The original SCUMM system was developed for the specific respective current game it was used for, and it didn't have compatibility to former games in mind. As far as I know, nobody tried to keep the SCUMM engine compatible to former, already published games, when a new game was being developed and that current game had specific new requirements. This is one of the reasons that the ScummVM project needs to deal with lots of one-shot anomalies, copy protections, etc.

The ScummVM on the other hand has been designed from the very get-go as a historic emulator project, after the demise of the Lucas Arts / Sierra studios. They painstakingly reconstruct copy protection mechanisms, one-shot kludges. and curious quirks of the ancient games, ideally in order to get them all to run.

In a similar vein, an "AGSVM" emulator could be maintained in a separate project to make ancient AGS executables continue to run. Developers that are into that kind of thing (I'm told there are some) can disassemble bytecode and find out how things used to be done in yonder years and olden times. More so, they can even access the olden Engine sources, due to the wonders of code version systems.

On the other hand, the AGS has been designed right from the get go in Chris Jones' times as a tool for developing new adventure games, just as the original SCUMM system was. Should compatibility to ancient versions impede progress seriously, it can drop compatibility and move on.

It is debatable whether this is indeed the case. As far as I understand, @AlanDrake 's argument is that this point has been reached. His point is that compatibility concerns have made the code so complicated that it has become inordinately difficult to implement new features and to hunt down bugs. The code can be made much sleeker and more maintainable if those compatibility features are all dropped that nobody uses any more in new games anyway.

If that is so, then that ballast could be jettisoned to re-gain the ability to make progress. Compatibility to old, already compiled executables would be a secondary concern.

fernewelten commented 5 years ago

I fear that this will become a moot point very soon, because AFAICS, nobody is doing anything non-trivial in the AGS4 branch, nor have they in the last year or so; and nobody whatsoever has championed the branch in the last year. All the efforts have been directed to just keeping the branch fairly in sync with AGS 3.5, with @ivan-mogilko being the only one saddled with that work. (I might be unfairly biased in this because I didn't really keep close tabs.)

Without any champions and without any momentum, this branch will be quickly overtaken by the power of the factual and left behind.

To make the branch thrive, developers will need good reasons to want to switch, so the branch will need to offer them advantages that can't be had in the 3.5 branch. And end users will need good reasons to prefer an "4.0 AGS" over an "3.5 AGS", so just taking out deprecated code isn't enough: Something new would need to be put in that would make "4.0 AGS" seem superior from a user perspective. But as far as I know, the 4.0 branch hasn't even compiled to a complete studio in the last three months or so so that end users can't test and compare it to the "3.5 AGS".

ghost commented 5 years ago

nobody is doing anything non-trivial in the AGS4 branch, nor have they in the last year or so; and nobody whatsoever has championed the branch in the last year. All the efforts have been directed to just keeping the branch fairly in sync with AGS 3.5

To be fair, early previous year I was not planning to continue development in ags3 branch at all and even started doing some further changes (stripping on legacy functionality) in ags4. The reason 3.5 exists as 3.5 was basically that there were continious "urgent" user requests regarding things that were "spoling their lives" in ags. As ags4 could still be kind of unstable after all the things ripped of (and ofc because user projects could still use old stuff) I decided to make "one last version" with ags3 code.

PS. Later on I also was very concerned about sonneveld wanting to stick to legacy functionality, while I was expecting his SDL port to finally come out, which had been an additional factor keeping me in ags3.

AlanDrake commented 5 years ago

Speaking of which, how close is the SDL port to completion? It was a big reason why I never attempted anything too drastic. I didn't want to make the branch unmergeable.

ivan-mogilko commented 4 years ago

I think to close this thread, it has been here for 3 years, last updated year ago, and guess everything that had to be said is already said, so...

PS. in any case it would be better to open a new one(s) dedicated to particular actions.