Library for parsing GMS2 project files

[Gamer-XP]

Is your feature request related to a problem?

There seem to be plans for IDE plugins and such already, but I'm not sure if it's part of it, so I'll ask separately.

I'm making some different external tools for working with GM's project (both standalone and .bat file post-processing), and being able to easily load/save project files externally is a must-have for that. I assume that will also be needed for IDE plugins in the future. For now, I'm making my own scripts, but they require to make a lot of compatibility checks for different serialization versions, and, when file format is inevitably chagned in the future again, update it to support new versions.

But GM already got their internal scripts for loading files obviously, including older formats. If it's not too tightly bound to other DLLs - maybe it can be released separately?

Describe the solution you'd like

Release a github repo with a library for loading/saving GMS2 projects. If it's internally used one - that's a plus since it will be supported automatically.

Describe alternatives you've considered

Will just continue writing my own library, nothing new here.

Additional context

No response

[adam-coster]

In the interim, I've got some projects for parsing and manipulating project files:

https://github.com/bscotch/stitch

In particular, for serializing/deserializing yy and yyp files: https://github.com/bscotch/stitch/tree/develop/packages/yy

[Gamer-XP]

I... wonder if it's possible to use such library in my C# projects? Never worked with TypeScript before.

[adam-coster]

No idea! Not directly, probably.

[rwkay]

We already have a library that is used by both the IDE and the Asset Compiler for doing exactly this, but we do not have any external documentation for it, it is in C# and if you feel inclined you could take a look (but documenting this is very low down on our list of priorities) and it is in a state of flux just now between versions while we change things in preparation for prefabs - the CoreResources.dll is the assembly that you want to look at, but it is not a simple thing to use (nor designed for anyone external to us using it for now)

[Gamer-XP]

Thanks, I"ll check it for now.

Edit: I think I've managed to make it work somehow. Kinda weird that loading is super slow in debug mode though - 8 seconds vs 0.5 seconds.

[stuckie]

As Russell said, there's no real documentation for it at the moment, and there are a few external setup areas that are assumed to be done. At the moment, there are two serialisation strategies - you've likely seen it complain about missing fields in .yy files now and then. The "fast" serialiser is a separate set of dlls that are loaded dynamically. If they can't be found, or the .yy format does not match, the "slow" serialiser is used instead; which is likely what you're seeing there. Functionally, they are the same, so you won't be missing out on anything feature wise. Also feel free to add comments as to anything in particular you would be looking to do so we can perhaps fast track documentation in those areas to help the community.

[Gamer-XP]

From what I see, it uses Fast Deserialization in both cases. But it's very slow when I lanch app in debug mode.

Debug: +++ FAST SERIALISATION SUCCESSFUL LOAD AND LINK TIME (with worker concurrency 8): 7662.9696ms Release: +++ FAST SERIALISATION SUCCESSFUL LOAD AND LINK TIME (with worker concurrency 8): 514.7147ms

As for things that are kinda annoying:

1. I had to initialize few separate things before project loading started working: FileIO.SetDefaultFileFunctions(); GMProject.LicenseModules = new PlaceholderLicensingModule();
2. I had to add few more separate DLLs before it stopped thowing errors: FileIO, HtmlAgilityPack, YYPSerialized, YYPSerializedAutoGenerated, YYPSerializedImplementation
3. It took some time to find what I'm supposed to load project from: GMProject.Load, GMProject.Deserialize, ProjectInfo.LoadProject. Should I do some weird things before those methods can even work? Well, I've found it in the end

This if current loading method I'm using:

public static Task<GMProject> LoadGMProject()
{
    FileInfo? gmProjectFile = GMSUtilities.FindProjectFileInParent();
    if (gmProjectFile == null)
        throw new FileNotFoundException("GameMaker project file not found");

    FileIO.SetDefaultFileFunctions();
    GMProject.LicenseModules = new PlaceholderLicensingModule();

    var loadingWait = new TaskCompletionSource<GMProject>();

    ProjectInfo.LoadProject( gmProjectFile.FullName, true, ( _r ) =>
    {
        Console.WriteLine( _r );
    }, ( _r, _progress ) =>
    {
        Console.WriteLine( $"Loading: {_progress*100}%" );
        if ( _progress >= 1 )
        {
            loadingWait.TrySetResult( ProjectInfo.Current );
        }
    }, ( _r ) =>
    {
        Console.WriteLine( _r );
    } );

    return loadingWait.Task;
}

[stuckie]

Our usual setup is something like:

• FileIO.SetDefaultFileFunctions();
• MessageIO.SetDefaultMessageFunctions();
• ResourceInfo.FindAllResources(); // for the fallback serialiser
• GMProject.LicenseModules = something to return a new List( .. list of options you're interested in; "Main", "Windows", etc... );

ProjectInfo.LoadProject will then go away and load the correct thing, using whichever serialiser method available ( the YYPSerialiser being the fast one as you've already noticed ) and return you a GMProject in the success callback:

ProjectInfo.LoadProject(_path, true,
  (_res) => { /* success */
      Console.WriteLine("Success");
      project = _res as GMProject;
  },
  (_res, _progress) => {
      Console.Write(".");
  },
  (_res) => { /* failed */
      Console.WriteLine("Failed");
  }
)

I'd also recommend to grab the project from the callback than rely on ProjectInfo.Current... Not sure why it's as slow to load though...

Either way, all pretty much unsupported at the moment until we get time to properly document it, but feel free to poke around and submit feature requests/comment here.

[Gamer-XP]

Not sure what LicenseModules do yet. Probably, no need for them for now: I only need objects, sprites and rooms so far.

For LoadProject, adjusted it a bit with callbacks and extra method calls. Still, that debug mode freeze makes it really hard to use. To be exact, in debug mode it hangs after Loading 25% point for 5-6 seconds, then quickly goes to 100%. Is there a way I can debug it? Debugging into DLL itself seems pretty tough. Maybe I can enable some debug messages that can help?

[Gamer-XP]

Some extra info. Tried removing YYPSerialiserAutoGenerated. Now it uses original loader, which is slower. But it works mostly the same in debug and release modes.

[Gamer-XP]

May be unrelated, but why there are no GUIDs for GM resources? I guess you can track renames when GM is launched with events, but from outside - doesn't seem possible.

[stuckie]

LicenseModules as far as CoreResources is concerned is just which Options to load. As for GUIDs, they got replaced with a "ResourceLink" which becomes a straight pointer to the resource on deserialisation, and a name/path combination on serialisation. That's also the only thing I can think of which is taking the time up for loading, but that should be just as painful - if not more so - on the older serialiser. Would need to investigate this further, but that's a bit beyond the scope of things just now. And in the case of GUIDs, I believe only the GMSprite still makes references to GUIDs for it's layer/frame system as it made it slightly easier on us for conversion purposes between 2.2.5 and 2.3 ( that was a while ago now! )

[Gamer-XP]

Few things about CoreResources:

1. Why GetResourcesByType returns list of ResourceBase instead of List? Quite inconvenient. I guess it's some backwards compatibility with other libraries?
2. A bit weird request, but could be good to have an option for project loading to filter what resources need to be loaded, to speed up loading times. When using those externally you need only specific resource types, and you won't be saving project too in most cases.
3. I assume project loading function can load older project version. But.. is there a way to save loaded project as the same version as if it was loaded? Or do I need to have some weird dynamic DLL loading system to support multiple GM versions for the tool I'm writing? Before, to keep maximum compatibility, I was loading only things I actually needed in the memory, and saves them to JSON additivly, overriding only things I actually change.

[stuckie]

1. Not so much backwards compatibility, more it's just not been updated since the last round of refactoring as it'd touch a lot of code.
2. That may not be possible as we need to resolve all the resource links, and we currently can't know where those are until everything is loaded. It might be something to look into in the future, but not something that we are planning on just now.
3. No, CoreResources as shipped will only support the "current" format going forward. So you'll need to do some DLL juggling to pin a version to each release to be sure of compatibility.

However... there is another tool for handling legacy formats and generally mucking about with projects, which is equally as undocumented, but you're welcome to see if it'll help with your tool, again in an unsupported manner until we find time to document it properly. It's a command line tool called "ProjectTool" and it currently hides in recent runtimes alongside Igor, AssetCompiler, etc... it deals with the issue of multiple formats by literally storing a snapshot of CoreResources at each release and namespaces them internally. This isn't particularly pleasant, and means it only supports a small handful of named formats so we don't go mad managing them all. It will have initial support for upgrading/downgrading between LTS-22, JUN-23, AUG-23 and whatever the NOV-23 format will be from the November release, which may help you in supporting multiple versions.

[Gamer-XP]

Mostly finished writing importer from GM, now need to send data back. But.. how do you create a new resource, room to be exact? I tried creating one with "new", then adding with AddResource. But after saving the project git shows no changes. If I modify existing room - it works just fine (at least adding tag works).

[stuckie]

This is partly why this is all still unsupported just now, as a lot of functionality is still in the IDE rather than the resources themselves. Generally, you new the resource, then you need to add the resource to the project or folder, then add it to project storage:

var room = new GMRoom();
// fill out values - name, views, layers, etc...
MyProject.AddResource(room);
MyProject.AddResourceToStorage(room);

AddResource sets the parent fields and does some checking that it's unique and so on, and returns false if it's not added. A Folder can also be used if you want to put it in a specific place in the asset tree. AddResourceToStorage specifically adds it to the Project's resources list, so it can be saved and reloaded.

In terms of rooms, you need to add and configure the 8 views manually and any default layers you want. The views are literally just new-ing them, setting width/height up, and adding to the room's view list. Usually there's a background and instance layer by default, but this is a preference so can be turned off. Also, layers have a finalise function that needs to be called after being added to the room layers list. Another holdover from the 2.2.5 era format.

Definitely getting into "here be dragons" territory with rooms!

[Gamer-XP]

Thanks, rooms are properly saved now. Is there a function to generate name for the instance in GM style?

[stuckie]

Essentially we do something like this:

        private static Random UniqueNameRandom = new Random();

        public static string CreateUniqueInstanceName()
        {
            return "inst_" + UniqueNameRandom.Next().ToString("X");
        }

Please pay no attention to the man pulling levers behind the curtain. The magic gets spoilt a bit when you see how simple things end up!

[Gamer-XP]

I assume it's a bit more complex? Maybe like this?

private static Random UniqueNameRandom = new Random();

private static string GetRandomInstanceName( GMProject _project )
{
    while (true)
    {
        string result = "inst_" + UniqueNameRandom.Next().ToString( "X" );
        if (_project.FindResourceByName(result, typeof(GMRInstance)) == null)
            return result;
    }
}

Or those instances are not cached in GMProject's list?

[stuckie]

Yes, that should work too.

Instances are slightly weird as there are some compatibility gotchas in that some versions of GameMaker were less restrictive on their instance names than others. At various times instances names have been required to be:

• unique only within the same room
• unique only if renamed from the default
• unique within the same room and renamed from the default

Going forward, as we now have ProjectTool to begin providing validation and project fixing capabilities, we will be looking to start to enforcing unique names across the project. So additionally checking each room's InstanceCreationOrder list would be wise just now - as this is a flat list of all instances within that room, regardless of layer - to prepare for this change. Though I can't say when we'll do it as yet, as there's a lot of testing involved to ensure we don't break projects, as you can imagine.

[Gamer-XP]

Saving project seems to re-save every single resource file. Is this an expected behaviour? At least, GM itself says every single resource was changed while it suggets to reload the project.

On a side note, I think GMTIieset need to also have field to in-editor column and row count. Can be calculated manually of course, but there may be issues with edge cases unless you know the exact internal math.

[stuckie]

Saving has just been recently bugged as no, it should not be resaving everything.

Any other data oddities you find, let us know.. we do have a big internal task to sanitise things before properly opening up CoreResources, but it's a massive maintenance task within how the IDE operates as well.

[Gamer-XP]

About resaving issue. Not sure why it's happeing, but every single resource is marked Dirty after loading (actually, this is also the case when loading the project with GM normally - it's shown dirty right away). I couldn't find how to reset it manually, so I had to use reflection. Doing that made it save only actually modified things (rooms). I'm using older GM version now (2023.1.1.62), so maybe this issue is already fixed in the actual version.

Also, not sure if still an issue, but reloading changes make GM reload them slower and slower with each time. At first it's almost instant, and after, like, 10 times it takes 5-10 seconds. Note: this was when it reloaded ALL project files, and my project got a huge ton of them.

[stuckie]

The dirty-on-load issue is being looked into just now. That other one we're not aware of, so if you can bug it and provide an example project to verify against, that would be lovely!

[Gamer-XP]

Probably, need to add some foolproof for adding instances to rooms. If you add a new one to the layer only - they won't load because they are missing from instance creation order list. Likely need to add some code somewhere to automatically update creation order.

Edit Not sure, but seems like adding entities to the creation order does not properly add them here. In GM they are sometiems displayed as ? icon. Updating list multiple times seems to help, but I don't really get why. From what I see, if I add instance to the list at the same time as when I adde them to the layer - it creates those ? elements. If I do it after project was already saved and loaded - it works fine. Am I missing something?

[stuckie]

When adding a new instance in a room, we:

1. Create a new GMRInstance() - instance
2. Set the instance.name
3. Set the instance.parent to the layer
4. Set the instance.x and instance.y
5. Set the instance.objectId
6. Trigger instance.AddToProject(room.project);

That should be it, though... are you editing a project that's opened by the IDE?

[Gamer-XP]

Doesn't seem to help. I added parent and AddToProject, but it changed nothing. Still getting ? marks

  gmInstance = new GMRInstance
    {
        name = instance.Meta.identifier,
        objectId = obj,
        parent = _instanceLayer
    };
    _gmProject.AddResource( gmInstance );
    gmInstance.AddToProject(_gmProject);
    _instanceLayer.instances.Add( gmInstance );

Right now, I'm testing it with opened IDE, but I first found this issue whne IDE was closed.

[Gamer-XP]

Looking at it with GIT. Seems like "path" value for instances is not correct. New ones got project as "path", GM-created ones got room as path. I can't find how to change that anywhere...

[Gamer-XP]

Ok, found the issue. _gmProject.AddResource( gmInstance ); This reassigned parent to the project. Remove it and it works correctly now.

[stuckie]

Ah yes, as that sets the parent.. another MVC hangover in that originally everything was in a flat database.. that breaks when things like the Animation Curves can both be a Primary resource by itself and a Secondary resource as part of a Sequence track. Room items got away with it by setting their immediate parent to the room, but those things can sometimes get lost ( remember "views" from the 2.2.5 format? ) as that's their only anchor point unless they are written directly within the primary resource.

[Gamer-XP]

Found an issue with fast project loader. I was messing with regional settings on my system those days, and, after that, fast loader started throwing errors when loading sprites: GMSC Error: float expected.

This happed because I had "," instead of "." as decimal separator in system settings. Was pretty common error in Unity too. Can be fixed either by forcing culture to en-us, or by ignoring culture when parsing numbers.

[Gamer-XP]

Encountered a weird thing when creating a room completely from scratch. Everything looks just fine in the editor, but when starting the game - nothing is created. Seems like layers do exists, but all data is missing (I'm iterating over layer data at the room start. It finds instance data, but instances do not exist). It got fixed only after I added a layer manually. What may be causing it? Do I need to call some magical function to fix this?

[stuckie]

I'd have thought that the editor would have been pickier about things being setup than the compiler.. without seeing the project, I wouldn't be able to tell you what's missing off-hand.

[Gamer-XP]

I'm not 100% sure yet, but seems like it was an issue with project reloading. Thing is, I generated a room with exactly same code few times.

• In first case, I had GM opened with the project. It reloaded the room, and instance creation order didn't contain newly added instances. It had ? question mark at the end of the list though. After saving the project - missing instances were still missing
• In second case, I closed the project. After loading everything was in place.

And, I think, this bug is not really consistent. I tested it before on another PC and it was fine even with opened project. Only difference was that on this PC project is reloaded automatically.

[stuckie]

That sounds like a bug in the resource refresh code... if you can get it to reproduce a bit more often, please bug it seperately.