Editor: for making templates, support a include/ignore file in a project folder

[ivan-mogilko]

As per @ericoporto 's suggestion here

Problem

Currently the template packaging from AGS Editor has a hardcoded list of files, and does not let to package custom files, let alone subfolders. At the same time it might be desired to get other assets into the template (sprite sources, for instance), which are located in subdirs for convenience.

Solution

We could implement a GUI dialog in the editor that selects files for template, but then it would also be wanted to save this information for the future use, as a user may want to recreate template from the same game project later, multiple times, like we do with our default template projects. This means that the GUI is not a solution, we need to begin with a file that would be saved in the project folder, and won't be a part of the game project itself (no data saved in Game.agf, Game.agf.user).

Note that this approach is not something new for template building, this process already supports template.txt and template.ico, which define optional template's description and an icon respectively.

As @ericoporto suggested, this may be done using a file formatted similar to .gitignore, as a list of patterns. It may be interpreted either as a "include list" or "exclude list", whatever seems convenient. It does not have to be exactly .gitignore kind of syntax, may be of standard regex syntax, or even classic wildcards. In terms of implementation, .NET framework already has a regex support, and C++ that we use in our tools has std::regex, which may be more or less enough to support using such file when gathering a package.

Tasks

• [ ] Decide on a file name and format.
• [ ] Implement using this file by the Editor when creating a template from the project, along with hardcoded standard game files.
• [ ] Need to decide how Editor uses this file: either as a complete override of hardcoded list, or as an addition to hardcoded list. This choice is important, as it will define how this feature is going to be used. Given a list may be passed to other tools, including possibly third-party tools, perhaps "override" is better (although that will require bit more effort from user to configure it).
• [x] Optionally, support such file passed as an extra arg to our agspak tool (This may be moved out to a separate ticket, as with command line tools there's more flexibility in which files they are applied to already).

[ericoporto]

It appears it's just redoing the function below here

https://github.com/adventuregamestudio/ags/blob/b403471f7e4737cf88fb6f46b89556b049c33e1a/Editor/AGS.Editor/Tasks.cs#L44

There is some additional details that when writing the CLIB package it appears there are some hardcoded conditions.

I played with the idea of using the Ignore package from NuGet that I mentioned in my comment. The only issue is the Ignore package requires .Net Standard 2.0 and this requires the .NET Framework target to be updated to at least 4.7.2, and this requires upgrading the Docker image used for the Windows builds, so I think I need to solve that first.

[ivan-mogilko]

I don't think we need any extra packages, this should be doable using file iteration + regex (a wildcard pattern may be converted to regex pattern, we have a example function for this ready). And this also has to be supported in agspak tool too, which will have to be done in C++. Eventually, if we switch to using standalone tools, this will be done simply by passing a file with patterns into agspak, so any extra .NET package will become redundant anyway. This has to be kept in mind.

[ericoporto]

Well at least we can learn something from their tests

https://github.com/goelhardik/ignore/blob/main/src/Ignore.Tests/GitBasedTests.cs

Whatever is implemented would require passing similar tests.

[ivan-mogilko]

Whatever is implemented would require passing similar tests.

This primarily depends on which format do we choose. .gitignore format has basic wildcard options, but also something on its own.

[ivan-mogilko]

Oh, looking at their code, in the nutshell they do similar thing by converting these ignore rules to Regex, except they have to parse more rules. Well, that should be expected, I guess. e.g. https://github.com/goelhardik/ignore/blob/main/src/Ignore/ReplacerStash.cs

EDIT: the overall code also does not look like a lot. Yes, it's mostly a table of converting all the .gitignore rules to regex patterns. This may be interesting for research.

[ivan-mogilko]

Hmm, okay, to summarize, TBH my primary concern was not the use of NuGet package per se (since they are just downloaded when building the editor), but the chosen pattern format, because then it will also have to be supported by standalone tools, and any third-party tools (if anyone would like to write their own for some reason). Simple format is easier to maintain. "Wildcard" is the minimal pattern format, so the question is mostly whether there's a practical need to support an extended .gitignore format in our case.

Looking at this "Ignore" package's code, it is converting to everything to regex, which makes total sense of course, so it should be not a problem to replicate in C/C++ code when we need to.

So, I cautiously think that may be alright...

Except, I still think that it the template's file pattern should be treated as "include" rather than "ignore". But I guess that's a matter of negating each rule?

[ericoporto]

Except, I still think that it the template's file pattern should be treated as "include" rather than "ignore"

Same as gitignore, it would have to support both as you may want to include a folder but exclude specific files. I think what it means is what is the default meaning (that the ! will then negate).

It could also have certain folders/types be included/excluded by default - in git, the .git folder is ignored technically.

[ivan-mogilko]

Yes, it's useful to have "negate" operation excluding things from the rule.

But to clarify, what I mean is that it might be more convenient to have a list treated in general as a list of files to include rather than a list of files to skip, because then the generated template will be explicitly defined by that list only, and not by "location". When people work with a game project they often may have random files in the folder, and IMO it would be nice to not have to add all these files in this list (and these files may be different from person to person).

I.e. an explicit list of files to gather like

Game.agf
*.crm
etc

[ericoporto]

OK, I made a prototype of list of files matching in here: https://github.com/ericoporto/filematcher/

It works like this:

• set a directory as "target"
• get recursive list of all files in directory as a list of strings relative to the target directory
• reads a file of predefined name containing the glob (fname) patterns of what files to exclude (details here)
• match each file to the patterns and take a decision to include or not the file
• at the end, it prints the files that match the pattern (files that don't match any pattern are INCLUDED)

[ericoporto]

I started to port my filematcher to the AGS Repository, still not in great shape but it's almost there for a working prototype in agspak https://github.com/ericoporto/ags/tree/refactor-agspak-cmdlineopts

[ericoporto]

I made a version of this idea on agspak. I think it's a good place to explore if the syntax and logic is alright and the filename to use for it.

Once we can get on some practicals decisions there, then the support in the Editor may be done by either switching to using agspak in it and shipping the tool with the Editor or by simply porting the code logic to C#. The code logic turned out to be very small and I believe it will be easy to port it if desired.

[ivan-mogilko]

agspak part completed by #2505.

Now Editor has to either reproduce this behavior (duplicate the code in C#), or call agspak (will at least require to distribute agspak as a part of Editor program files).