Add some CI

[suve]

It would be useful to have Continuous Integration set up for the project, so we can check if PRs compile successfully. I have some experience with CircleCI and GitHub Actions, so I can handle setting this up. I'm opening this issue mostly to discuss the matter and ask some questions.

• Which platforms do we want to test with? GHActions support Linux, Windows and MacOS runners. Setting up a Linux workflow would be trivial; Windows and MacOS would be more complicated.
• Do we define a specific version of the compiler we want to use, or just go with whatever version can be installed via apt / dnf / brew / chocolatey?
• FPC doesn't seem to have an equivalent to GCC's -Werror option (i.e. treat warnings as errors). Do we want to try and figure out some solution for detecting warnings and marking the CI job as failed when they do?
• Do we only compile the headers, or do we want to add some small tester program and then compile and run it?

[Free-Pascal-meets-SDL-Website]

Wonderful idea, it came up to me lately, too! Unfortunately, I'm not experienced in setting up these systems and in test developing, which is why I was hesitant. Would be a nice oppotunity to get into it, though.

• Which platforms do we want to test with? GHActions support Linux, Windows and MacOS runners. Setting up a Linux workflow would be trivial; Windows and MacOS would be more complicated.

I would prefer GHActions, but merely because it is integrated with GH already. Other than that, I have no technical reasons.

• Do we define a specific version of the compiler we want to use, or just go with whatever version can be installed via apt / dnf / brew / chocolatey?

Do you consider it more beneficiary to have the same compiler on all systems or to have the latest package manager-available compiler on particular systems? - I think setting up the latest stable compiler on all systems is way too time-costly as this is really a community project and should stay fun :-). What do you think?

• FPC doesn't seem to have an equivalent to GCC's -Werror option (i.e. treat warnings as errors). Do we want to try and figure out some solution for detecting warnings and marking the CI job as failed when they do?

I agree, FPC cannot do this.

This sounds like an own task. I like the idea, but maybe we should put this to a task with lower priority. Maybe there could be a "warning test" (see next point).

• Do we only compile the headers, or do we want to add some small tester program and then compile and run it?

I guess the tester program is a more solid choice. Do you think FPCUnit could be a suitable choice?

Another point to consider: Where do we store the test program or test programs. I wonder if it would be a good idea to have a subfolder fpc-tests or pascaltests. It should be clear from the folders name that these tests are not related to the original sdl2 package. (Once I had the idea to put some example codes in an example folder, Here applies the same, it should not suggest that these are translations from original sdl2 examples or something, so I came up with pascalexamples - I will create another issue to discuss this soon)

[suve]

I would prefer GHActions, but merely because it is integrated with GH already. Other than that, I have no technical reasons.

I played around with GHActions and, to my surprise, setting up jobs for MacOS and Windows turned out to be fairly simple.

Do you consider it more beneficiary to have the same compiler on all systems or to have the latest package manager-available compiler on particular systems?

Always using the same version has the benefit of having a well-known environment. Installing from a package manager, the version may change one day and suddenly the CI starts failing. Though I guess one could argue that from this project's perspective, we want to always be using the latest version, so if a package manager updates FPC and stuff breaks, we can react quickly.

This sounds like an own task. I like the idea, but maybe we should put this to a task with lower priority. Maybe there could be a "warning test" (see next point).

Setting this up on Windows will probably be a bit tricky, but on Linux and MacOS we could just have a shell script sift through the compiler's output. There's the -FeXXX option, which tells the compiler to write errors and such to file XXX - after compilation, just check if the file is empty or not.

I agree that this could be added later, though.

Do you think FPCUnit could be a suitable choice?

I've never used it, so I can't really say. I don't think that creating a "proper" test suite has much sense; it seems to me that it'd be quicker and simpler to just have a bunch of small, basic programs that we compile and run - the test criteria being simply "it compiles successfully and doesn't crash when run".

[Free-Pascal-meets-SDL-Website]

So, I summarize, if I may: 1) We use GHActions. 2) We use the latest package manager-available version of FPC (which may different for different systems). 3) We do not use a test suite but a collection of simple test programs and test if they compile without exceptions. 4) (later) We add a parser script to check the compiler output for error/warning details.

Did I miss anything? Do you agree?

[suve]

Yes. 1) and 2) are already covered by the workflow I submitted in PR #25. 3) we can add later, once we sort out #22 and actually have a place to store test/example programs separately.

As for 4), since we've settled on using GHActions, I've been thinking about looking into the Actions API and trying to create an Action myself - this would allow for using some GitHub-specific features, such as having errors/warning show up in the Pull Request web view, instead of requiring us to navigate to the CI run logs and skimming through them manually. But that's a project of its own and I'm a bit busy lately, so it'd probably be done no sooner than end of the month.

[Free-Pascal-meets-SDL-Website]

Sounds great!

Please let me know, what you think about my new proposal at #22 .

[suve]

I have created a GitHub Action for running FPC: https://github.com/marketplace/actions/free-pascal-compiler It handles generating annotations on commits and pull requests, and also has a fail-on-warning feature.

There's still a couple of minor issues I want to iron out, so I haven't slapped a v1.0.0 on it yet, but the core functionality is working fine. If you think that's ok, I can open a PR to change our CI workflow to use this Action.

[Free-Pascal-meets-SDL-Website]

This is great news and the Action looks beautiful. Can't wait to try it out with the pending PR.