Add none SM specific natives to SP so they are accesible by the SP Shell to allow for Unit Tests in some cases

[c0rp3n]

I believe this change is needed as with a lot of SM codebases becoming larger the use of Test / Unit testing is more needed. It is'nt practical to setup a game server just for CI and testing so being able to use spshell would be really useful as it gives access to SP without a SM gameserver.

So the main aim of this is to be able to have unit tests for at least some parts of libraries, for example with my library colorlib-sm. I would like to be able to have tests for functions such as CFormat, CRemoveTags, CRemoveColors, these have no interaction with the SM specific functions but for ease of writing tests it would be better to have access to functions such as stringcmp from string.inc.

Instead of having to create new stock files to replace the default includes of SM though this is something I am currently considering.

Other libraries may require functions from these include files for tests:

• adt_array
• adt_trie
• bit_buffer
• files
• functions
• handles
• profiler
• regex
• sorting
• string
• testing
• vector

[asherkin]

This is something I've attempted to work on a few times in various iterations (Spider has a clean implementation of a bunch of natives, which was unmaintainable), it is definitely something I'm interested in having but it's quite a huge amount of work. Anything implemented in logic in SM should be fairly simple to extract, but there is a surprising amount of coupling with the game even for basic string functions (e.g. almost everything is format-class, and formatting has Source-specific format specifiers like %N - all FS stuff is also coupled with the engine filesystem interface).

A few specific issues with the includes you've listed:

• adt_*, handles - these rely on Handles which are an entirely SM concept, although there is some core machinery moved into SP I doubt we want all of IHandleSys as part of SP.
• bit_buffer - this isn't part of SM, it's a game engine wrapper.
• files - these can use the engine interface per-function, which'll make the internal implementation horrible to decouple.
• regex - this isn't even part of SM core, it is provided by an extension.
• string - the formatting stuff mentioned above.

I think there are two paths forward here which I've been investigating on-and-off:

• Extract everything not coupled to the game runtime environment (which as I say is not a lot - and I worry about it being not enough) to a static library that can be shared between SourceMod and another embedder (which probably wouldn't be spshell, but something built as part of SM). The API surface can slowly be expanded by decoupling and introducing host extension points (like a pluggable format function the embedder has to provide). This has the advantage of working with some existing plugins, but will hit hard roadblocks with the existing API.
• Introduce a completely new stdlib for SP designed specifically to be decoupled and for SourceMod to use as well. I've been watching the WASI spec for ideas here. This'll allow solving fundamental issues like the FS API (by having e.g. a FS interface factory rather than a bool per method) and improving the consistency of the API, but at the cost of being a completely new API for plugins.

[c0rp3n]

Yeah I see the second path will be better in the long run, prefarably this would mainly wrap around the c++ stdlib so there is less maintenance required, or at the least amtl. If we did not have handles SP would still need some abstraction for native types, so that we could have say a vector wrapper instead of ArrayList, whether this could just be an abstraction over pointers for simplicity.

In the short term I plan to just write out the functions I need in SP so I can have basic unit testing running and will keep those public.

And for working on a sp-stdlib I'd be happy to help work towards this.

[dvander]

As SourcePawn's testing has expanded I've more and more wanted some kind of standard library. I tried doing this for floats, but maybe we should go whole-hog and and move a bunch of stuff over. I can't really see any downside.

Handles are indeed the tricky part, but... handles are essentially a big sticky note saying "TODO: insert garbage collection here". GC would have to be implemented at the VM level, so why not Handles? I don't see anything horribly wrong with moving HandleSys over to the VM. The compiler already hardcodes intrinsics for them. Stuff that's SM specific can be factored out.

I'd have to take a closer look at the file API to understand the problems there. It doesn't sound insurmountable though. The formatting stuff is harder. As asherkin noted that's completely dependent on a whole bunch of srcds machinery. We could certainly factor the core of formatting over to the VM, and allow SM to hook into it, but the "%N" stuff would be completely untestable. So that gets me to this statement:

For unit testing the non-srcds functionality of SourcePawn, a standard library would help a lot. For everything else, you'd want something like Selenium for srcds.

We've talked in the past of another idea: having a fake srcds that provides mockable implementations of all the SDK interfaces. This would be an additional build of SourceMod with a clean-room implementation of the SDK, and a command-line tool that loads everything and acts like srcds. Eg, you'd run mock_srcds, it'd load Metamod:Source and SourceMod, but the actual binary that gets loaded would be "sourcemod.mock.2.dll" or something instead of one compiled against an actual game. This has a ton of disadvantages, namely, being a HUGE amount of work, not being functionality usable for game-specific stuff, and components like gamedata would be really weird. Main advantage: it'd be fast to test with, and no problem for Travis/AppVeyor to run. Possibly, we could take it a step further and not have a "mock" build, but instead, mock each SDK within the fake_srcds. But then you've multiplied your work by ~60x (# of SDKs * platforms) or something.

So, I keep coming back to: actual srcds seems like the best thing to test with, and any kind of driver to automate it would be great. Independently, I'm happy to start moving SourceMod components over to SourcePawn. That also improves its standing as an actual language. (Not really, but I can pretend).

[c0rp3n]

I had done some prototyping with a alternate handle like system, but it was more just a experiment and is just setup like handles with a new name (https://github.com/c0rp3n/sourcepawn/tree/native-object-prototype).

From looking through the handle system it seems heavily wrapped with a few of the other systems, though some of which would be useful like having extensions and plugin systems as part of the VM too as could allow for using SP shell with both extensions and SP plugin libraries.

My aim currently is to get something that can run in Travis and AppVeyor currently, not sure how well setting up a gameserver would go for testing plugins though SM, depends if you would be able to cache the game server files but could be practical if we have a test driver there was a recent plugin to provide features in a plugin but would still need to be automated (https://github.com/clugg/sm-testsuite).