Is there a way to call the serialization/deserialization from C++ to string?

[Demolishun]

In api_usage_example/Source.cpp it looks like it is looking up the handles to the function calls provided to the Papyrus interface. I am not seeing a way to directly call the serialization functions.

I would like to use this plugin to serialize JSON data to be sent over a network connection. So I need byte access to the resulting JSON data. Right now the API reads and writes to file. Is there currently a way to get serialize to a string in the API? I cannot find one.

[SilverIce]

@Demolishun Hi there! Sounds like an interesting project you have. Well, I can add such a method

[Demolishun]

@SilverIce Hello! I wrote this plugin for OBSE a while back: https://github.com/Demolishun/NetworkPipeOBSE It basically gave the ability to pass JSON to and from the scripting environment to an external program. The plugin itself was threaded and provided a port for external apps to connect to. I am finally getting around to modding skyrim and wanted to recreate the plugin. I would love to have mods that allow you to journal in game and have it post on a webpage, or store data in a database for some interaction. Maybe even a chat interface between people.

Anyway, I would want 2 methods: one method would serialize the json from your data structure and return a byte array containing the data, the other method would take a byte array and populate your json scripting object. I am writing the plugin in c++ so I am not sure what form the functions should take. To keep from reinventing the wheel I would like to attach to your plugin through my plugin like your example shows.

What do you need from me? Also, what encoding is the serialized data in? UTF8?

[SilverIce]

Yeah, the data is encoded into utf-8 string. I can add something like JValue.toJSONString and JValue.fromJSONString.

PS Would love to have some integration between Skyrim and my kettle :D

[Demolishun]

"Skyrim and my kettle" That would be an awesome example project to include in the plugin! Definitely a must do.

[SilverIce]

You can try this one - JContainers.3.3.0.zip. There are JValue.objectFromPrototype (it's been there for years and it parses a string) and JValue.toJSONString.

I am writing the plugin in c++ so I am not sure what form the functions should take.

The definitions should look like:

// context is a domain pointer. Pass NULL since more likely you'll use the default one.
    BSFixedString(*JValue_toString)(void* context, SInt32 jc_object_handle, SInt32 json_format_indent) = nullptr;
    SInt32(*JValue_objectFromPrototype)(void* context, BSFixedString input_string) = nullptr;

I'm a bit concerned, as Skyrim string may have length limit - and my C API returns methods adapted to Skyrim

In regards of kettles - https://github.com/NARKOZ/hacker-scripts/blob/master/README.md

[SilverIce]

@Demolishun any news?

[Demolishun]

@SilverIce I modified my plugin to use your JValue_toString and JValue_objectFromPrototype. I get an assert when it checks the function for JValue_toString. If I comment it out it just keeps skyrim from starting. I included jc_interface.h into the project. Do I need more than that in the project? I noticed when adding jc_interface.cpp it wanted to include a lot more things. I also noticed jc_interface.cpp fleshes out the stuff in jc_interface.h. I was going to test with your plugin example next to try and see what I am doing wrong. Also, does the plugin need to be compiled in release mode?

[SilverIce]

I get an assert when it checks the function for JValue_toString

What assert? I haven't tested as I don't have Skyrim installed ATM

I included jc_interface.h into the project. Do I need more than that in the project? I noticed when adding jc_interface.cpp it wanted to include a lot more things.

No, you need to have jc_interface.h only.

Also, does the plugin need to be compiled in release mode?

Well, it shouldn't matter

[SilverIce]

@Demolishun That was a mistake. I absolutely forgot that you HAVE to get a pointer to the default JC domain/context (like in the example) and pass it as a first argument of JValue_toString, JValue_objectFromPrototype. It can't be null

void* default_domain = root->query_interface<jc::domain_interface>()->get_default_domain();

[Demolishun]

I am having trouble getting the example plugin to work. I built it using VS 2013 VC++. I copied the example plugin into the plugins directory of SKSE. When I run skyrim without any mods that use the plugin, it won't ever load skyrim. It spins up the exe, but it sits there and never brings up the main menu. I have to kill the process. Then if I start without the plugin it starts up and loads to the main menu. This is using the example plugin from the JContainers repo. I have not made changes to it yet.

I am loading JContainers (the version you linked above) using Mod Organizer. The example plugin is copied directly to the skyrim/skse/plugins directory. I tried both release and debug versions.

[SilverIce]

@Demolishun , there were several issues. It wasn't OK to publish C API (and use C++ thread-related functions) from DllMain. Also folder structure in the prev. archive weren't proper

TLDR

Here is the fixed and working version (the new function renamed to JValue.toString): JContainers.3.3.0.zip

[Demolishun]

I did test the plugin and it seems to be getting the pointers now. I will let you know what my testing shows for string size. If its big enough I will just document the size and live with it. I will do some sort of test that exercises this to make it easy to see what is going on. I am working on the plugin structure and deciding what functions to include. I also need to get my head around the threading part of papyrus to decide if I need to create my own thread or not.

[SilverIce]

@Demolishun that's good news! I heard (not proven) that Skyrim string length may have a limit like 4kb

[SilverIce]

@Demolishun, the functions, you'll register in Papyrus can be called in undefined way (i.e. amount of Papyrus threads is unknown, call order is unknown and so on). These methods had to be thread-safe

[SilverIce]

@Demolishun , also you may with to focus on Skyrim SE (since SE modding scene can be more active than Oldrim), and maybe talk to @ryobg

[Demolishun]

@SilverIce The mods I want to run are in oldrim.
I have tested creating a thread for a udp server this weekend. It did a loop back test. The next test will be something responding in the scripting. During that test I will be testing string sizes and exercising your functions. I have not done any skyrim scripting so it will take a bit to get my head around it all.

[SilverIce]

@Demolishun , I may even imagine functionality in user mods like downloading mod's JSON data from a server in order to update the mod, or retrieving dX dY dZ data from miBand to turn camera with a gesture, or syncing real weather with Skyrim..

Feel free to ask about scripting

[Demolishun]

@SilverIce I really like these ideas. I am thinking I need to make it so the scripts can create multiple services. I am going to stick with UDP, but allow multiple independent ports to be opened. That way each mod can create their own protocol to talk to an external process.

I have not thought of allowing skyrim to act as a client. I mainly have wanted it to act as a server on a local port, then have external processes do whatever they want. In the OBSE version I have mechanisms for launching external apps/scripts to make it all self contained in a mod. Python is my goto for scripts so examples will be using that. I don't think it is unreasonable for end users to download python. I mean some mods require java to do their config.

[SilverIce]

The mods I want to run are in oldrim.

@Demolishun, Are you going to modify some existing mods?

[Demolishun]

@SilverIce probably not. I do want a journal mod though. One I can enter a journal and it will get pushed to a file/website. Maybe a local git account that gets pushed periodically. Mostly I want the challenge and the fun. If someone wants it on SE I will help make that too. I just don't run SE.

[Demolishun]

@SilverIce Is it still appropriate to make a permanent running script inside a Quest? This is how I used to do this in Oblivion. Or is there a better way to do a management script to oversee things? I see some scripts use threading this way, but I am unclear how this works.

[SilverIce]

@Demolishun, it depends on what that script does. If that script is a part of new SKSE plugin, used by lot of mods, then I don't believe it's a good idea to have a threading on Papyrus level

[SilverIce]

FYI I have pre-released yet another v3.3 - https://github.com/SilverIce/JContainers/releases/tag/v3.3-misc-functionality

[Demolishun]

@SilverIce I didn't realize the scripting would be this involved. I cannot figure out how to get an OnInit event of a Quest to be called consistently when starting the game. It seemed to work, then I changed some things to fix other bugs and it stopped being called. Totally confused. Am trying to find tutorials on this.

[SilverIce]

@Demolishun , OnInit called only once. Do you have a link to creation kit wiki? https://www.creationkit.com/index.php?title=OnInit

[Demolishun]

@SilverIce The problem is I attached a script to the quest in the Scripts tab. It ran a few times and showed me my syntax errors. Then I fixed those and tweaked some things and it never ran again. The wiki says OnInit will run once every time I start the game.
"For Quests and Aliases: On game startup, and again whenever the quest starts, due to the quest being reset." So I don't understand if I put the script in the right place in the Quest itself.

[SilverIce]

@Demolishun , IDK I had experience with OnPlayerLoadGame only: https://github.com/SilverIce/PosePicker/blob/yet-another-way/Scripts/Source/PSM_PosemanagerAlias.psc

@RealAntithesis can you help with OnInit?

[SilverIce]

I.e. I had an alias, attached to the player, that receives OnPlayerLoadGame event

[Demolishun]

@SilverIce are you attaching a ReferanceAlias script to the Player object in the esp? 0000007 or whatever it is, or is the script attached to the quest itself?

I tried this: https://www.creationkit.com/index.php?title=OnPlayerLoadGame_-_Actor But could never get the OnPlayerLoadGame to fire. I do see the OnInit running the first time the esp is loaded in a game. But once it is saved and loaded again it does not fire. That is where according to the link the OnPlayerLoadGame is supposed to kick in. I added a bunch of debug.trace statements in the code to see what was going on.

[RealAntithesis]

@SilverIce I didn't realize the scripting would be this involved. I cannot figure out how to get an OnInit event of a Quest to be called consistently when starting the game. It seemed to work, then I changed some things to fix other bugs and it stopped being called. Totally confused. Am trying to find tutorials on this.

I've just used the OnPlayerLoadGame() event to launch scripts that need to be run on loading a save game. I use OnInit() for the initial startup script when the mod is first installed (it also runs when the quest is reset, which should never normally happen).

Taking AH Hotkeys as an example, the main quest script extends ReferenceAlias and which receives the OnPlayerLoadGame() event, with a player property populated in the CK.

; Properties to be populated using the CreationKit
Actor property playerREF auto

The player reference is assigned to the player property via the ESP in the CK: add a Quest Alias for the PlayerRef to the mod's quest and force the player reference into the alias. The script that receives the OnPlayerLoadGame() event (of type ReferenceAlias) is then added to the Player Alias in the Quest Aliases tab. Then open up the script properties for the script that was just added to the player alias and assign the player reference value to the script's playerRef property (that was added to the script - see code above).

The other scripts that don't receive player events are added to the Scripts tab for the mod quest in the CK.

It might be best to have a look at another mod that already has this all set up to see how they did it, such as AH Hotkeys. AH Hotkeys is primarily all scripts, so the ESP content is fairly minimal to only load the scripts and run them on startup (i.e. OnPlayerLoadGame()).

[Demolishun]

@RealAntithesis Thank you so much for the details on this! I now have a consistently starting function for my mods. Very useful for adding spells/effects to the player at startup for testing.