ShipEffects update

[rkagerer]

People were asking for ShipEffects to be fixed to work with KSP 1.1.3+, so I did that.

Please add my CKAN file for ShipEffects-1.0.5 to the official repository.

Note it references "cdn.rawgit.com" - if that's problematic feel free to change to "rawgit.com".

I tested it on KSP 1.1.2 and 1.1.3, and expect it will work on KSP 1.0.4 through 1.1.x, so I configured the metadata accordingly. I've asked the community to test it on earlier versions and will open a ticket if the ksp_version_min needs to be bumped up. If you prefer to take a more conservative approach, I'm totally fine with that, provided it doesn't restrict users of earlier KSP's from accessing this mod via sources pointed to in the existing CKAN files in your registry.

I also compiled a version compatible with the KSP 1.2 prerelease, ShipEffects-1.0.6-for-KSP-1.2.zip, and would like to get it up on CKAN as well if possible. But I need help setting up the CKAN file(s) such that they point to different sources for the different KSP versions. Note ShipEffects-1.0.6 is specific to KSP 1.2 and will NOT work with earlier versions.

ps. If you looked at this ticket an hour ago, please ignore my earlier posts about a single release that worked in both KSP 1.1.x and 1.2. Steam tricked me into thinking I was testing against the 1.2 prerelease when I was actually using 1.1.3 (duh). Due to the way Squad eliminated KSPUtil.dll and consolidated its functionality into Assembly-CSharp.dll, I think it would be difficult to target both versions of the game from a single mod DLL.

[benjwgarner]

Yeah, targeting both versions with the single .dll would be unnecessarily complicated since CKAN supports this functionality already. You're going to want to submit a NetKAN file rather than a CKAN file. The current NetKan file for ShipEffects is frozen because its $kref points to KerbalStuff, so you should replace it. There are a few different ways to do multiple versions, but in your case you probably should use the #/ckan/github... value for a $kref field (unless you want to write individual CKAN files for each version and point to them with #/ckan/netkan). If you set up both versions in your repo correctly, then CKAN should automatically generate a CKAN file for each and add them to CKAN-meta.

[rkagerer]

Thanks! I'm completely new to the NetKAN syntax superset, or even how CKAN in general handles cases with multiple CKAN files targeted to different versions. I appreciate the pointers you gave me, and had a quick look at the docs, but couldn't quickly figure out how to set this up appropriately. I think it's more a case of understanding what output I'm trying to achieve, than the nuts and bolts of each NetKAN tag (although I am encountering a little bit of friction on the latter, e.g. the #ckan/github docs are missing a definition / description of asset_match). If you want to send over a small example geared toward my specific submission (even if it has some "{fill this in yourself}" placeholders), I'd be game to take another crack at it.

Otherwise would it be acceptable for me to just give you two different CKAN files? Here's one set up for KSP 1.2, and the existing one from my original message for earlier KSP's. Does the CKAN client handle this intelligently (i.e. if multiple CKAN's are available, pick the one with the highest "version" tag who's "ksp_version_min" and "ksp_version_max" encompasses the presently installed version of the game?)

Finally, note I don't think any NetKAN script ought to be pulling CKAN metadata directly out of my personal CKAN repository (i.e. out of the CKAN-meta-fwiffo folder tree). Once my versions of ShipEffects are in the official repository, I believe I will want to remove them from my personal one i.e. delete the folder KSP-Fwiffo-Repository/CKAN-meta-fwiffo/ShipEffects/, although I will of course leave the zip files at KSP-Fwiffo-Repository/Mods/ShipEffects/. So, essentially I'm only using my GitHub repository as a CKAN repository, with the /Mods/ folder as a dumb file host for releases (which happen to contain source code in the zips). I don't actually have a traditional, individual GitHub repository set up specifically for each mod I work on (and don't have a desire to at this time), so I don't think there's any metadata available for NetKAN to pull.

If any of this seems odd or awkward feel free to educate me, just keep in mind my time and attention available for it are limited.

[benjwgarner]

I will edit this comment in a few hours with a response; I don't have a huge amount of time right now.

EDIT:

Note: I will be using text in boldface to indicate specifically the repository itself rather than the whole system.

To be honest, I feel like the way that CKAN/NetKAN works is counter-intuitive, unnecessarily complicated, and nearly inscrutable, so don't worry about having difficulty. On a good day, I know just enough to be able to barely coax it into doing what I'm trying to do.

There are two main repositories that contain the metadata: CKAN-meta and NetKAN. The former holds multiple .ckan files for each mod (one per version) but cannot be contributed to directly. The latter holds one .netkan file per mod but can be contributed to directly.

All valid .netkan files either will "compile" (for lack of a better word) into .ckan files, or are .ckan files. Every time the NetKAN indexer bot runs, the .netkan file for each mod in the NetKAN repository is used to generate its corresponding .ckan file, which is then stored in CKAN-meta. This can also be done manually with netkan.exe (syntax is netkan.exe <filename.netkan>), but this just runs locally and does not upload anything.¹ If it's silent, no errors occurred. For more info, the syntax for verbose mode is netkan.exe -v <filename.netkan>. To generate the .ckan file from a .netkan file during an indexing, netkan.exe is simply run server-side (I'm not sure if there's any documentation on this, but I discovered this by looking at the code for the indexer, NetKAN-bot).

Basically, the point is that in order for CKAN-meta to contain multiple versions (that is, one .ckan file per version), the NetKAN indexer must run at least once while the .netkan file in NetKAN points to or generates a .ckan for each version. If it hasn't changed since the last indexing, a new .ckan file is not added. I am not sure, though, what criteria it uses to detect change. If it has changed, the new .ckan file is added to CKAN-meta. This means that in order to add both versions to CKAN, your .netkan file will have to point to or be for each version at two separate times when the NetKAN indexer is run. The NetKAN indexer runs every time a pull request to NetKAN is merged. Once a .ckan file is added to CKAN-meta, it is not removed (although it can probably be modified, although I haven't tried), so the .netkan file can be just for the newest version (it couldn't be for both versions anyway because the spec doesn't allow for that).

I'll show what goes on with my simple mod as an example (it is set up, though, as a traditional repo.) I have a .ckan file for each version of RadioFreeKerbin included in the repo that I update for each release. I have a .netkan file in NetKAN that points to whatever the current .ckan file in the master branch is. When NetKAN indexes, the current .ckan file is added to CKAN-meta. The advantage of this is I don't have to submit a pull request for each update. However, it doesn't go through Jenkins evaluation that checks more things than netkan.exe does. Jenkins automatically checks the .netkan files in pull requests to NetKAN before they are merged. The biggest disadvantage is that if I were to update the .ckan file once, and then once again before the next index runs, the first update would not be picked up by CKAN-meta since it was never pointed to by the .netkan file in NetKAN during an indexing. Changing the .ckan file doesn't trigger an indexing.

I hope that that makes it clearer what's actually going on. Knowing more about how you're doing stuff, I now think (contrary to what I originally thought) that because of your workflow and GitHub repo setup, the easiest option would just be to rename each .ckan file to the filename to ShipEffects.netkan and submit each version as a .netkan file in a separate pull request to NetKAN. I ran both through netkan.exe to verify them and they didn't produce any errors. You should probably wait until the first one shows up in the CKAN client upon refreshing before the second pull request, just to make sure it picks it up.

Endnotes:

1) Oddly, that link is the official place to download the latest stable release of netkan.exe, and there doesn't seem to be anywhere else. The justification for putting it there is rather weak, IMO:

I assume all people who wants to add mods will at some point end up at that page

EDIT 2:

Also, I forgot to mention: yes, the client does handle it intelligently! It works just the way you described:

if multiple CKAN's are available, pick the one with the highest "version" tag who's "ksp_version_min" and "ksp_version_max" encompasses the presently installed version of the game

[rkagerer]

I will edit this comment in a few hours with a response; I don't have a huge amount of time right now.

No problem, I know how it is.

[benjwgarner]

Comment edited. I posted this one to make sure it notifies you. Perhaps I should have just said I'd post another comment. Oh well.

[politas]

@benjwgarner, you have a pretty good grasp of how it all works. It is possible to edit and add information directly to CKAN-meta, but at that point, you're getting into the tricky territory of making sure the upgrade path is maintained, and we really prefer that be done by CKAN team members. We're always on the lookout for new team members who understand how things work, by the way!

@rkagerer, your ckan files are a bit of a special case. I should be able to get them sorted out the next time I'm at my home computer doing CKAN stuff. I was held up processing stuff due to the long download times for the Stock Visual Enhancements mods, but that's done, now.

[rkagerer]

@benjwgarner Wow, thanks for diving into the technical detail for me! That response could practically be copy-pasted into an ancillary / example portion of NetKAN documentation somewhere :-). Looking at your .netkan file, although it is just 3 lines, really crystallized things for me.

I imagine this might become a more popular topic as several mod authors choose to maintain <= 1.1.3 and >= 1.2 releases of their mods. Out of curiosity, is it not possible to simply submit two NetKAN files (one for each version, that point to the different CKAN's)? Or is it not possible / advised for two different NetKAN files to each target the same CKAN identifier?

@politas Thanks a bunch for your help. Sorry if the way I'm doing things is making more work for you guys. I'll have to try doing a Pull request next time around.

[benjwgarner]

@rkagerer You're welcome; writing all that out actually helped me solidify my understanding of how it works as well. :) And no, you can only have one NetKAN file per identifier. The files must have the name of .netkan (as per the spec) so any new file submitted just replaces the old one.

@politas If you're looking for additional team members, I'd definitely be happy to join and help out. How should I contact you so we can discuss this further?

[politas]

@benjwgarner If you're happy to abide by the Code of Conduct, you'll be welcome. You can start contributing PRs without being a member, though.

[rkagerer]

Hi folks, I learned a lot here but I'm not clear what needs to happen next in order to move my original request forward and add my CKAN's to NetKAN.

I would prefer to just stick the CKAN's straight into your repository, rather them hosting them elsewhere on my own and jumping through hoops with kref's. I'm fine with having separate CKAN files for the different versions. I won't be releasing often enough to set up more automation on my end.

I tried a Pull Request but it failed checks with no explanation.

Also note since I originally opened this request there's been an update (ShipEffects 1.0.7).

[techman83]

@rkagerer CKANs no longer get added directly to CKAN-meta. Manually updated mods still follow the process via NetKAN.

https://github.com/KSP-CKAN/CKAN/wiki/Adding-a-mod-to-the-CKAN

Just create your CKAN with all the information manually as 'Identifier.netkan' and once merged the bot will do the rest.

[rkagerer]

Ok so if I read you correctly, all we need to do is rename the two .ckan files to .netkan? I submitted the files a month ago (i.e. linked them at the top of this issue) and updated one of them today...

ShipEffects-1.0.5.netkan

ShipEffects-1.0.7.netkan

I'll try another pull request with the renamed files to KSP-CKAN/NetKAN...

EDIT: Re-read benjwgarner's comments.. so I need to rename each of these files to ShipEffects.netkan and submit them via pull requests one at a time, waiting for an unknown duration in between?

[rkagerer]

Ok, so I tried the first file: Pull request #4772

Now I'm getting another failure message that has no details specified.

BAAAAAAAAAAH!

[rkagerer]

I tried a different approach, using a near carbon-copy of @benjwgarner 's RadioFreeKerbin.netkan. Also produced an error with "no problems identified".

I give up.

Maybe one of you folks can have a look at the last two attempts in my Pull request and let me know what's wrong.

[techman83]

@rkagerer there is a build analyser, but it's something we have manually add a search for each failure type. You would need read the console log if it didn't find anything. I've added an analyser for this case and triggered a rebuild.

I left instructions on the PR of what you would need to do. It looks like you've hit some road blocks on your journey and we appreciate the effort you've undertaken to get past them!

[rkagerer]

Yes it was painful and took way too long, thanks @techman83 and everyone else for your help. In fairness I'm pretty new to the mechanics of GitHub (and can think of at least 10 ways off the top of my head it could be made more intuitive ;-) ).

Didn't realize I can see the console output, that's pretty cool. Out of curiosity why did this one fail (where I just pasted in the .ckan contents into the .netkan file instead of using $kref)?

[techman83]

Transient GitHub network error by the looks.

[rkagerer]

Looks good now