Revisit Starfield support

[pStyl3]

In issue https://github.com/loot/loot/issues/1882 we discussed the implementation of Starfield support. After some initial confusion it became clear, that at this point in time loading plugins for Starfield wasn't working as it was known from previous BGS titles. It was only after much trial and error from many people, as well as the release of the SFSE plugin Plugins.txt Enabler, that support for Starfield was ready to be released in LOOT v0.22.0.

Shortly after however @ElminsterAU and others from the xEdit team discovered that Starfield had deeper routing issues, which led to this news article on Nexus, and LOOT disabling the sorting feature for Starfield in LOOT v0.22.1 (we also had a discussion for this on our Discord server here).

---

On the 9th of June 2024, BGS released their June 24 Update, which means that Creations and the Creation Kit are now available, as well as that Plugins.txt is also working now.

• Bethesda.net
• Video
• Creation Kit on Steam

Fully supporting Starfield with LOOT would be ideal - it remains to be seen, if it can be done. Let's revisit the situation within this new issue.

[pStyl3]

So far these were the official plugins:

Starfield.esm
Constellation.esm
OldMars.esm
BlueprintShips-Starfield.esm
SFBGS006.esm
SFBGS007.esm
SFBGS008.esm

With the new game version 1.12.30.0 there is one more official base game plugin:

SFBGS003.esm

This is the free part of Trackers Alliance. There is a Creation available that expands on this.

Since I own the premium version of the game, I automatically received 1000 CC (Creation Credits), which I've used to buy

• Trackers Alliance: The Vulture
  • 700 CC
  • sfta01.esm
• Constellation Plushie Set
  • 300 CC
  • sfbgs00e.esm

I've also downloaded the free Creations:

• Starborn Gravis Suit
  • sfbgs023.esm
• Blackout Drumbeat Skin
  • sfbgs00a_a.esm
• Observatory
  • sfbgs021.esm

My Plugins.txt in %localappdata%\Starfield\ currently looks like this

# This file is used by Starfield to keep track of your downloaded content.
# Please do not modify this file.
*sfta01.esm
*sfbgs00e.esm
*sfbgs023.esm
*sfbgs021.esm
*sfbgs00a_a.esm

It seems that a Starfield.ccc file doesn't exist.

[pStyl3]

xEdit v4.1.5g is available on xEdit's Discord server, in the xedit-builds channel here

• interim build to add minimal support for new Starfield release
• no updated what's new
• no correct support for mid masters (they will load as full in this version
• a proper build will follow in the future

[pStyl3]

The workflow in Starfield now is:

• work on mods with the CK as .esp files
• convert .esp files to .esm files for ingame use
• it is strongly implied, that .esp files should not be used ingame (some .esp plugins might even introduce bugs ingame)
• the CK offers three alternatives: Small Master, Medium Master and Full Master
• "Full Master" are standard .esm
• "Small Master" are Light/ESL
  • uses FE slot and loads modules FExxxYYY where x is module index and y is record IDs
  • allows 4095 records in each of 4096 plugins
• "Medium Master" are masters with a new flag, the xEdit team named the flag ESH (as in "Half Master")
  • uses FD slot and loads modules in FDxxYYYY
  • allows for 65535 records in each of 256 plugins

[pStyl3]

Nexus - Starfield Creation Kit is now available

[Silarn]

Info that may be helpful:

Previously, it was reported that the BlueprintShips-Starfield.esm was not locked in place and could be adjusted in the load order with plugins.txt. However, this update has made it so that this plugin is forcibly removed from the plugins.txt by the game, so adjusting its load position is no longer possible.

There were also some format changes to the save files, though I don't know if that info is relevant to LOOT.

[pStyl3]

@Silarn

There were also some format changes to the save files, though I don't know if that info is relevant to LOOT.

Format changes to save files shouldn't be relevant to LOOT.

---

LOOT currently uses these icons:

Master Plugin	Light Plugin

For Starfield it might be necessary to rename these to Full Master and Small Master, and also add a new icon and name it Medium Master.

Doing this would mean, besides that we are in line with how the CK describes them, that plugins should only be assigned one of these icons at a time - so far it is possible for a plugin to have both icons, i.e. 'Master Plugin' and 'Light Plugin' (the example below is from SSE).

[Silarn]

I think master files are still distinct as it's possible to have medium/light flagged ESPs. It's just that they're not well supported and all plugins 'should' be saved as ESMs. Basically an ESM is a master, but it COULD be saved 'small' or 'medium'. I'm assuming ESP files can also still be flagged as masters, light, or medium as well, though there's little reason to do so. It might happen.

Logistically it may make more sense to retain the master icon but change the light icon to something like a pie which is 1/4, 1/2, or full - to represent the different plugin types.

[Ortham]

I've downloaded the Creation Kit and had it sitting open for a couple of days ( and I noticed it eats all my CPU when it loads) - I might be able to start looking into this tomorrow evening. As an initial to do list:

• Rename light masters to small masters for Starfield
• Rename masters to full masters for Starfield
• Add support for medium masters
• Add a warning for any .esp or .esl files that are installed for Starfield
• Check if plugins.txt behaviour has changed
• Check if Starfield.ccc is still resolved by the game when it loads
• Update handling of BlueprintShips-Starfield.esm if necessary (I think it's referenced in places but I can't remember why)
• Check if hardcoded plugins have changed
• Check if plugin data like offsets and flags have changed (besides the medium master flag)
• Investigate whether it's now feasible to enable sorting for Starfield - it was disabled in 3abad011b933132599e88bb5742f072327885426 because of instability of FormIDs across changes between plugin flags, but it seems Bethesda's official stance on that is basically to ignore the problem, we might need to do the same and just have a message advising users not to make such changes.

I've probably forgotten things that should be in that list, it's a while since I thought about Starfield. If you think of anything not there, please let me know!

[sibir-ine]

A few observations re Creations: it looks like the 4 Creations/DLC bundled with the game (SFBGS00{3,6,7,8}) are hardcoded to load. The other Creations, however, aren't, & it looks like you can disable them & change their load order in the Creations menu. Presumably, that means the .ccc file has been superseded.

It looks like BlueprintShips-Starfield.esm's load order is indeed hardcoded now, but the 4 small masters behave weirdly (haven't looked at the medium masters). If they're present in plugins.txt, they're eventually purged from the file after game load, but the game does appear to respect their relative order within the small master block (no idea relative to other types of masters) as defined by plugins.txt for that session. However, when they're not present, they'll load after all other official small master Creations (e.g., the 3 free ones). I still need to test if they load after unofficial small master Creations.

[Silarn]

They are. The hard-coded small masters always load after all other small masters. Which is completely bizarre.

So you will end up with:

• core full masters
• core medium masters
• plugins ordered by plugins.txt
• core small masters

I honestly have to wonder if this is a bug because it makes no sense, although I have no idea how defining a core light master as a dependency would affect this final load order.

To be clear I can't say for sure how custom medium masters fit into this as I don't have any in my list. Is there a known Creation that's a medium master?

[sibir-ine]

To be clear I can't say for sure how custom medium masters fit into this as I don't have any in my list. Is there a known Creation that's a medium master?

Yes, Trackers Alliance: The Vulture.

[sibir-ine]

It appears that since sfta01.esm has SFBGS003.esm as a master, it does indeed load after it, but SFBGS006.esm loads after sfta01.esm. Not sure if there's a medium master Creation that doesn't have either as a master to confirm that SFBGS003.esm loads after it.

[Silarn]

So I think the most likely scenario here is that core light and medium plugins will either load immediately before the earliest plugin that has them as a master, or after other custom / creation plugins in the plugins.txt.

Again, bizarre.

[pStyl3]

Material Symbols & Icons - Google Fonts

For Medium Masters, how about using one of the following two icons?

Radio Button Partial	Star Half

[pStyl3]

Investigate whether it's now feasible to enable sorting for Starfield - it was disabled in 3abad01 because of instability of FormIDs across changes between plugin flags, but it seems Bethesda's official stance on that is basically to ignore the problem, we might need to do the same and just have a message advising users not to make such changes.

I would suggest adding a global message through prelude & masterlist.

For example, following the statement that Bethesda provided the Nexus with (see this news article)

prelude.yaml

##### Global Anchors #####
  - &changeUnsupported
    type: say
    content:
      - lang: en
        text: 'Changing the type of an existing plugin isn''t supported. Additional details are provided [here]({0}).'
    subs: [ 'url' ]

The url could lead to somewhere in our documentation, perhaps to a new bullet point under Help here which for example could be named "Changing Plugin types in Starfield". In that page we could display Bethesda's detailed statement from Nexus' news article (I've asked @Pickysaurus if that would be okay, to which he agreed) and provide some additional context.

Then, we could add this to the masterlist:

masterlist.yaml

prelude:
  common:
  # Global Anchors
    - &changeUnsupported
      type: say
      content: 'Changing the type of an existing plugin isn''t supported. Additional details are provided [here]({0}).'
      subs: [ 'url' ]

globals:
# Unsupported Plugin type change
  - *changeUnsupported

[pStyl3]

In addition to the above message, I think that it would be beneficial to add another message anchor, non-global in this case, that can be assigned to Starfield plugins, where it has been found, that a plugin type change has occured after release/publishing of the mod in question.

prelude.yaml

##### Message Anchors #####

  # Subs can be Full Master, Medium Master or Small Master
  - &unsupportedChangeVerified
    type: say
    content:
      - lang: en
        text: 'This plugin has been changed from **{0}** to **{1}**.'
    subs:
      - 'Previous Type'
      - 'Type'

This message should likely be of type say instead of warn or error, since this plugin type change only becomes an issue, if other plugins use a plugin with a changed type as master .. and since there's no reliable way to know, if there are such plugins, I'd say it's better to keep this message neutral.

Then in the masterlist we could write:

masterlist.yaml

prelude:
  common:
  # Message Anchors
    - &unsupportedChangeVerified
      type: say
      content: 'This plugin has been changed from **{0}** to **{1}**.'
      subs: [ '' ]

plugins:
  - name: 'Example.esm'
    msg:
      - <<: *unsupportedChangeVerified
        subs:
          - 'Full Master'
          - 'Medium Master'
        condition: 'checksum("Example.esm", 6FFE25B7)'

[Silarn]

As per some additional testing and discussions, it seems plugins.txt is processed before anything else.

It processes each plugin and loads their masters in reverse order as defined in the header.

Typically this leads to Starfield.esm loading first but you can actually force a different plugin to load first if it doesn't have Starfield.esm in its dependency chain.

It'll process all plugins.txt plugins this way, and if any core plugin is left unloaded, they will then be loaded at the end in a preset order.

It's possible to create a plugin at the top of your load order that will force the core plugins to load first in whatever order seems ideal.

Do with that what you will.

[Ortham]

I've started to look into Starfield support again, though my time is limited so I don't expect to get much done until some time next week.

I've updated esplugin to expose the medium plugin flag, and I'm now taking a look at what needs to be changed in libloadorder - it needs to know about which Data folder(s) to load plugins from, which plugins are hardcoded, the CCC file (which is still read by the game, IDK if it's used though), which plugins are not hardcoded but are implicitly active (hardcoded plugins load before all others, implicitly active load after all in plugins.txt).

I'm currently looking into how the Data folder in the My Games directory is used, as plugins there are still read by the game, but they don't appear in the in-game load order menu - next I'll try manually adding them to plugins.txt and see if they actually get loaded.

I think the approach I'll take with the new focus on ESMs is that LOOT will still recognise ESPs, whether or not they've got flags (so you could have full/medium/small ESPs as well as ESMs), but we can have warnings that get shown for all installed ESPs since they're no longer recommended for non-dev use.

Thanks for all the info so far!

[Silarn]

which plugins are not hardcoded but are implicitly active

I'm pretty sure the answer to this is all of them. The game removes all of these 'implicity active' plugins from plugins.txt as well, though it may adhere to that load order for that session. So the only reason anything loads before plugins.txt is because of the dependency chain. Even Starfield.esm can be coopted by making a plugin that does not have any masters.

[Ortham]

Looks like Starfield.ccc is still used if it exists (I haven't checked where its plugins load relative to those in plugins.txt), and it's loaded from the game path and also Starfield's My Games folder (which takes precedence).

Starfield still reads plugins in My Games\Starfield\Data, but they don't actually get loaded (I can't see their records in-game), so I'm not sure what's going on there.

The ini's sTestFile<N> entries still work like they do in Fallout 4, disabling plugins.txt and Starfield.ccc if present.

[Silarn]

Starfield still reads plugins in My Games\Starfield\Data, but they don't actually get loaded (I can't see their records in-game), so I'm not sure what's going on there.

This sounds like a bug. Apparently MO2 works around this by virtualizing both locations. Presumably it would 'see' the plugins in My Games but for some reason only loads them from game Data. Since they don't exist there, they don't load.

I was under the impression that the game would prioritize My Games if it existed and only read files from game Data if it wasn't there. Is this no longer the case?

[Ortham]

I was under the impression that the game would prioritize My Games if it existed and only read files from game Data if it wasn't there. Is this no longer the case?

It seems so, in the initial release version of Starfield it would load plugins from My Games, but 1.12.30 and the just-released 1.12.32 don't.

[Ortham]

The plot thickens: I had a test.esp in My Games and a test.esp in the normal Data folder, and I was trying to add an item that was added by the latter plugin, but the game did not recognise the item's record until I renamed the test.esp in the My Games folder. However, the game didn't warn about the plugin being missing (even though it was recorded in the save I loaded), and didn't remove it from plugins.txt.

That's got to be a bug, it makes no sense.

[Silarn]

Wait.. so it was loading the My Games plugin, but only if it also existed in Data? That's truly bizarre.

[Ortham]

Wait.. so it was loading the My Games plugin, but only if it also existed in Data? That's truly bizarre.

No, the other way around, it was loading the plugin but only if it wasn't in My Games.

[Silarn]

Strange. I'm not sure how to square that with MO2 because the game technically sees files in both locations (they happen to be the same file), but the plugins appear to be loading just fine.

[Silarn]

I am curious about the Starfield.ccc. Whether it actually loads plugins and where they get loaded. That's potentially another community 'solution' to the weirdness with core plugins if they still load before plugins.txt.

[Ortham]

Strange. I'm not sure how to square that with MO2 because the game technically sees files in both locations (they happen to be the same file), but the plugins appear to be loading just fine.

How does MO2's virtual filesystem present those directories to the game though? That might affect the behaviour you see.

I am curious about the Starfield.ccc. Whether it actually loads plugins and where they get loaded. That's potentially another community 'solution' to the weirdness with core plugins if they still load before plugins.txt.

Yes, Starfield.ccc is used, see here.

[Silarn]

How does MO2's virtual filesystem present those directories to the game though? That might affect the behaviour you see.

The core Windows APIs are hooked so when the game looks for files it sees all of the mod files. This happens both for the My Game and the core Data directory. If it tries to open a file, we intercept that and return a handle to the mod file. The game simply thinks it's seeing files in both locations. Though again, it ends up being the same file.

[Ortham]

@Silarn My mistake, I think I had two test plugins with the same name and different content before...

On retesting, if my test plugin is only in My Games, the game complains that it's missing when I try to load a save that depends on it, but if it's present in My Games and the normal Data folder, the save loads without complaint and I'm able to use player.additem with the item added by that plugin, so plugins in My Games don't block plugins in the normal Data folder being loaded.

I think that matches up with your experience with MO2, right?

Given what I saw before, it might be that plugins in My Games are only loaded if a file with the same name is present in the normal Data folder, and if so the former take precedence over the latter.

[Silarn]

That may be the case. So for some reason plugins must exist in the game Data but if present in both locations it loads from My Games. :shrug:

[Ortham]

Yep, I changed the name of the item added by the plugin in My Games and that's the name that showed up when I added the item in game, so My Games plugins override Data plugins but don't load if there's no Data plugin to override.

[pStyl3]

Ref: Discussion on xEdit's Discord server, in the ck-modding channel here.

Regarding the message that should warn people to not use .esp plugins, we could do the following:

prelude.yaml

##### Global Anchors #####
  - &espError
    type: error
    content:
      - lang: en
        text: '{0} does not support the usage of **{1}** plugins ingame, but your load order contains such plugins. In order to avoid irreversible damage to your game, it is recommended to not use **{1}** plugins.'
    subs:
      - 'Game'
      - 'File extension'
    condition: 'file("([^\.]+\.esp)")'

masterlist.yaml

prelude:
  common:
  # Global Anchors
    - &espError
      type: error
      content: '{0} does not support the usage of **{1}** plugins ingame, but your load order contains such plugins. In order to avoid irreversible damage to your game, it is recommended to not use **{1}** plugins.'

globals:
# File extension .esp
  - <<: *espError
    subs:
      - 'Starfield'
      - '.esp'
    condition: 'file("([^\.]+\.esp)")'

[Ortham]

I've filed Ortham/esplugin#42 for tracking how to resolve plugin FormIDs, which I think is easily the biggest blocker to properly supporting Starfield that I know of.

[pStyl3]

With the introduction of &unsupportedChangeVerified we now are able to assign a message to Starfield plugins, for when it is found that the plugin in question has changed it's master type - while the plugin is still "self-consistent".

While talking to Ortham about this, we came to the conclusion that another message is needed, for the case where a plugin adds new records that have FormIDs that don't match the plugin's flag. I would suggest the following non-global message:

  - &mismatchedFormIDs
    type: error
    content:
      - lang: en
        text: 'This plugin is a **{0}**, but it contains FormIDs that do not match this type of Master. Using this plugin would have unpredictable consequences for your game and is therefore not recommended.'
    subs:
      - 'Master Type'

{0} could be once again Full Master, Medium Master or Small Master. Notice that this time and in comparison to &unsupportedChangeVerified the message type is error and not warn.

[Ortham]

The starfield branch now has a build that should be functional, in that it should support medium plugins and sorting Starfield load orders, with the following caveats:

• There are 3 generated error messages which still refer to small plugins as light plugins when shown for Starfield.
• The icon used for medium plugins in plugin cards (a half-full glass) is a placeholder: I think the small and medium plugin icons will both need to be replaced to have a consistent visual metaphor. I've been referring to small/medium/full as a plugin's "scale" (e.g. full-scale, medium-scale, small-scale), since "size" seems too ambiguous (e.g. it could also mean the plugin file size), so icons that communicate that would be good. Since "light" will be the term used for other games, they wouldn't necessarily use the same icon as "small", but it could be good to replace the current icon with one that communicates "light" as in weight rather than illumination (e.g. a feather).
• The Flatpak build is failing due to unreleased dependencies, I'll fix this once everything is closer to being finalised.

Given how much has changed with Starfield relative to its initial release and to earlier games, I'm not super confident that I've understood all the relevant changes or correctly implemented that understanding, so if people could poke around and report any unexpected behaviour that would be great.

[pStyl3]

Using loot_0.22.4-57-g420723f_starfield-win64.7z here's a first screenshot:

On first glance there are no major hiccups, though it needs more testing. Sorting worked, but the Starfield masterlist needs to receive a number of updates.

The filter Hide Creation Club plugins is still getting displayed .. should we disable that filter for Starfield?

[Ortham]

The filter Hide Creation Club plugins is still getting displayed .. should we disable that filter for Starfield?

It's not hidden for games that don't have a Creation Club, but hiding it would make sense.

[Ortham]

I've just pushed another couple of commits that finish replacing light plugin with small plugin for Starfield, and also consistently use "full plugin" to refer to non-light, non-medium plugins in all games that support light plugins, as before LOOT inconsistently referred to them as "regular" or "normal", and in the absence of an official term for them in earlier games I think we might as well use "full".

[pStyl3]

The Nexus has introduced new modpage tags for Full/Medium/Small Masters, which can be used to quickly find, identify & test mods containing plugins for each type.

Plugin - Full Master Plugin - Medium Master Plugin - Small Master

[Ortham]

I've now released libloot v0.23.0 with support for Starfield's medium plugins and the game's current load order behaviour, and I've merged most of the corresponding LOOT changes to the master branch.

While I've got lots of tests and I couldn't spot anything obviously broken in manual testing, given how much has changed with Starfield, I'm not super confident that I haven't missed anything important. It might just be a case of releasing it and then refining it based on feedback from a wider pool of users, especially as it's still early days for the Starfield modding community.

Still to do:

• [x] Add an icon to the plugin cards for medium plugins - I've got a placeholder icon added in the medium-icon branch, I left it out of the merged changes because I still need to find/create a suitable icon
• [x] Replace the icon that's used for small plugins when run for Starfield
• [x] Replace the icon that's used for light plugins in games other than Starfield.
• [x] Add logging of the full path to Starfield.ccc that's used when it exists.
• [x] Update the external data paths support to only try to load Starfield plugins from My Games when they also exist in the game install path.
• [x] Hide the "Hide Creation Club plugins" filter for games that don't have the Creation Club (including Starfield). Don't tie it to the existence of the CCC file, as modding tools may create that themselves to help manage the load order.

If there's anything else you can think of, please let me know!

[Ortham]

I've created new icons for light, small and medium plugins. For light plugins, the icon is now a feather:

For small and medium plugins, I tried to go for boxes of varying size:

[pStyl3]

The feather icon is great, it suits the light term.

The new box icons on their own are good as well, but I kinda think that they look a bit too similar. Maybe something like this could be done?

[pStyl3]

@Silarn Does MO2 v2.5.2 dev5 automatically apply a Starfield.ccc file? Because if I run LOOT through MO2, my LOOTDebugLog.txt includes [06:14:59.141466] [debug]: Using CCC file at D:\Steam\steamapps\common\Starfield\Starfield.ccc

However, I do not have a Starfield.ccc in my Data root folder. And when I run LOOT by itself, LOOT doesn't log a Starfield.ccc file, as I would expect it.

[Silarn]

Yes, that was implemented in dev3, although it can be turned off.

• Implement automatic Starfield.ccc generation to fix base game plugin load order
  • Core game plugins currently load after plugins.txt unless required by another plugin
  • CCC file virtualized from profile directory, generated on refresh and when executable is run
  • Can be disabled in the Starfield plugin settings (but your plugin load order will probably be wrong)

[Ortham]

The new box icons on their own are good as well, but I kinda think that they look a bit too similar. Maybe something like this could be done?

What do you think of this?

or with lighter shading:

[pStyl3]

or with lighter shading:

I would say this version looks fine.

[pStyl3]

If MO2 automatically applies Starfield.ccc then we should at least discuss the difference in LOOTs sorting result.

Because for me LOOT sorts the base plugins, if run by itself. like this:

That's with the following metadata in place through the masterlist:

  - name: 'Starfield.esm'
    group: *mainGroup
  - name: 'Constellation.esm'
    group: *mainGroup
  - name: 'OldMars.esm'
    group: *mainGroup
  - name: 'BlueprintShips-Starfield.esm'
    group: *mainGroup
  - name: 'SFBGS003.esm'
    group: *mainGroup
  - name: 'SFBGS006.esm'
    group: *mainGroup
  - name: 'SFBGS007.esm'
    group: *mainGroup
  - name: 'SFBGS008.esm'
    group: *mainGroup

mainGroup is the earliest loading group, even before default.

If LOOT is run through MO, the base plugins are sorted like this:

[Ortham]

I want to retest the order that the official plugins load in by default, but FWIW while I don't think it actually matters, the second order looks more "correct" to me.

[Ortham]

The fact that Starfield will remove the official plugins from plugins.txt is a problem though, as it means LOOT can't actually set their load order for more than one game load. As I see it, there are two options:

• Take the approach that MO2 has and write the load order for those plugins to Starfield.ccc, though that can only be done if I add logic to LOOT that forces them to always load before all other plugins LOOT already has sorting logic to handle early-loading plugins, which includes plugins in CCC files. It would need to have logic added to write the official plugins to Starfield.ccc in a fixed load order and then reload the load order.
• Write the load order for those plugins by changing their timestamps, though that means they load after all plugins in plugins.txt unless otherwise hoisted, which would need changes to LOOT's sorting logic.

I think the first approach makes more sense given that despite the game's default behaviour these plugins should probably load early, not late.

@Silarn if LOOT is run through MO2 and writes to My Games\Starfield\Starfield.ccc, what effect would that have when taking into account the filesystem virtualisation?