Internal Formids above index 02 get clamped to use index 02 in memory

[robertgk2017]

The CK for some reason clamps form ids above 02 to use 02, which causes errors because the record doesn't exist at that index. On-save since references to these records are pointing at the wrong form id, it gets deleted potentially destroying plugins.

Simple example. create a dummy plugin with all 5 Main Vanilla masters, as masters. Copy as override the DLC2Solstehim worldspace from Dragonborn.esm into that dummy plugin, then load up the CK with that dummy plugin as active file.

You should see the following line in the log, among probably others

FORMS: Unable to find master location (02016E2A) on worldspace 'DLC2SolstheimWorld' (04000800).

The XLCN - Location tag for DLC2SolstheimWorld is actually at 04016E2A, but in memory the reference to that record within your dummy plugin is being clamped to use 02 at the beginning. Since that Location record doesn't exist there, the CK errors out on save, and if you look at the plugin in xEdit after hitting save, you'll see the XLCN - Location tag has been deleted. Also the 2 Water fields get reverted to the "DefaultWater" ones since those are required fields. I think I mentioned this to you in the Nexus comments a while ago, but this one is a serious one that if at all possible should be looked at.

This is likely due to Version Control limiting the .ESM your merging into, to having only 2 masters. However, this clamping still happens when Version Control is disabled. Ideally this should be disabled completely with VC both On and Off.

Edit: The CC Fishing .esm has this problem in it directly from Bethesda so you know this is an issue for them too, though they don't seem to notice,

[Perchik71]

The high-order bits of the FormID are cut off when loading, and their loading order is taken into account. I have already touched on that part of the code so that the old editors can read it. I'll take a look at this.

[robertgk2017]

Wait no it's not taking into account Load Order. the issue is ALL plugins above 02, have references to records in master slot 03 or higher put at 02.

Meaning if you had 50 plugins loaded, and all 49 non-active file plugins we're masters of the active file, then references in the active file to ALL of the plugins from 4-49 will be set to use index 02. load order is destructively ignored.

Skyrim.esm 00 Update.esm 01 Dawnguard.esm 02 Hearthfires.esm 03 Dragonborn.esm 04 CC Fishing 05 CC Survival 06 CC Curios 07 CC Saints 08 USSEP 09 MyMod.esm 0A

If MyMod.esm contains records referencing anything from Hearthfires.esm to USSEP, in memory within MyMod.esm the CK clamps references to those records to use index 02 regardless of the fact that it's the wrong index.

Meaning if USSEP defined a new record, the internal form id for a reference to that record would start with 09 inside MyMod.esm. The CK will overwrite that with 02. Since that record doesn't exist at 02, the CK will delete references to it from MyMod.esm on save.

[robertgk2017]

Example of this just happened on a mod im working on. On save it nuked this CELLs Location, Ownership, and Lock List tags, because it clamped the reference to them to index 02, which those records do not exist at.

[Perchik71]

Okay. I'll try to fix it as soon as I get out.

[robertgk2017]

Hold up, the latest build now has an odd quirk on the record reference ids. The form id of the "Master Location" is 02016E2A not 03016E2A. Its showing 03 as if its being defined by the active file, which is not correct.

make an override of DLC2SolstheimWorld in some dummy plugin "MyMod.esp"

give it the following masters

Skyim.esm 00 Update.esm 01 Dragonborn.esm 02 a new record in MyMod.esp would have 03. Except if you have a reference to a record in Dragonborn.esm it needs to stay as 02, not use the active files index of 03.

The current build is treating the overrides XLCN - Location tag as if its a reference to a new record, hence the 03, when it in fact is not a new record

[Perchik71]

I have done absolutely nothing for Skyrim in the last 10 days. Nothing has changed, just a strange behavior that needs to be tested yourself.

[robertgk2017]

Yea this issue is hard to pin down. i'll keep an eye out and put any other information/conclusions here. More information the better.

[Perchik71]

Hmm, well, I took a quick look to be approximately aware of what to do. I didn't like that there is a form with an obvious error in IDs. And I decided to look for her. And do you know what I found in the USSEP? It seems that the authors of USSEP do not use CK for their mod...

[Perchik71]

If you open all the DLC, then DLC 2 will be at index 4. If you open only all DLC in CK. There is no such error. And there are no mistakes at all.... Although he is also behind the index 4.

[Perchik71]

The mistake is that he can't find the parent. I thought about it and decided to take a look at what's in the files... I'll try to fix it by 4 by 2, which will be.

[Perchik71]

The error has disappeared. Edit: xEdit It seems to change the highest index at the parent location and comparing CK cannot be found. Edit2: It looks like a ton of errors are due to the fact that the USSEP plugin has corrupted indexes and CK is lost in the search. Edit3: From this we conclude that the problem is not in the CK. And not some kind of mystical limitation of 2 or 3 masters (FO4). This is incorrect indexing. CK does not know anything at this stage, and the tree is only being built, and it needs to be in the form as it was in the target file.

Edit4:

MASTERFILE: Unable to find editor location (0200520A) for NPC 'BYOHCarriageDriver3' (03010FBA) for Location 'BYOHHouse3Location' (0300520A)
MASTERFILE: Unable to find editor location (0200BE1D) for NPC 'BYOHHouse3Bard' (03019631) for Location 'BYOHHouse3Location' (0300520A)
MASTERFILE: Unable to find editor location (02005209) for NPC 'BYOHCarriageDriver2' (0300BE12) for Location 'BYOHHouse2Location' (03005209)
MASTERFILE: Unable to find editor location (0200BE0A) for NPC 'BYOHHouse2Bard' (03019630) for Location 'BYOHHouse2Location' (03005209)
MASTERFILE: Unable to find editor location (02035BF3) for NPC 'DLC2EbonyWarrior' (040285C3) for Location 'DLC2POIFallForest28Location' (04035BF3)
MASTERFILE: Unable to find master file location 0202659F. Master location data will be removed.
MASTERFILE: Unable to find master file location 02005209. Master location data will be removed.
MASTERFILE: Unable to find master file location 0202659F. Master location data will be removed.

And the like, it looks like it needs to be fixed.

Edit5: So that you understand, the index in the file is set solely on its dependencies, and Dragonborn depends only on 2 and its native index 2. It does not need to be changed to 4. He does not personally have so many dependencies.

[Perchik71]

I'll close it for now, because it's not CK problem. And how to describe the search algorithm for two different indexes, I can't imagine yet.

[Perchik71]

Although I'm still going to open it, I haven't tried saving it in the CK itself yet. Later.

[Perchik71]

CK saves correctly. In general, I will leave it open for discussion for now.

[robertgk2017]

The error has disappeared. Edit: xEdit It seems to change the highest index at the parent location and comparing CK cannot be found. Edit2: It looks like a ton of errors are due to the fact that the USSEP plugin has corrupted indexes and CK is lost in the search. Edit3: From this we conclude that the problem is not in the CK. And not some kind of mystical limitation of 2 or 3 masters (FO4). This is incorrect indexing. CK does not know anything at this stage, and the tree is only being built, and it needs to be in the form as it was in the target file.

Edit4:

MASTERFILE: Unable to find editor location (0200520A) for NPC 'BYOHCarriageDriver3' (03010FBA) for Location 'BYOHHouse3Location' (0300520A)
MASTERFILE: Unable to find editor location (0200BE1D) for NPC 'BYOHHouse3Bard' (03019631) for Location 'BYOHHouse3Location' (0300520A)
MASTERFILE: Unable to find editor location (02005209) for NPC 'BYOHCarriageDriver2' (0300BE12) for Location 'BYOHHouse2Location' (03005209)
MASTERFILE: Unable to find editor location (0200BE0A) for NPC 'BYOHHouse2Bard' (03019630) for Location 'BYOHHouse2Location' (03005209)
MASTERFILE: Unable to find editor location (02035BF3) for NPC 'DLC2EbonyWarrior' (040285C3) for Location 'DLC2POIFallForest28Location' (04035BF3)
MASTERFILE: Unable to find master file location 0202659F. Master location data will be removed.
MASTERFILE: Unable to find master file location 02005209. Master location data will be removed.
MASTERFILE: Unable to find master file location 0202659F. Master location data will be removed.

And the like, it looks like it needs to be fixed.

Edit5: So that you understand, the index in the file is set solely on its dependencies, and Dragonborn depends only on 2 and its native index 2. It does not need to be changed to 4. He does not personally have so many dependencies.

Yes I'm aware of internal form IDs being based on that specific plugins masterlist.

The problem is the CK won't accept internal form IDs above 02. And it says why that's the case "does your master file depend on more than 2 masters?"

The internal IDs for Dragonborn are 02 but the load order Resolved IDs are 04. It clamps any internal IDs to 02 regardless of what they are or what plugin they are from.

And when references to a record that has been clamped existing they are stripped on save because the CK is erroring out as they don't exist at the Load Order Adjusted ID that the clamped Internal ID suggests it does.

I can demonstrate this result with 100% certainty and repeatability

Edit: look at CC Fishings override of the Solstheim worldspace from Dragonborn. It has its Location, and Music Tags stripped as a direct result of this.

xEdit always resolves Internal IDs to Load Order Adjusted IDs correctly. It's the CK that has this problem of not allowing anything above 02

[robertgk2017]

Hmm, well, I took a quick look to be approximately aware of what to do. I didn't like that there is a form with an obvious error in IDs. And I decided to look for her. And do you know what I found in the USSEP? It seems that the authors of USSEP do not use CK for their mod...

Ah that second screenshot is exactly what I'm talking about. Except you have it backwards. USSEP specifically DOES use the CK

The error happens from Internal IDs above 02. Not Load Order Adjusted IDs. So only plugins with more than 3 masters will have errors. And those errors will specifically be on references in the plugin to records from the 4th or higher master of the plugin.

[robertgk2017]

Also, xEdit writes the Internal Form ID in the actual file. What you are seeing in the program visually is those internal IDs resolved to Load Order Adjusted IDs exactly as the game does.

Edit 1: Files must be physically written with internal IDs based on the files masterlist, with New Records having an internal Index 1 higher than the highest master. Regardless of where the plugin is in the Load order.

And all files must resolve those internal IDs to Load Order Adjusted IDs in memory for showing to the user, based on its position in Load Order.

So a plugin at Load Order spot 16, that has 2 Masters say... Skyrim.esm and Update.esm. the Internal IDs for new records MUST use index 02.

And when showing new records from that plugin in the CK, they MUST show index 0F as that's the load order resolved index of the plugin, if it were at position 16 for example.

Edit 2: This must always be true in all conceivable circumstances up to the max of 256 plugins being loaded in the CK at once. Although the CK will crash and burn for other reasons if you try to actually load that many plugins at once.

Currently the CK won't accept internal ID index's above 02. Which is a problem once the 2 indexes are understood

[robertgk2017]

I got a better idea do you have a discord server by chance? Would be easier to communicate and explain things as opposed to writing up long comment threads here.

[Perchik71]

I am in Discord with the same nickname, I tried to open a fishing mod. huh, there's a problem again. Then there is the problem of how to find out what this index should be behind it when it is not yet known. Well, at least there is something to start testing.

[robertgk2017]

The index should be immediately "know-able" in all circumstances. The internal IDs are set by a given plugins masters, and the Load Order IDs you actually see in the CK/xEdit/In game are set by the Defining plugins Load order Position.

The way xEdit does this perfectly matches the way the game runs which is as follows

[Plugin Name] - [Internal ID for New Records] - [Load Order Adjusted ID for New Records]

Skyrim.esm - 00 - 00 Update.esm - 01 - 01 Dawnguard.esm - 02 - 02 Hearthfires.esm - 02 - 03 Dragonborn.esm - 02 - 04

All of the CC plugins have all 5 of the above as masters so the Internal IDs for New records in them would be 05 for all of them individually. With the Load Order Adjusted IDs Adjusting to match the given CCs load order position at large. So, assuming you have all 74 CC plugins active it'd show the following.

ccasvsse001-almsivi.esm - 05 - 05 ccbgssse001-fish.esm - 05 - 06 ccbgssse002-exoticarrows.esl - 05 - 07 ccbgssse003-zombies.esl - 05 - 08 ccbgssse004-ruinsedge.esl - 05 - 09 ccbgssse005-goldbrand.esl - 05 - 0A ccbgssse006-stendarshammer.esl - 05 - 0B ccbgssse007-chrysamere.esl - 05 - 0C ccbgssse010-petdwarvenarmoredmudcrab.esl - 05 - 0D ccbgssse011-hrsarmrelvn.esl - 05 - 0E ccbgssse012-hrsarmrstl.esl - 05 - 0F ccbgssse014-spellpack01.esl - 05 - 10 ccbgssse019-staffofsheogorath.esl - 05 - 11 ccbgssse020-graycowl.esl - 05 - 12 ccbgssse021-lordsmail.esl - 05 - 13 ccmtysse001-knightsofthenine.esl - 05 - 14 ccqdrsse001-survivalmode.esl - 05 - 15 cctwbsse001-puzzledungeon.esm - 05 - 15 cceejsse001-hstead.esm - 05 - 16 ccqdrsse002-firewood.esl - 05 - 17 ccbgssse018-shadowrend.esl - 05 - 18 ccbgssse035-petnhound.esl - 05 - 19 ccfsvsse001-backpacks.esl - 05 - 1A cceejsse002-tower.esl - 05 - 1B ccedhsse001-norjewel.esl - 05 - 1C ccvsvsse002-pets.esl - 05 - 1D ccbgssse037-curios.esl - 05 - 1E ccbgssse034-mntuni.esl - 05 - 1F ccbgssse045-hasedoki.esl - 05 - 20 ccbgssse008-wraithguard.esl - 05 - 21 ccbgssse036-petbwolf.esl - 05 - 22 ccffbsse001-imperialdragon.esl - 05 - 23 ccmtysse002-ve.esl - 05 - 24 ccbgssse043-crosselv.esl - 05 - 25 ccvsvsse001-winter.esl - 05 - 26 cceejsse003-hollow.esl - 05 - 27 ccbgssse016-umbra.esm - 05 - 28 ccbgssse031-advcyrus.esm - 05 - 29 ccbgssse038-bowofshadows.esl - 05 - 2A ccbgssse040-advobgobs.esl - 05 - 2B ccbgssse050-ba_daedric.esl - 05 - 2C ccbgssse052-ba_iron.esl - 05 - 2D ccbgssse054-ba_orcish.esl - 05 - 2E ccbgssse058-ba_steel.esl - 05 - 2F ccbgssse059-ba_dragonplate.esl - 05 - 30 ccbgssse061-ba_dwarven.esl - 05 - 31 ccpewsse002-armsofchaos.esl - 05 - 32 ccbgssse041-netchleather.esl - 05 - 33 ccedhsse002-splkntset.esl - 05 - 34 ccbgssse064-ba_elven.esl - 05 - 35 ccbgssse063-ba_ebony.esl - 05 - 36 ccbgssse062-ba_dwarvenmail.esl - 05 - 37 ccbgssse060-ba_dragonscale.esl - 05 - 38 ccbgssse056-ba_silver.esl - 05 - 39 ccbgssse055-ba_orcishscaled.esl - 05 - 3A ccbgssse053-ba_leather.esl - 05 - 3B ccbgssse051-ba_daedricmail.esl - 05 - 3C ccbgssse057-ba_stalhrim.esl - 05 - 3D ccbgssse066-staves.esl - 05 - 3E ccbgssse067-daedinv.esm - 05 - 3F ccbgssse068-bloodfall.esl - 05 - 40 ccbgssse069-contest.esl - 05 - 41 ccvsvsse003-necroarts.esl - 05 - 42 ccvsvsse004-beafarmer.esl - 05 - 43 ccbgssse025-advdsgs.esm - 05 - 44 ccffbsse002-crossbowpack.esl - 05 - 45 ccbgssse013-dawnfang.esl - 05 - 46 ccrmssse001-necrohouse.esl - 05 - 47 ccedhsse003-redguard.esl - 05 - 48 cceejsse004-hall.esl - 05 - 49 cceejsse005-cave.esm - 05 - 4A cckrtsse001_altar.esl - 05 - 4B cccbhsse001-gaunt.esl - 05 - 4C ccafdsse001-dwesanctuary.esm - 05 - 4D

The new _ResourcePack.esl only has Skyrim.esm, Update.esm, and Hearthfires.esm as masters, so its Internal/Load order IDs for NEW records would be the following, assuming all 74 CCs and the 5 main masters are active

_ResourcePack.esl - 03 - 4E

USSEP.esp has Skyrim.esm, Update.esm, Dawnguard, Hearthfires, Dragonborn, CC Fishing, CC Survival, CC Curios, and CC Saints and Seducers as masters. Its Internal/Load order IDs for NEW records would be as follows, assuming all of the above is active

Unofficial Skyrim Special Edition Patch.esp - 09 - 4E

and so on and so forth for the entire load order.

Edit: I sent you a friend request on Discord, same username robertgk2017.

[Perchik71]

Fixed. There are no Bethesda fixes for 1.6.438 and there are bugs in loading quests. There is no patch for 1.5.XX. source