Converting the Fallout New Vegas masterlist

[kymckay]

You can find the main issue on masterlist conversion here.

The metadata syntax document can be viewed here.

Remember to claim a section before starting work on it.

What Needs Doing

These groups need to be converted (checked means claimed, scored out means completed):

• [x] ESMs
• [x] Earliest
• [x] Unofficial Patches and Fixes
• [x] Operation Fixed Terrain
• [x] Early Loading Mods
• [x] Alternative starts
• [x] Faces
• [x] NPC
• [x] Places
• [x] New Vegas Restoration
• [x] BlackCompany HarderCore Pack
• [x] Fallout New Vegas Advanced & HAM-Hardcore Advanced Mods
• [x] FOOK
• [x] New Vegas Replayability
• [x] Frontsalats Modular Mods and Tweaks
• [x] Book Of Earache
• [x] NEVAX - New Vegas Expanded (German)
• [x] Puce Moose Tweak and Balance Center
• [x] Reload
• [x] Superior Firepower
• [x] XFO NV - The New Vegas Overhaul
• [x] Other
• [x] Sounds
• [x] Quests
• [x] Perks
• [x] Early Misc
• [x] Crafting
• [x] Weapons
• [x] Armor and Clothes
• [x] Cosmetics
• [x] Middle Misc
• [x] Companions
• [x] Later Misc
• [x] Game-Settings and Overrides
• [x] Weapon Mods Expanded or WMX
• [x] Weapon Mod Expansion or WME
• [x] Special Cases for WMX and/or WME support
• [x] NV Game Stabilizer
• [x] Weather\Light\Water
• [x] Lastish
• [x] nVamp
• [x] Pose mods
• [x] Beauty Packs & Race Addons
• [x] Hair
• [x] Merged Patch
• [x] ModFusion
• [x] Bashed Patch
• [x] Post Bashed Patch

[kymckay]

Group lines added, scored out some groups that were not initially added (should still check for conditional placements)

[egocarib]

I noticed something peculiar as I was editing Early Misc - there were about 8 versions of the Mail Order Catalogs esp, but only the first three said they required NVSE in the masterlist.yaml

I double-checked the original masterlist, and all the versions say they require it there. This got me curious so I poked around a bit and I noticed just a few entries above the Mail Order esps, there was this entry in the original masterlist:

Isul-MagExchange.esp IFNOT VAR(NVSE) REQ: New Vegas Script Extender

but this doesn't show up in the new .yaml masterlist. Maybe this has already been accounted for somehow and I just don't know exactly what's needed in the conversion, but I wanted to mention it in case this is a bug or something and requirements are being accidentally left out of the new updated list.

~egocarib

[Ortham]

I wanted to mention it in case this is a bug or something and requirements are being accidentally left out of the new updated list.

Isul-MagExchange.esp shows up on line 4948 for me.

The other req messages are getting skipped because they contain a plugin filename, and @SilentSpike recommended that I make the converter do this since he found that pretty much all of the plugin filenames given were masters, and so didn't need to be kept, when doing the FO3 masterlist.

[egocarib]

Okay, good to know. I don't know how I missed the entry you point out there - sorry for any confusion.

Did I not need to add the [ *NVSE ] req to those Mail Order esp entries then? Because I did.

[Ortham]

Did I not need to add the [ *NVSE ] req to those Mail Order esp entries then? Because I did.

Yes, they should still have the NVSE req added.

[egocarib]

I can't find a link anywhere that will take me somewhere I can learn about regex expressions. I think I need to use a regex at the beginning of the Armor and Clothes section here to account for any number of 20+ plugins that all start with "nVamp" so that I can properly conditionalize a message, but I don't know how to set that up. (Looking over the existing regex in the masterlists hasn't clarified things either for me. Any help is appreciated)

[Ortham]

I can't find a link anywhere that will take me somewhere I can learn about regex expressions.

This page covers all the details of the regex supported by BOSS. For a higher-level, more general overview, try here.

[egocarib]

Okay, thanks. I think this should work then, but correct me if I'm wrong:

condition: "regex(\"nVamp.*\.esp\")"

edit: fixed syntax, mostly just wanted to figure out how to offset code/characters like the above so they wont be parsed.

[kymckay]

If you have any specific questions about regex. Drop them in this area and we can answer them!

Heh, you actually did while I was typing this. Firstly I'd drop the quotations so that you don't need to escape the inner ones, you'll want to use \.esp so that the . matches itself and not its special character (wildcard).

As for the .*: this will work, but it could potentially be too vague for the purposes of your condition. I suggest leaving it that way for now and I'll take a look at it once you make the commit.

All of that leaves you with:

regex("nVamp.*\.esp")

[egocarib]

Hmm, looks like it messed up my syntax in that post, it actually looks a bit different (edit: now fixed). I'll let you take a look at the file. I just committed it. I haven't crossed it off the list yet as I'm uncertain whether this Armor section still needs refinement.

However, your post makes me wonder - so I don't need quotations in the conditions: sections then? I've been using them all along, heh. It did seem odd to me to use them for conditions but not for msg content. But I was mainly going off what I saw looking at other people's edits in the previous fallout list.

Also, I ran into one other issue in the Armor section. A lot of mods require the "Type 3 Body" - but since that is only a mesh/texture replacer, I don't know how to check for it. I've just left them as messages for now. But feel free to conditionalize them or set up requirements if there's some way to do it.

[kymckay]

Could potentially search for the mesh/texture files with matching checksums. But simply messages will be fine too :smiley:

You only need the quotes for strings which contain -?:,[]{}#&*!|>"%@ or `, but as far as I know this isn't a problem for conditions as the strings are inside the actual condition's quote marks. (That's an easy way of thinking about it at least)

[kymckay]

I did almost all of the converting for the fallout 3 list with the syntax doc open in an adjecent window, just now am I getting the hang of the new YAML syntax.

I'll take a look at the file now!

[Ortham]

As far as quoting things goes, it's always safer to quote them, but it's not always necessary. It depends on things like spacing and context, but it's easy to check if something should be quoted by copy/pasting it here and seeing if what is generated is what you want.

[kymckay]

I just used inc: and put the main nVamp file in there as it has the same purpose as the message did.

[kymckay]

Added a display too, since it makes more sense that way

[kymckay]

Although...I suppose the user wouldn't know it had been merged into the mod this way, I'll revert the message with a file condition since it's probably a bad idea to presume users have read the description :stuck_out_tongue_closed_eyes:

[egocarib]

Also, a general question: What are the purpose of tags? I haven't touched any of the tag lists as I've been editing, but I'm just curious what they are used for.

[egocarib]

I've finished everything else. I'll leave the Armor section unchecked as it sounds like Silent Spike wants to revert something there yet.

I am going to leave putting up the next list (Oblivion, right?) to someone else, as I haven't used the masterlist converter and I'm still getting adjusted to the gitHub wiki. But I'll get to work once it's posted.

[Ortham]

Also, a general question: What are the purpose of tags? I haven't touched any of the tag lists as I've been editing, but I'm just curious what they are used for.

They're used by Wrye Bash when building a Bashed Patch.

[kymckay]

Yeah tags can be a bit daunting at first. Building a bashed patch and understanding what it does in general terms is a good starting point. I found that I had to actually understand what was going on at a sub-plugin level before I was fully comfortable with them.

Though I don't actually know that much about all the bashed tags that aren't supported in Skyrim. Their name is usually a good indication of what they do though.

[kymckay]

Seems like everything is done now. Really nice work ego, wish I could have helped out a bit more on this one. :thumbsup:

[kymckay]

Ran the file through the online parser and it doesn't seem to like the double quoted regex on line 1818. Will simply remove those quotes.

[kymckay]

Okay, getting a bad conversion error from BOSS. Going to sift through the list to try and find out what the problem is.

[Ortham]

Okay, getting a bad conversion error from BOSS. Going to sift through the list to try and find out what the problem is.

That reminds me, I've been meaning to make those errors a bit less useless.

[kymckay]

I'm wondering if it might be the quotes around the conditions that are causing it

[kymckay]

Oh, would it be possible to have those errors display one more character of the revision's SHA?

That way it'd match up perfectly with the short one displayed on github.

[Ortham]

Oh, would it be possible to have those errors display one more character of the revision's SHA?

Yes, but IIRC GitHub actually looks at all the SHAs in a repository and calculates for each one what the shortest unique substring is (from the beginning of the SHAs), then displays all the SHAs to the same length as the longest substring calculated. So if I increased the characters displayed by BOSS by one, that wouldn't necessarily match what GitHub displays at a later date.

I'm pretty sure I could do the same in BOSS, but I figured it's unnecessary since it'll be either the latest masterlist or a fairly recent one, even if the string displayed isn't unique, it shouldn't be difficult to figure out which it's referring to.

[Ortham]

The v2 masterlist has some conditional plugin positions which should be looked at to see if they can be converted to "load after" metadata - use the regex ^IF(NOT)?.+MOD: to find them.

Once that's done, this can get marked as complete, and we can move on to Oblivion. I've started to look through the YAML parser's code to see how I can improve its speed, and implement support for merge keys. I've got it reporting the correct line number and position of errors, thanks to a patch someone posted in the original code repository, but the rest will take some time, I think.

[kymckay]

I was originally - and still am - a little confused by some of these conditional placements in the ESM group.

They are placed based on nVamp - Core.esm, except half of them are still loaded before it whether it's there or not.

[Ortham]

Well, the Meltdown Medley.esp condition is wrong, there's a patch for that which should be used instead, so I'm not going to convert the condition.

As for the Nevada Skies conditions, it seems like they're just there to ensure that overhauls have their changes override the weather mod's. However, this may be handled by BOSS without the need for "load after" metadata, so I think we should leave them too, especially considering that the current conditions aren't exactly comprehensive.

I don't understand the nVamp stuff either, I say we just leave it.

[TsunaMoo]

The nvskies.esp conditions are mostly because this file by default ahs some dirty landscape/nav edits. If the file is cleaned most of the conditions requiring nvskies can be neglected.

[TsunaMoo]

here is something i found: we have 3 types of data here as example i got this: say: "i like text" say: 'i like text' say: i like text

so which one is correct? We have things like this in a lot and not limited to say its all over the place.

[Ortham]

so which one is correct?

They all are: see the second set of bullet points here. Strings don't have to be quoted in YAML unless they contain special characters, and even then it depends on context. (Basically, if the string is ambiguous to the parser, then it needs to be quoted to show it's a string and not a list or a map or anything else.)

[TsunaMoo]

isnt that kind of messy?

[Ortham]

You mean having 3 different styles? We could agree to stick to one (double quoting everything), but then that means a lot of escaping quotes and backslashes.

[kymckay]

We could single quotes all the time, then doubles whenever there are singles in the string itself.

[Freso]

I think it makes sense to always do quotes, thus narrow it down to two styles. If there's no performance difference, I think double quotes look better, so I'd recommend something like:

For strings, enclose them in double quotes (") unless the string itself contains double quotes, in which case use single quotes ('). If the string contains both, use whichever looks best or feels more natural. E.g.: "Khajiit, like Argonians, make good thieves." and 'There are rumored to be "more than twenty" forms of Khajiit.', but also 'The title "khajiit" is derived from the Ta\'agra words "khaj" and "-iit"'.

(Sorry about referencing the wrong game, but I've yet to play any FO games, so I don't know any of the FO lore. :p)

[Ortham]

Actually, I misread the spec, and it looks like you can single-quote strings with single quotes in them, but you need to repeat any of the single quotes in the string, eg.

This string's not twine.

becomes

'This string''s not twine.'

The limitation of single-quoted strings as opposed to double quoted strings is that you don't get escaping for non-printable characters (like newlines). However, I don't see why we'd ever be using non-printable characters (unless some mod author was a complete ass and used them in a filename).

So that means that we could choose to use either only single quotes or only double quotes. I vote single quotes, because double quotes appear more frequently in the masterlist strings, and there's less possible syntax confusion over escaping characters (which I've already had to correct for before).

As for the existing unquoted and double-quoted strings, I could tweak the converter to single-quote everything, and also to input the existing YAML masterlists and output them again with single-quotes.

[kymckay]

Single quotes sounds good to me too :+1:

[Ortham]

Actually, I can't auto-convert the quotation style, because if I specify a certain style, then it applies the same style to the keys of maps, eg.

- 'name': 'Foo.esp'
  'req': [ 'Bar.esp' ]

Which is pointless since all the keys are single-word strings, and it ends up just looking really messy. I could get around this by explicitly turning off any quotation for each key individually, but it's not really worth it.

[kymckay]

We could just slowly replace them as we work on the lists. It's not an urgent thing

[egocarib]

Noted.