Converting v2 masterlists to v3

[Ortham]

What Needs Doing

I have written a masterlist converter can be used to partially convert the v2 masterlists. The conversion is partial because some data gets skipped and must be double-checked, and some data must be adapted into suitable forms for the v3 masterlists.

The following get skipped but might contain useful info, so should be checked manually. Regex search strings are provided to find them, make sure to search case-insensitively:

• Requirement messages containing .esp or .esm. Just about all of these are just the plugins' masters, but there might be other info in the message worth retaining. Search regex: REQ:.+(\.esp|\.esm).
• Plugin conditions. These conditions should be turned into "load after" metadata. Search regex: ^IF(NOT)?.+MOD:.
• Silent comments. These may contain useful notes on nearby plugins. Search regex: ^(/\*|//).

The following get partially converted, but need adapting:

• Requirement and incompatibility messages get turned into note messages. These must then be manually converted to the new formats for requirements and incompatibilities.
• The automatic converter will keep translated sets of a message as separate individual messages, so these must be converted into content lists manually.

In addition, while not necessary for correct functionality, the following tasks make the masterlists easier to maintain:

• The automatic converter converts conditions in such a way that there will be lots of duplicate conditions where variable were checks used previously. It would be good to go through and create a set of anchors and references for these conditions. The references could go in a top-level common node.
• Similarly, messages which use the same text could be replaced by a string in the common node with an anchor that's referenced where the message text is used.

  Organisation

I think it's best if everybody works on the same masterlist, with each person claiming a group to convert before they start working on it. Then whenever a group gets finished, they can move on to the next unclaimed group.

v3 masterlists don't have group syntax. That means that someone first needs to go through and re-add the group beginnings and endings to the converted masterlist. All it requires is going through each of the group lines (ie. find all "BEGINGROUP", then find all "ENDGROUP") and looking to see what the first plugins with messages above and below it are, then looking for those in the converted masterlist and adding the line between them. When adding the group lines, I'd suggest just sticking a hash (pound) in front of the v2 syntax to comment them out, so # BEGINGROUP: FOO and # ENDGROUP: FOO.

This breaks it down into nice manageable chunks, and means that we get whole masterlists finished quicker than if everyone was working on different masterlists, and also means that any improvements to editing "style" can get propagated to later masterlists.

I think that it's probably best to work through converting the masterlists starting with the smaller and less frequently updated ones, so an order of:

1. Fallout 3
2. Fallout: New Vegas
3. Oblivion
4. Skyrim

I've already converted the Nehrim one, and I want to merge it in with the Oblivion masterlist once it's converted, since there is just a few messages difference and most of the plugins are already in the Oblivion masterlist.

This issue will be the overall tracker, but each masterlist should track its individual progress on the changes highlighted above as separate issues in it's own repository tracker.

[Sharlikran]

I read what is above and I am reading the 'BOSS Metadata Syntax.html' so I see what has to be done now. Thanks.

[Ortham]

Do I understand that correctly?

yes

[Sharlikran]

Okay there is a problem with the Beta. I can't figure out how to force the "Bashed Patch, 0.esp" to load last after all the other plugins. I can say, after "pluginx.esp" for myself, but nobody else will have the same plugins as I do. What do I do?

[Ortham]

Okay there is a problem with the Beta. I can't figure out how to force the "Bashed Patch, 0.esp" to load last after all the other plugins. I can say, after "pluginx.esp" for myself, but nobody else will have the same plugins as I do. What do I do?

Use the priority field.

[Ortham]

One thing @Sharlikran just brought up:

If you've got a single message that you want to apply to lots of different plugins (eg. "use only one of XXX"), it's better to use a regex entry than a separate entry for each plugin.

Related to that is that in v3, if a plugin has more than one matching entry, they get merged, rather than just one picked. Not everything can be merged, but it's useful for things like when you have a "use only one" message and also some plugin-specific metadata.

I'll be updating the syntax doc to include this info, but I thought I'd mention it here too.

[Ortham]

I'm currently going through the Oblivion lists and adding ITM counts to all the dirty metadata, which is something that the converter didn't do due to a bug.

Another few things that are related that also need doing are:

• Addition of wild edit messages. These were appended to the end of dirty messages in the v2 masterlists, so got cut out.
• Addition of CRCs in compound conditionals for dirty info. I think the converter only recorded the first CRC encountered.

There shouldn't be that many of them to do, so I'm happy to do that too. Skyrim will also need the additional CRCs added, unless someone else has already done that.

[Freso]

What needs to be done here? Can this issue be closed?

[Ortham]

Yes, this can be closed.

[Freso]

For the record, I just removed the section in the "Other Tasks" on https://github.com/loot/loot/wiki/LOOT-Beta-Testing about helping with converting the masterlist. I don't know if it's mentioned any other places too.