Converting v2 masterlists to v3

Ortham commented 11 years ago

What Needs Doing

I have written a masterlist converter can be used to partially convert the v2 masterlists. The conversion is partial because some data gets skipped and must be double-checked, and some data must be adapted into suitable forms for the v3 masterlists.

The following get skipped but might contain useful info, so should be checked manually. Regex search strings are provided to find them, make sure to search case-insensitively:

Requirement messages containing .esp or .esm. Just about all of these are just the plugins' masters, but there might be other info in the message worth retaining. Search regex: REQ:.+(\.esp|\.esm).
Plugin conditions. These conditions should be turned into "load after" metadata. Search regex: ^IF(NOT)?.+MOD:.
Silent comments. These may contain useful notes on nearby plugins. Search regex: ^(/\*|//).

The following get partially converted, but need adapting:

Requirement and incompatibility messages get turned into note messages. These must then be manually converted to the new formats for requirements and incompatibilities.
The automatic converter will keep translated sets of a message as separate individual messages, so these must be converted into content lists manually.

In addition, while not necessary for correct functionality, the following tasks make the masterlists easier to maintain:

The automatic converter converts conditions in such a way that there will be lots of duplicate conditions where variable were checks used previously. It would be good to go through and create a set of anchors and references for these conditions. The references could go in a top-level common node.
Similarly, messages which use the same text could be replaced by a string in the common node with an anchor that's referenced where the message text is used.
Organisation

I think it's best if everybody works on the same masterlist, with each person claiming a group to convert before they start working on it. Then whenever a group gets finished, they can move on to the next unclaimed group.

v3 masterlists don't have group syntax. That means that someone first needs to go through and re-add the group beginnings and endings to the converted masterlist. All it requires is going through each of the group lines (ie. find all "BEGINGROUP", then find all "ENDGROUP") and looking to see what the first plugins with messages above and below it are, then looking for those in the converted masterlist and adding the line between them. When adding the group lines, I'd suggest just sticking a hash (pound) in front of the v2 syntax to comment them out, so # BEGINGROUP: FOO and # ENDGROUP: FOO.

This breaks it down into nice manageable chunks, and means that we get whole masterlists finished quicker than if everyone was working on different masterlists, and also means that any improvements to editing "style" can get propagated to later masterlists.

I think that it's probably best to work through converting the masterlists starting with the smaller and less frequently updated ones, so an order of:

Fallout 3
Fallout: New Vegas
Oblivion
Skyrim

I've already converted the Nehrim one, and I want to merge it in with the Oblivion masterlist once it's converted, since there is just a few messages difference and most of the plugins are already in the Oblivion masterlist.

This issue will be the overall tracker, but each masterlist should track its individual progress on the changes highlighted above as separate issues in it's own repository tracker.

kymckay commented 11 years ago

All sounds good to me! I still need to get used to the new workflow here, should I (or whoever) open issues in the separate game repositories for their conversion?

Ortham commented 11 years ago

Sure, go ahead. Also feel free to have a play around with everything to get used to it - you can't really break anything important. Worst case I suppose would be that you wipe out the history for a repository, but then I can always recreate it using my local repositories.

kymckay commented 11 years ago

Need to expand on the issue a bit, not entirely sure how to set it out. Got the different sections listed for now :grinning:

Ortham commented 11 years ago

Looks good to me. Referring back here for the details is a good idea, less repetition that way. Maybe a note saying something about how when people start to add stuff to the common node and reference it, they should first check to make sure there isn't anything already there that's suitable?

Common sense, but so is "test your edits before you commit". :P

kymckay commented 11 years ago

BOSS will give requirement messages on missing explicit masters without extra metadata, right?

kymckay commented 11 years ago

Whoops, closed the issue by accident :P

Ortham commented 11 years ago

Yes, if lumps masters and requirements together when checking to see they exist. It doesn't cancel sorting though, for some reason. It did, but then I remember changing it for some reason, but can't remember why...

Ortham commented 11 years ago

I can't recall if I've mentioned this anywhere else, but v3 masterlists don't have group syntax. That means that since we're splitting the manual conversion up group-by-group, someone first needs to go through and re-add the group beginnings and endings to the converted masterlist. All it requires is going through each of the group lines (ie. find all "BEGINGROUP", then find all "ENDGROUP") and looking to see what the first plugins with messages above and below it are, then looking for those in the converted masterlist and adding the line between them.

When adding the group lines, I'd suggest just sticking a hash (pound) in front of the v2 syntax to comment them out, so # BEGINGROUP: FOO and # ENDGROUP: FOO.

I've added this info to the first post.

Ortham commented 11 years ago

POSSIBLE SYNTAX CHANGE

In v2, regular expressions use the POSIX Extended syntax, and this is currently the case in v3 too. However, Javascript uses a subset of the Perl syntax, so I'm going to have to change to that if the masterlist search is going to happen.

The good news is that I should be able to automatically convert any v2 regular expressions that need it in my converter utility.

kymckay commented 11 years ago

Sounds fine to me On 18 Sep 2013 13:09, "WrinklyNinja" notifications@github.com wrote:

POSSIBLE SYNTAX CHANGE

In v2, regular expressions use the POSIX Extended syntax, and this is currently the case in v3 too. However, Javascript uses a subset of the Perl syntax, so I'm going to have to change to that if the masterlist search is going to happen.

The good news is that I should be able to automatically convert any v2 regular expressions that need it in my converter utility.

— Reply to this email directly or view it on GitHubhttps://github.com/boss-developers/boss-developers.github.io/issues/4#issuecomment-24659345 .

Ortham commented 11 years ago

Well, I looked through all the masterlists' regex uses (SVN r7505) and found that none of them need any changes, so I won't be implementing automatic conversion (since I'd probably make mistakes).

Ortham commented 11 years ago

MASTERLIST CONVERTER BUG

Just discovered that I had the wrong files being substituted into the v2 placeholders for FOSE, NVSE and SKSE: the converter was putting fose_loader.dll, nvse_loader.dll and skse_loader.dll when they should have been fose_loader.exe, nvse_loader.exe and skse_loader.exe respectively.

I will be releasing a fixed converter, but for the Fallout 3 masterlist, a search and replace will be suffice.

kymckay commented 11 years ago

Excuse my last question, I noticed the answer straight after posting. Another for you:

Can we have conditions on things like requirements or incompatibilities? The syntax doc seems to only cover messages and tags with conditions.

EDIT: Why would anyone decide to put the "close and comment" button beside the "comment" button? :stuck_out_tongue_closed_eyes:

Ortham commented 11 years ago

Can we have conditions on things like requirements or incompatibilities? The syntax doc seems to only cover messages and tags with conditions.

Yes, requirements and incompatibilities are file objects, so just use the key-value map syntax given for them in the syntax doc. E.g.

plugins:
  - name: Foo.esp
    req:
      - name: Bar.esp
        condition: 'checksum("Foo.esp", AC2402DF)'

The above means "Foo.esp requires Bar.esp if Foo.esp has a CRC of AC2402DF".

kymckay commented 11 years ago

If I have:

display: "http://fose.silverlock.org/ FOSE"

Will it display as a link? Or does that only work in messages?

Ortham commented 11 years ago

Will it display as a link? Or does that only work in messages?

I'm not sure. Probably not, but it would be good to be able to include links like that, so I'll change it if needed. I can't remember anything about how/where file objects get outputted to the masterlist, so I'll probably have to go back through and check everything anyway.

kymckay commented 11 years ago

Okay, I'll leave it for now then. It would be handy in future, especially for something like FOSE where we can add an entry to the common node for it. Which I also need to ask about actually.

If I use this:

common:
  - &FOSE
    name: "..\fose_loader.exe"
    display: FOSE

Then the correct way to reference it is this?

    req: 
      - [ *FOSE ]

I was looking at the nehrim list to copy, but I'm not 100% sure that's right. Oh and is this the best place to be asking these questions? :stuck_out_tongue_closed_eyes:

Ortham commented 11 years ago

Actually, it would be better if you start using the link syntax in display strings wherever you want, I should have it implemented later today, and it means you won't need to go back and change anything.

Yes, that's the right way to reference it, and this is the best place to ask questions, since anyone else who converts the masterlist is most likely to see Q&As here.

kymckay commented 11 years ago

Okay, I'll do that.

One more question before I make my next commit:

Can regex be used in the file data structure? The syntax doc shows that it can in the plugin data structure, but there's no mention in the file data structure section. You (or any of us) may want to reflect whatever the answer is in the document also.

Ortham commented 11 years ago

No, regex can't be used in the file structure, because it's supposed to be specific. Unless you've though of a use I haven't?

kymckay commented 11 years ago

The one I'm thinking of right now is a plugin that has version number in it's file name and is incompatible across all versions.

Though perhaps it's best to enter them all separately just incase.

Ortham commented 11 years ago

Yeah, it's more verbose, but considering that a false positive could mess with how the plugins get sorted, it's better to be precise.

Also, stupid version numbers in plugin names...

Ortham commented 11 years ago

Turns out that display values in File objects can take URLs just fine, but that the display values were never actually being used. I've fixed that now, and I'll update the syntax doc soon with the various points brought up here so far. :)

kymckay commented 11 years ago

Been doing some more conversion, came across my first dirty message and realised that we're now using warning messages for those.

Maybe it's just me but I think having a unique message type (dirty) is a lot less alarming to the user - it also allows us to easily display a count of the dirty plugins they have. Is there a reason you've chosen not to include one? (I can't remember, but I feel like this was discussed a long time ago)

I could see it being something like:

  - name: example.esp
    dirt:
      - crc: 02933245
        itm: 24
        udr: 97
        nav: 2

The simple version would just be: crc

Which, if the CRC was met, could output something such as:

Contains 24 ITM records, 97 deleted records and 2 deleted navmeshes.

Or if no extra data was given simply:

Requires cleaning.

Obviously I'm not familiar with the back end of the program so I have no idea what the best way to do it would be. Just sharing my thought :stuck_out_tongue_closed_eyes:

Ortham commented 11 years ago

The reason is that I think it's more intuitive for all messages to adhere to a three-level system.

Bash Tag suggestions are the exception because they're more of a notification than a suggestion, and I'm thinking that they should be styled in the report so that they don't appear on the same "level" as messages (perhaps have them only displayed on hover or something...). (Really, I need to take another look at the styling of everything in the details tab.)

The dirty message count is misleading, since people can (and do) assume that anything not in the count is clean. I intentionally left it out in v3.

I'd be happy to have a dedicated structure for dirty metadata, since it would be more efficient to just store the metadata and have BOSS generate messages from it, but unfortunately there's more variation than just the four values you identified: there's also different cleaning guide URLs between games, and some plugins also have further manual cleaning instructions. The utility used also differs between games (yes, they're all just renames of TES5Edit, but users aren't generally aware of that). I don't want to hardcode either utility or guide URLs.

So that means we'd need a structure like:

dirt:
  - crc: <hex string>
    itm: <int>
    udr: <int>
    nav: <int>
    util: <labelled url>
    guide: <labelled url>

though we could define the last two in common and merged into each dirty object like so:

common:
  - &dirty
    util: <labelled url>
    guide: <labelled url>

plugins:
  - name: Foo.esp
    dirt:
      - <<: *dirty
        crc: <hex string>
        itm: <int>
        udr: <int>
        nav: <int>

or the object in shorthand:

dirt:
  - { <<: *dirty, crc: <hex string>, itm: <int>, udr: <int>, nav: <int> }

Further manual cleaning instructions would have to be split into a separate actual message, though I think that's for the best, since manual cleaning is quite separate from automated cleaning anyway.

I'd still like the dirty messages that would be generated to be displayed as warnings though.

@boss-developers: Thoughts?

kymckay commented 11 years ago

I see what you mean about the three level system, that does make sense. I also see your point with the clean plugin presumptions, that was an issue - though I can see users thinking the same thing about plugins that don't have warning messages.

As for the URLs, I did think about that myself and was curious as to whether we could put some kind of message in the global section that'd only appear if there were dirty messages. That way the plugin entries stay a bit cleaner and the URLs can still be changed at any time.

Ortham commented 11 years ago

As for the URLs, I did think about that myself and was curious as to whether we could put some kind of message in the global section that'd only appear if there were dirty messages. That way the plugin entries stay a bit cleaner and the URLs can still be changed at any time.

Could do, but that would require more coding, whereas using the common node and merging doesn't.

kymckay commented 11 years ago

Then yeah, common sounds fine to me :smiley:

kymckay commented 11 years ago

Some feedback on the auto-converter:

For requirement messages it'd save a lot of time if it'd ignore them if they had .esm/.esp as the majority of those are explicit masters and take time to remove :stuck_out_tongue_closed_eyes:

Perhaps we'd lose some data in the conversion process but I don't think it would be much effort to add back in future compared to the effort required to remove a lot of useless data.

kymckay commented 11 years ago

Fallout 3 masterlist has been converted fully.

Ortham commented 11 years ago

For requirement messages it'd save a lot of time if it'd ignore them if they had .esm/.esp as the majority of those are explicit masters and take time to remove

OK, I'll make the change before you start on the FNV masterlist. Term is starting up again for me soon, so my time is getting limited again, and I can't ETA on that. The dirty info stuff is almost finished being implemented now though.

Also, I saw your post on BethSoft, I'll take a look at that ASAP.

Ortham commented 11 years ago

I have released a new version of the masterlist converter. Unfortunately, as well as implementing new features (required plugin filename skipping, dirty message conversion), it fixes some bugs. I'll fix anything broken in the FO3 masterlist, so you don't need to worry about that.

kymckay commented 11 years ago

Okay, thanks! Just about to start the issue for the NV masterlist

Ortham commented 10 years ago

I have forked yaml-cpp (the YAML parsing library BOSS uses) to make improvements where I can, and its issue tracker lists the known bugs in its YAML support. I'll probably update the syntax doc to refer to the tracker for limitations in YAML support.

Ortham commented 10 years ago

@boss-developers: Regarding the writing style discussion brought up here, I'd like to suggest another change.

Currently, I have the Masterlist Editing wiki page saying that we should quote all string values using single quotes. However, it seems a bit silly to be quoting Bash Tags and the say, warn and error strings, since they're all "constants" with no need to ever be quoted.

Do people think it would be OK if we stopped quoting them, or would you prefer to quote all string values?

kymckay commented 10 years ago

Yeah that makes sense to me +1 On 31 Oct 2013 11:12, "WrinklyNinja" notifications@github.com wrote:

@boss-developers https://github.com/boss-developers: Regarding the writing style discussion brought up herehttps://github.com/boss-developers/boss-fallout-new-vegas/issues/1, I'd like to suggest another change.

Currently, I have the Masterlist Editing wiki page saying that we should quote all string values using single quotes. However, it seems a bit silly to be quoting Bash Tags and the say, warn and error strings, since they're all "constants" with no need to ever be quoted.

Do people think it would be OK if we stopped quoting them, or would you prefer to quote all string values?

— Reply to this email directly or view it on GitHubhttps://github.com/boss-developers/boss-developers.github.io/issues/4#issuecomment-27477710 .

kymckay commented 10 years ago

Now that I have access to a PC: I think that since they're constants it'd look very messy if we were needlessly quoting them all the time. So we'd only be quoting names, displays (file structure) and contents (messages)? (URLs?)

Ortham commented 10 years ago

So we'd only be quoting names, displays (file structure) and contents (messages)? (URLs?)

It's easier to say that we'd be quoting all value strings but Bash Tags and message types. :P

kymckay commented 10 years ago

Potentially useful feature for consideration:

Plugins inherit "after" data of plugins they're set to load after.

I was just adding some after data to the masters of the NV list and it became apparent that it'd be time saving and neater if they'd inherit the same "after" data as the other plugins they're set to load after. I've been trying to think of a scenario where this would be unwanted with no luck so far, but I'm sure one of you guys can come up with something :stuck_out_tongue_closed_eyes:

Ortham commented 10 years ago

Potentially useful feature for consideration:

Plugins inherit "after" data of plugins they're set to load after.

I've got a feeling that sort of thing could introduce unwarranted cyclic behaviour in a load order, but can't come up with a concrete example right now.

Like how you could have B needing to load after A, and C needing to load after B, but C might not have any overlap with A, so forcing C to load after A is unnecessary at best, and while lacking an overlap would mean those plugins wouldn't have any issue with that, if other plugins in turn depended on them and overlapped, that could result in issues.

Still, that doesn't completely shoot the idea down, so I'll keep going over it in the back of my mind.

kymckay commented 10 years ago

Just curious about what exactly a version condition will check for?

I've just tested the addition of a letter on the end of a number. From what I can tell BOSS is comparing those as a file with version "2.0.0c" returns greater than "2.0.0a" and less than "2.0.0d". Also greater than simply "2" and "2.0.0".

Is there some kind of syntax the version detection follows? Might be worth noting in the editing doc.

Ortham commented 10 years ago

Is there some kind of syntax the version detection follows? Might be worth noting in the editing doc.

The algorithm is described here. I'll include a link to it in the syntax doc.

kymckay commented 10 years ago

Thanks, perfect!

kymckay commented 10 years ago

I had a thought for consideration earlier, since we're progressing pretty slow with the masterlist conversion (yaaay real life :stuck_out_tongue_closed_eyes: ) it might be worth thinking about setting up working versions of each list containing the essential information. Then using branches to convert the existing data over time. Could be a more productive solution if we're all going to be quite busy for a while to come.

kymckay commented 10 years ago

Just dumping these find and replaces here so I remember them for the other lists. "name" = "name, display, condition, content" (must go through them all)

Stripping things down (in order).

Keynames:

name:\s'(.{1,300})'\r\n

name: \1\r\n

Lists:

-\s'(.{1,300})'\r\n

- \1\r\n

Double quotes:

''

kymckay commented 10 years ago

Adding quotes in (in order)

Quotes:

''

Keynames:

name:\s(.{1,300})\r\n

name: '\1'\r\n

Lists:

-\s(.{1,300}\.es[pm])\r\n

- '\1'\r\n

kymckay commented 10 years ago

Side effect of this procedure:

Bashed tags that use "name:" may be given quotations.

Sharlikran commented 10 years ago

I was initially planning to learn about the mods and Bash tags for FO3 and FNV working with the current v2 syntax. I felt like it would be simply academic to apply a new syntax. I just e-mailed WrinklyNinja prior to posting this and his recommendation was to manually add my additions. I agree if the v3 masterlist for FO3 and FNV are done already then entering oh say 30-50 plugins is a much better approach then using the masterlist-converter.exe and starting over.

The masterlist-converter.exe seems to be skipping plugins and I can't figure out why. I just ran the the masterlist-converter.exe and the file produced is 104kb which is about 12kb bigger then the current v3 masterlist on the Git Hub. Initially I thought that my additions to the v2 masterlist accounted for the difference in size. However, when I searched for plugins that I added, or plugins just above what I added they are not there.

I will have to evaluate what is going on a little more. I presume that the v3 masterlist.yaml file should in essence be exactly the same as the v2 masterlist, just with the new syntax. All the plugins are to be sorted in exactly the same order as they were in v2.

Ortham commented 10 years ago

I presume that the v3 masterlist.yaml file should in essence be exactly the same as the v2 masterlist, just with the new syntax.

No, plugins with no metadata are skipped, that might be what you're seeing. The file size being larger is just because the syntax is more verbose, but that gets countered by Git's compression so it's not an issue.

Sharlikran commented 10 years ago

Okay so let me if I got this right.

Zeta.esm <- No metadata, skipped. ACES - Casino Data Extension.esm <- No metadata, skipped. TaleOfTwoWastelands.esm IF VAR(TTW-WARN) WARN: Need check for Tale of Two Wastelands load order IF VAR(TTW-WARN) GLOBAL WARN: Need check for Tale of Two Wastelands load order

TaleOfTwoWastelands.esm has metadata so then it's converted? Do I understand that correctly?

loot / loot.github.io