A question regarding file/folder priority

leontristain commented 5 years ago

Hi GandaG,

Not sure if this is the right place to do this. I just have a simple question about the fomod specification, regarding file/folder priority.

If two folders containing different files have the same destination, would the result be merged or would the winning folder completely override the existence of the losing folder? Likewise, if a folder wins in priority over a file with destination path underneath that folder but does not actually present a file conflict, would the file end up getting installed or omitted?

Thanks in advance for any answers! leontristain

GandaG commented 5 years ago

Here is fine :)

The problem with not having a standard is that... there's no standard. So each mod manager can do it their own way. AFAIK, Mod Organizer 2's installer merges the result (I'm not very fluent in C++) and Wrye Bash's upcoming installer completely overrides the losing folder.

leontristain commented 5 years ago

Thanks for responding!

If you don't mind me asking, what were the reasons for Wrye Bash's design choice of deviating from what MO2 already does, if it's "upcoming" and there's no standard? Wouldn't it cause some FOMODs to break on one mod manager but not the other?

Thanks, leontristain

GandaG commented 5 years ago

I would highly recommend not relying on implementation details such as this on your installer.

That said, Wrye Bash's way was not so much a design choice as the dev that is working on that (me) not really thinking about that specific case :P That said, it's still in alpha and thank you for bringing this to attention, we'll have to make a decision on it. wrye-bash/wrye-bash#380

leontristain commented 5 years ago

For my installer, I'm currently having it support a mo2 mode and a wrye_bash mode. The current algorithm is that it starts with a dictionary mapping priority numbers to lists of install paths (collected over all the pages), and it performs two steps: 1) expand all the folders into the files underneath, and 2) flatten the priorities into one list of install paths, with higher priority numbers winning on any conflicts.

The mo2 mode would simply expand before flatten, while the wrye_bash mode would flatten before expand. It's a simple toggle switch of no more than 4-5 lines. So I'm not worrying about "reliance on implementation details" since worst case scenario it's just fixing up 4-5 lines. If there's a "right way" I'd go after that, but it sounds like there's no right way so might as well support both interpretations (mo2 is the default currently).

I briefly looked at the wrye bash project, and it doesn't look like it's something that's pip-installable as a library, so any code in it seems to be difficult to import and use. Do you expect to backport the wrye bash installer to pyfomod at some point? Or perhaps have another more general implementation in pyfomod? Someday I hope to get rid of my installer and just use one that comes with the library if possible (assuming it has enough features). I know back in pyfomod 3.3 there was an installer (I tried it and there were issues - but that's old history 😃), are there plans to bring that back in pyfomod later?

GandaG commented 5 years ago

The pyfomod installer is getting rewritten for the new API.

Can you test the folder priority on MO2? That would clear any issues.

leontristain commented 5 years ago

Hi GandaG,

So I made a sample fomod to test out MO2's behavior: myfomod.zip

The fomod has the following files:

.
├── bar
│   └── file2
├── baz
├── fomod
│   └── ModuleConfig.xml
└── foo
    └── file1

In ModuleConfig.xml, I added 9 options, in the hopes of covering all the recent conversations we've had both here and in the issues I raised over at the pyfomod github. I figured that if I'm going to do an experiment, might as well cover everything in one go.

case1:

<folder source="foo" destination="destdir" priority="2"/>
<folder source="bar" destination="destdir" priority="2"/>
<file source="baz" destination="destdir/baz" priority="2"/>

case2:

<folder source="foo" destination="" priority="2"/>
<folder source="bar" destination="" priority="2"/>
<file source="baz" destination="" priority="2"/>

case3:

<folder source="foo" priority="2"/>
<folder source="bar" priority="2"/>
<file source="baz" priority="2"/>

I did 3 installs, each time for each of the above cases, and observed the following results:

case1:

├── destdir
│   ├── baz
│   ├── file1
│   └── file2
└── meta.ini

case2:

├── baz
├── file1
├── file2
└── meta.ini

case3:

├── bar
│   └── file2
├── baz
├── foo
│   └── file1
└── meta.ini

So, the following is MO2's behavior:

[related to this conversation] Files are definitely merged when not directly conflicting. This is true in cases where destination specifies a directory, destination is empty, and destination not given. This is also true between folder declarations and file declarations.
[related to one of the issues I filed at the pyfomod github] Actual install behavior differs between destination empty and destination not given. When destination is given and empty, the contents of the source folders are copied to the root of the destination directory. When destination is not given, the source folders are copied to the equivalent location in the destination directory, preserving the folder structure from the source root.

Anyway, I think my questions are sufficiently answered by now, and maybe it would be a bit helpful to you too? In any case, thanks so much for documenting fomods and also to developing pyfomod. It made my life tons easier 😃.

GandaG commented 5 years ago

Could you also test with different priorities? To check if they are merged as well?

leontristain commented 5 years ago

Oh crap, I forgot about that case. My mind was stuck with the idea of the same priorities overriding each other for some reason.

Anyway, will test after I get home today.

leontristain commented 5 years ago

Okay, modified my test fomod to have different priorities: myfomod-differentpriorities.zip

case1:

<folder source="foo" destination="destdir" priority="2"/>
<folder source="bar" destination="destdir" priority="1"/>
<file source="baz" destination="destdir/baz" priority="0"/>

case2:

<folder source="foo" destination="" priority="2"/>
<folder source="bar" destination="" priority="1"/>
<file source="baz" destination="" priority="0"/>

case3:

<folder source="foo" priority="2"/>
<folder source="bar" priority="1"/>
<file source="baz" priority="0"/>

case1 output:

.
├── destdir
│   ├── baz
│   ├── file1
│   └── file2
└── meta.ini

case2 output:

.
├── baz
├── file1
├── file2
└── meta.ini

case3 output:

.
├── bar
│   └── file2
├── baz
├── foo
│   └── file1
└── meta.ini

Basically same as before, files are merged as much as possible.

GandaG commented 5 years ago

Thanks for all the help :P

GandaG commented 5 years ago

@leontristain the last test case I need (I think) is installing folders under a file tag. Can you do that? Something like:

<file source="foo" priority="2"/>
<file source="bar" priority="1"/>
<file source="baz" priority="0"/>

For both priority cases (different and equal) and destination fields (empty, missing, etc.). If you come up with other permutations it would be appreciated.

Thank you!

leontristain commented 5 years ago

Sure, I will do that within the next few days (probably tomorrow - going to get home quite late today).

leontristain commented 5 years ago

myfomod-allfiletags-differentpriorities.zip myfomod-allfiletags-samepriorities.zip

Installing folders with file tags with different priorities:

Case 1: destination specified nonempty

<file source="foo" destination="destdir" priority="2"/>
<file source="bar" destination="destdir" priority="1"/>

<file source="baz" destination="destdir/baz" priority="0"/>

.
├── destdir
│   └── baz
└── meta.ini

Case 2: destination specified empty

<file source="foo" destination="" priority="2"/>
<file source="bar" destination="" priority="1"/>
<file source="baz" destination="" priority="0"/>
```
.
├── baz
└── meta.ini
```

Case 3: destination unspecified

<file source="foo" priority="2"/>
<file source="bar" priority="1"/>
<file source="baz" priority="0"/>
```
.
├── baz
└── meta.ini
```
In all 3 cases, MO2 displays warning messages in the logs with the messages "foo not found" and "bar not found".

Installing folders with file tag with same priorities is the same as different priorities.

So in conclusion, it seems that when folders are being referred to with file tags, MO2 will simply warn and ignore.

GandaG / fomod-docs

A question regarding file/folder priority #2