Issue #71 saw a lot of useful discussion, and a number of prototypes and pull requests. Now that manifests for mods and graphics packs have been supported for several months, I think it's time to move to a more focussed issue about utilities.
(But first - manifests have been adopted by several graphics packs, and a few mods. I think this reflects the proportion of their users using PyLNP, so utilities would probably adopt pretty quickly.)
The rationale is basically the same as for graphics packs or mods (compatibility, tooltip, title), with a few extra points:
There are more compatibility issues for utilities (needs a terminal, which OS)
Specifying the executable file is also important
I can see a couple of additional challenges compared to current manifests:
utilities, unlike graphics or mods, may be in nested folders (so a manifest could be anywhere in the directory tree, not just at LNP/Utilities/*/manifest.json)
it must be backwards-compatible, retaining the current system. However we don't want the current system to autodetect something covered by a manifest, so it'll need some upgrades.
Utilities are currently identified by the relative path from LNP/Utilities to the executable. This is a good system, and makes it fairly easy to walk back up the tree looking for a manifest. The manifest for a given utility is thus the first one found by so walking.
There are no cases I know of with more than one user-facing executable per utility; in the case of Soundsense there's an executable per platform, and it and Quickfort both have 'library' executable not intended for direct use.
Each manifest, working 'down' the tree from the root, would disable autodetection and manifest-searching in that and any 'lower' folder.
Given the above, I propose that a utility manifest should be the same as the current, with the addition of the following keys:
{
"needs_terminal": false, # boolean, default false
"win_executable": "", # string; relative path from manifest dir to executable, default ""
"osx_executable": "", # as above
"linux_executable": "", # as above
}
I think I've addressed all the questions raised in the last issue, so unless something else comes up I'll make a pull request with an implementation in a few days/months when I have some free time.
[Issue created by PeridexisErrant: 2015-12-13]
[Last updated on bitbucket: 2016-03-14]
[Comment created by PeridexisErrant: 2016-03-13]
Thanks for the update - I'll release a starter pack after PyLNP hits 0.11 then :)
[Comment created by Pidgeot: 2016-03-13]
Almost. #104 still needs to be done - I hope to take care of that one tomorrow - and I also have a few ideas I need to test for #100, to see if I can get that in for 0.11 as well. I will not let that one hold back a release longer than until March 23, though.
It'll probably end up happening before then. Most likely guesses are within the next 48 hours, or next weekend, depending on the outcome of those tests.
[Comment created by PeridexisErrant: 2016-03-13]
Are we now due for 0.11?
At least three feature changes (util manifests, keybind format, DFHack init files)
[Comment created by Pidgeot: 2016-03-12]
Merged in PeridexisErrant/python-lnp/utility-manifests (pull request #61)
Utility manifests (fixes #103)
→ <>
[Comment created by Pidgeot: 2016-03-12]
Merged in PeridexisErrant/python-lnp/utility-manifests (pull request #61)
Utility manifests (fixes #103)
→ <>
[Comment created by Pidgeot: 2016-03-14]
Removing milestone: 0.11 (automated comment)
[Comment created by PeridexisErrant: 2016-01-21]
There's a mostly-done version in this branch, which once testing and documentation are done should be ready for 0.11
(I think once pull request #60 and this are merged 0.11 will be due, especially with the bugfixes from 0.10e)
[Comment created by Pidgeot: 2016-01-20]
Perhaps I'm just not picturing your proposed algorithm correctly, but it sounds like you're making it more complex than it needs to be.
My take on the algorithm for detecting utilities is like this (pseudocode):
#!python
def read_utilities():
read_metadata() # Gets the metadata from utilties.txt, already in there
foreach directory d in LNP/Utilities:
scan_dir(d)
def scan_dir(d):
if exists(d/manifest.json):
add_from_manifest(d/manifest.json)
return # With a manifest, we know exactly what the utility is, so no need to process further
foreach file f in d:
if (f in includes) or (f not in excludes and f matches pattern):
add_utility(f)
foreach directory d2 in d:
if platform == darwin and d2 not in exclusions and d2.endswith(.app):
add_utility(d2)
else:
scan_dir(d2)
def add_from_manifest(path):
#(This might optionally be extended to allow more than one utility in the manifest, in case of "sub-utilities". I remember e.g. QuickFort has more than one executable, but I don't remember if both are worth showing, so I'll leave this part up to you.)
#load correct executable key
#load into metadata - optionally overwriting anything that's already there
def add_utility(path):
metadata.setdefault(path, {}) #in case we already have metadata through utilities.txt, we don't want to clobber it
metadata[path].setdefault('foo', 'bar') #repeat for all the properties that might come from the manifest and we want to preserve - again, using setdefault to let metadata from utilities.txt through
IMO, that doesn't feel very different to the current approach.
For the metadata, there's already a dictionary which can hold arbitrary data, so add_utility/from_manifest can just write to there. The final list of paths can be acquired with metadata.keys() after the scan is complete.
I didn't specify how to get patterns, includes and excludes through to scan_dir, because it's not that important - you can just send those as parameters or whatever.
If you're having trouble following that approach, feel free to ask.
[Comment created by Pidgeot: 2016-01-21]
AFAIK, OSX app bundles must explicitly be named .app, so include patterns should not be necessary.
[Comment created by PeridexisErrant: 2016-01-20]
Yep, that's pretty much what I imagined - some details to work out, but it's looking easier than I expected or remembered (I just didn't have any experience a year ago).
(also @fricy ) Is it possible for OSX app directories not to end .app? If so I'll try to match them against custom include patterns too.
[Comment created by PeridexisErrant: 2016-01-20]
@Pidgeot - as I work on this, I'm thinking a lot about backwards-compatibility and deprecating features.
Basically, we have two and soon to be three systems for configuring utilties:
include.txt and exclude.txt - global config for which files to display, no tooltip support.
utilities.txt - global config for which files to display, titles, and tooltips. Annoying to maintain.
Utility-specific manifest.json - 'local' config for file selection, tooltips, additional metadata (eg requirements for DFHack, a terminal). I really like letting utility maintainers set their own metadata.
The plan is to walk down the file tree, applying a combination of (1) and (2), plus screening off any subdirs with a manifest and applying only(3) in there.
Initially, I'll keep this fully backwards-compatible - but the complexity of where to find config is starting to look pretty ugly. My preference would be to slowly remove (2), first waiting for takeup of manifests, then logging a deprecation warning, and finally removing the feature. I could agree to keep a limited form that doesn't deal with EXCLUDE or implicit include, but it's just hard to explain or reason about otherwise.
Issue #71 saw a lot of useful discussion, and a number of prototypes and pull requests. Now that manifests for mods and graphics packs have been supported for several months, I think it's time to move to a more focussed issue about utilities.
(But first - manifests have been adopted by several graphics packs, and a few mods. I think this reflects the proportion of their users using PyLNP, so utilities would probably adopt pretty quickly.)
The rationale is basically the same as for graphics packs or mods (compatibility, tooltip, title), with a few extra points:
I can see a couple of additional challenges compared to current manifests:
utilities, unlike graphics or mods, may be in nested folders (so a manifest could be anywhere in the directory tree, not just at
LNP/Utilities/*/manifest.json)it must be backwards-compatible, retaining the current system. However we don't want the current system to autodetect something covered by a manifest, so it'll need some upgrades.
Utilities are currently identified by the relative path from
LNP/Utilitiesto the executable. This is a good system, and makes it fairly easy to walk back up the tree looking for a manifest. The manifest for a given utility is thus the first one found by so walking.There are no cases I know of with more than one user-facing executable per utility; in the case of Soundsense there's an executable per platform, and it and Quickfort both have 'library' executable not intended for direct use.
Each manifest, working 'down' the tree from the root, would disable autodetection and manifest-searching in that and any 'lower' folder.
Given the above, I propose that a utility manifest should be the same as the current, with the addition of the following keys:
I think I've addressed all the questions raised in the last issue, so unless something else comes up I'll make a pull request with an implementation in a few days/months when I have some free time.
[Issue created by PeridexisErrant: 2015-12-13] [Last updated on bitbucket: 2016-03-14]
[Comment created by PeridexisErrant: 2016-03-13] Thanks for the update - I'll release a starter pack after PyLNP hits 0.11 then :)
[Comment created by Pidgeot: 2016-03-13] Almost. #104 still needs to be done - I hope to take care of that one tomorrow - and I also have a few ideas I need to test for #100, to see if I can get that in for 0.11 as well. I will not let that one hold back a release longer than until March 23, though.
It'll probably end up happening before then. Most likely guesses are within the next 48 hours, or next weekend, depending on the outcome of those tests.
[Comment created by PeridexisErrant: 2016-03-13] Are we now due for 0.11?
It's been (source):
[Comment created by Pidgeot: 2016-03-12] Merged in PeridexisErrant/python-lnp/utility-manifests (pull request #61)
Utility manifests (fixes #103)
→ <>
[Comment created by Pidgeot: 2016-03-12] Merged in PeridexisErrant/python-lnp/utility-manifests (pull request #61)
Utility manifests (fixes #103)
→ <>
[Comment created by Pidgeot: 2016-03-14] Removing milestone: 0.11 (automated comment)
[Comment created by PeridexisErrant: 2016-01-21] There's a mostly-done version in this branch, which once testing and documentation are done should be ready for 0.11
(I think once pull request #60 and this are merged 0.11 will be due, especially with the bugfixes from 0.10e)
[Comment created by Pidgeot: 2016-01-20] Perhaps I'm just not picturing your proposed algorithm correctly, but it sounds like you're making it more complex than it needs to be.
My take on the algorithm for detecting utilities is like this (pseudocode):
IMO, that doesn't feel very different to the current approach.
For the metadata, there's already a dictionary which can hold arbitrary data, so add_utility/from_manifest can just write to there. The final list of paths can be acquired with metadata.keys() after the scan is complete.
I didn't specify how to get patterns, includes and excludes through to scan_dir, because it's not that important - you can just send those as parameters or whatever.
If you're having trouble following that approach, feel free to ask.
[Comment created by Pidgeot: 2016-01-21] AFAIK, OSX app bundles must explicitly be named .app, so include patterns should not be necessary.
[Comment created by PeridexisErrant: 2016-01-20] Yep, that's pretty much what I imagined - some details to work out, but it's looking easier than I expected or remembered (I just didn't have any experience a year ago).
(also @fricy ) Is it possible for OSX app directories not to end
.app? If so I'll try to match them against custom include patterns too.[Comment created by PeridexisErrant: 2016-01-20] @Pidgeot - as I work on this, I'm thinking a lot about backwards-compatibility and deprecating features.
Basically, we have two and soon to be three systems for configuring utilties:
include.txtandexclude.txt- global config for which files to display, no tooltip support.utilities.txt- global config for which files to display, titles, and tooltips. Annoying to maintain.manifest.json- 'local' config for file selection, tooltips, additional metadata (eg requirements for DFHack, a terminal). I really like letting utility maintainers set their own metadata.The plan is to walk down the file tree, applying a combination of (1) and (2), plus screening off any subdirs with a manifest and applying only(3) in there.
Initially, I'll keep this fully backwards-compatible - but the complexity of where to find config is starting to look pretty ugly. My preference would be to slowly remove (2), first waiting for takeup of manifests, then logging a deprecation warning, and finally removing the feature. I could agree to keep a limited form that doesn't deal with EXCLUDE or implicit include, but it's just hard to explain or reason about otherwise.