Add clone of pysilfont to provision.sh

[jvgaultney]

Add a line to provision.sh to clone the pysilfont repo (git clone https://github.com/silnrsi/pysilfont.git). This will provide easy access to all the scripts, particularly those not installed.

[bobh0303]

This is a possible solution to the problem of accessing the scripts from vagrant-based smith VMs. Are you suggesting to then (a) install from that clone or (b) use the existing binary install for most things and, for scripts not installed, the user locates the repo and runs the files from there?

If our fontutils history is any reference, there are some benefits to (a) in that as bugs in pysilfont are fixed then we users don't have to wait for the binary packaging to happen (though perhaps in the case of pysilfont that would be more often than it was with fontutils).

If you mean (b) then I could foresee possible conflicts if the binary and current source are out of sync w/r/t library interface changes.

In any case, I guess I'd still argue for eliminating the middle class ("complete and useful" but not installed) of scripts in pysilfont -- seems simpler, easier for users, and safer.

[DavidRaymond]

Depending on how the packaging is done, it may be straight forward for other scripts etc to be included in the binary and installed somewhere.

[jvgaultney]

This is a possible solution to the problem of accessing the scripts from vagrant-based smith VMs. Are you suggesting to then (a) install from that clone or (b) use the existing binary install for most things and, for scripts not installed, the user locates the repo and runs the files from there?

I can't say whether installing from that clone or letting it be installed as a Smith dependency is easier. I'd hope that both will give you the same up-to-date version, but I realize that's not trivial. :-)

In any case, I guess I'd still argue for eliminating the middle class ("complete and useful" but not installed) of scripts in pysilfont -- seems simpler, easier for users, and safer.

I'd say there will always be the middle class to some extent. For example, if I write a quick and dirty script, check it in so I and others have access to it, do I need to also modify the setup files, etc. to make it installed? I'm guessing that all that work would discourage me from checking in useful scripts, or delay it at least, and others like PM might be less likely to put in installed scripts.

There's another issue here - proliferation of scripts in usr/local/bin (I can't remember the exact place). We don't often go hunting in there for things, but I actually have had to recently, and the more scripts we install the worse that gets. There is a good reason that many libraries (Apple's font tools, some of Google's, fontbakery...) give all their scripts a unique prefix that identifies them as part of that package. It's a pain to type out, but it is useful!

However my preference is to make more of our scripts installed. Up to this point the main judgement of which scripts get installed has been whether preflight or Smith needed it. I'd like to see that changed to also include more scripts that, like UFOsetVersion, make certain standalone tasks simpler.

[bobh0303]

Re the potential for "quick-and-dirty scripts": this has always been a benefit of environments like fontutils and fonttools, so we can expect it to continue with pysilfont and we ought to make it easy. This suggests that for our internal purposes we just stop using debian binary and always use (and run directly from) a repo clone.

Re proliferation of scripts, we didn't seem too concerned with the 31 scripts installed by fontutils (many, but not all, of which do have prefixes, e.g., 'ttf', 'sfd', 'volt')

[DavidRaymond]

For example, if I write a quick and dirty script, check it in so I and others have access to it...

If we did install all scripts currently in the scripts folder, examples could still be used for such scripts

do I need to also modify the setup files, etc. to make it installed?

Actually setup.py now automatically installs any script found in lib/silfont/scripts so it does not need changing when scripts are added. They do need to be correctly set up, with a cmd() function, but since new scripts are normally written by copying an existing one and adapting, that should always be the case. The .py extension is also automatically removed when creating the command name.

However my preference is to make more of our scripts installed.

Mine too.

[devosb]

Agree with installing more scripts. The Debian packaging system can install more scripts than what setup.py installs, but that seems messy not to follow what the upstream project (which is pysilfont) does. So I would say modify pysilfont to control what scripts get installed. Installing from the repo rather than the Debian package will make non-vagrant use of smith harder (not sure if anyone is using such a configuration). The command dpkg -S /usr/bin/ or dpkg -S /usr/local/bin/ will show where each script came from, if the script came from a package. If the script was installed straight from a repo, then dpkg will have no information about it.

[nrsiward]

it may be straight forward for other scripts etc to be included in the binary and installed somewhere ... examples could still be used for such scripts

One problem with providing scripts that are not ready to be directly executable is that the path to them may be non-trivial. If they are part of a cloned repo, the location may be more obvious.

The Debian packaging system can install more scripts than what setup.py installs, but that seems messy not to follow what the upstream project (which is pysilfont) does

I agree this would be bad.

[nrsiward]

After reading a bit about Python packaging, I would guess the Debian install is based on the pysilfont binary distribution. It's normal for that type of distribution to exclude any files not actually in the package (ie not under silfont/). Such scripts are normally provided using a source distribution. It the Debian install switched to using that type, it's unclear where the scripts outside the package would be placed (though others may know). Normally a user would download the source distribution, unpack it, and install the package. The other scripts would be wherever they were unpacked.

In our current state of development, we could certainly move more scripts to silfont/scripts/ to be included in the binary package used in the vagrant smith vm. I think the other scripts would be best obtained by cloning the repo -- either to a standard folder by the provision.sh or by leaving it to the user. Either approach needs to be documented in our workflow docs.

FYI, adding the files to MANIFEST.in only adds them to the source distribution.

[n7s]

I agree with @devosb.

@nrsiward Not sure what you mean here with source distribution. I can adjust what is needed with the packaging. If the underlying issue here is getting the very latest available but not wanting to do it all manually then IMHO it's down to how the setup.py is written and what it includes.

The Debian packaging for pysilfont (and by extension what lands into the PPA) just runs setup.py and installs the results along with the docs (just README.md right now). I'd vote for not having a duplicated scripts/ folder with scripts that don't get installed by the package. Just have one scripts/ (or lib/silfont/scripts/) with all the useful script in there and then put the rest you don't want promoted to usage in examples/. scripts/tools/ is probably redundant too.

[nrsiward]

Not sure what you mean here with source distribution

Background: In the world of Python packaging, you can create either a source distribution (using setup.py sdist) or several kinds of binary distributions (setup.py bdist (for the machine's native kind) or setup.py bdist_foo). (The term "distribution" is used since "package" already has a meaning in Python, as in a folder with an init.py file). The main difference between the source and binary types is whether Python extension modules written in C are included in compiled form or source form (which is irrelevant for us). Another difference is that a source distribution includes extra files including setup.py and files specified in MANIFEST.in.

it's down to how the setup.py is written and what it includes.

I agree. (I would add MANIFEST.in also has a role in a source distribution and setup.cfg can also come into play, but we aren't using these.)

The Debian packaging for pysilfont (and by extension what lands into the PPA) just runs setup.py and installs the results along with the docs (just README.md right now).

Setup.py has to be called with a command, like "sdist" or "bdist". Otherwise, it just produces a usage message. I was speculating that somehow in your packaging process it's being called with "bdist" (or a variant of that). This is the right thing to do, IMHO, so what you are doing now doesn't need to change.

I'd vote for not having a duplicated scripts/ folder with scripts that don't get installed by the package. Just have one scripts/ (or lib/silfont/scripts/) with all the useful script in there and then put the rest you don't want promoted to usage in examples/. scripts/tools/ is probably redundant too.

I agree. The question though is how do some users get those scripts in pysilfont/examples/ which aren't installed as part of the package? I'm saying that the best way to provide them is from a clone of the pysilfont repo, which could be cloned by the user or during provision.sh execution. I personally prefer the former.

A second way to provide those scripts would be to create a source distribution and include them using the MANIFEST.in file, but I think that raises other issues. A third way would be to add them during the Debian packaging as you do with the README.md file, but I dislike this since we should use the project's own distribution generation mechanism (as you are currently doing by somehow calling setup.py).

[n7s]

After talking about this some more we agreed that we'll "graduate" the remaning scripts so they can get installed by the package. We think that cloning and installing any remaining examples scripts from a repo to a project folder is best done manually, so we'd rather not add it to the provision.sh script.

[nrsiward]

At the 2017-05-10 FontTech meeting we decided that since most useful scripts will be moved to silfont/scripts that is was acceptable for the others to be accessed from the source repo. Those not in the package are ones which are: examples for developers, quick hacks for a specific need, project specific ones which require modification, or not executable from the command line. It was agreed that those wanting such scripts could clone the repository themselves and that should not be done by provision.sh.