Python `contrib` conventions

[skef]

In the past year most of the contents of FontForge's contrib directory were either removed from the repo or pulled out of the release build process. That was not out of skepticism about external contributions in general (I don't think), it had more to do with what had wound up there. contrib had become a place to put "fragile" projects that mixed C and python interfaces, and the move to GDK had altered the cost/benefit ratio against them.

I've made noises in the past about creating a robust "plugin" system that would allow "outside" contributors to deliver their own functionality through python modules (installed with pip, for example). That's still a worthy goal but a) it hasn't happened (yet?) and b) such a system still relies on users being able to find what they need. So it would be nice to have something concrete in the mean time.

Anyway, I'm opening this issue to suggest and start a discussion about some conventions for accepting contributions to be shipped with FontForge. Here are my thoughts:

1. I think there has been some mild resistance in the past to adding scripts for things that could be added in C code without too much trouble. I suggest we just warn contributors that "we" can implement pieces of their contributions in C and remove their work as we see fit in the future and not worry about it beyond that.
2. Acceptance into contrib should be based on the combination of: a. General applicability of the functionality b. Code quality c. Engagement of the contributor at the time of contribution. (This is a weak indicator of likelihood of future engagement, but it's something.) Code quality is, of course, something that can be improved through engagement.
3. I support a convention of each contributed tool having its own sub-menu under "Tools", so that FontForge's (in effect) mandatory contributions don't eat up a lot of menu space, but each contribution has its own area.
4. I also support adding default hotkeys for the more useful contributed functions, using more or less the same decision process we would when deciding on hotkeys for "native" menu options.

These questions are newly relevant because of @linusromer's offer in #3216. After seeking input from other FontForge users that tool has evolved into https://github.com/linusromer/curvatura, which I personally think is very close to meeting the bar for release in FontForge's contrib directory. It definitely adds functionality that has already been requested, and does so in a way that isn't compromised by being in script form. (That is, if we add hotkeys for it. @linusromer has some suggestions about those here.)

Therefore I suggest we discuss this a bit, and if we come to an agreement we either invite @linusromer to submit a PR, or construct some infrastructure that allows our build process to pull contributions from their own repositories. (If we do the latter we might want to request some level of access to the outside repo for some or all team members.)

[frank-trampe]

I agree with all of this, and you seem to be best positioned to propose a specific technical architecture.

[skef]

@frank-trampe I'm well-positioned in some respects but a) it does seem like being able to pull contributions from their own repos would be nice and b) I only have a vague understanding of the build infrastructure.

I guess there are actually three options:

1. Pull from externally managed repos,
2. Have contributors merge into the main fontforge repo
3. Open a new fontforge/contributions repo for this stuff and give all contributors edit permissions on it.

Option 3 has some attractive aspects.

Any thoughts @jtanx? If we can work out a mechanism I'm happy to help guide curvatura into the next release.

[ctrlcctrlv]

The problem is, as always, #3663. Curvatura works by registering a bunch of menu items. That's fine, and a lot of scripts can work that way, but to come up with contrib conventions that don't solve this problem is a half solution.

For example, you failed to mention at all @glukfonts' project, SVGinOTlayersCompositor. Maybe you don't know about it? It works via Tkinter! Tkinter, in 2019! The first commit was Oct 1.

And you know what? If we write in our conventions that Tkinter is what is to be used, I will even be OK with that. Even though we had a well working GTK integration, I will be quiet about it. But this cannot continue to be ignored if we are to have sane guidelines. I salute the work @glukfonts did but we will break it at any time. We need to make a promise that GTK will work from Python, or that Tkinter will, and that that will not be broken, for me to bother writing scripts.

cc: @jtanx @skef

[skef]

My proposal, and therefore this issue, is about the bar for including contributed scripts in FontForge releases. The question of what should be possible to do with a script is an important one, but also a separate one.

[ctrlcctrlv]

does a script that includes Tkinter pass that bar

[skef]

@ctrlcctrlv, the most basic answer to that question is: Not automatically by default. Some thought would have to go into it before it could.

tkinter does have the advantage of not being X-specific. Beyond that I know very little about it on a practical level. (I played around with tcl/tk some ages ago in its heyday.)

One requirement on any such script living in contrib would be acting sensibly when dependencies aren't installed. I doubt FontForge would add a "hard" tkinter dependency. Beyond that, we'd have to consider how the result acts on different systems and how "fragile" the combination is likely to be over time.

[frank-trampe]

@ctrlcctrlv, it says that it is for Python 2.x. Does it work with Python 3?

[ctrlcctrlv]

@skef Well we ought to have those thoughts because whether or not Tkinter is allowed is going to have a tremendous influence on what scripts get in, and if we let in one script with Tkinter that's an enduring promise that such will be allowed longterm.

@frank-trampe It does not but I could just as easily write/find an example that uses Python 3.

[skef]

It would be fairly typical if the easy consensus cases wind up in a contrib context while scripts that are more contentious or difficult to evaluate don't wind up there.

@ctrlcctrlv To the extent tkinter-like UI extensions are important you might want to focus on pushing for the plugin-type support discussed elsewhere. That way any worries about fragility are beyond the project members' scope.

[ctrlcctrlv]

I'm not really making a big ask here.

Either we promise not to break Tkinter, or we fix GTK. Either one closes #3663 as far as I'm concerned.

We should only accept scripts for contrib that check whether or not they can import what they need as far as C dependencies go, and which bundle any pure Python dependencies.

[frank-trampe]

I second the motion to draw @jtanx into this discussion for perspective on GTK and the build system.

[ctrlcctrlv]

Since someone is already using Tkinter, and since the Python developers are still bundling it by default, we should just promise not to break it. I can write #4066 with Tkinter as a demo.

[ctrlcctrlv]

By the way @frank-trampe, here's a dead simple demo that works in Python 3 if you're curious:

# Python 3 only
from tkinter import messagebox, Tk

window = Tk()
# On my system, lack of the following line causes a superfluous window to appear
window.wm_withdraw()

messagebox.showinfo("I matter", "Tkinter matters")

# Very important, otherwise window will never close
window.destroy()

I really don't understand why @skef doesn't think that once we open the repository to outside Python scripts people aren't going to (rightfully) want to do stuff like this...we need to know if we're going to allow it, or if we're going to merge the GTK fix and force people to use GTK.

[skef]

@ctrlcctrlv I think two different issues may be getting confused here.

There was discussion at one point of a script repository. If the FontForge project created a such a repository and then decided what could and could not go into it based on, for example, certain dependencies, that would raise the issues you've been discussing.

That's not what the contrib directory is. Scripts put into the shared contrib directory are not a matter of user choice -- each one is installed by default, probably can't be uninstalled without admin access in most cases, and automatically runs and takes up "real estate" in the Tools menu. Only a very small percentage of user-created scripts are likely to wind up in that context. When I wrote "General applicability of the functionality" I meant very general -- only those scripts likely to be useful to a high percentage of users.

If the linked-to SVG font support script had no dependencies at all I doubt I would personally support adding it to contrib, not because I didn't think it was valuable or well-written but simply because SVG fonts are currently a bit niche. People who need that tool support can install it themselves.

The idea that the project would include a small number of key scripts in contrib does not weigh one way or another on additional repositories or distribution mechanisms for a broader set of scripts.

[skef]

On the other issue:

I'm happy to argue again for what I argued before: The ideal repository for (Python) FontForge extensions would be PyPI. The obstacle in the way of that solution is a discovery mechanism/convention. That's a solved problem for Python. We should probably use the namespace solution. If folks were to say to me "we think this is the top priority" I would probably do it.

With a discovery mechanism we can add hooks that will be called on packages when FontForge starts up, so that they can (e.g.) register menu items. It would also be nice to have a place in 'Prefs' where you can turn a given discovered plugin off. None of this is very difficult to code up.

There's one problem, however, that I would need @jtanx's help with. We need to be able to install real python-y files in addition to our C-API modules. (I'm not going to bother trying to code up a namespace package in C just because we don't have a way to install a python file. We write far too much Python in C as it is.)

[ctrlcctrlv]

I'm not sure PyPI is ideal. On Windows for example the Python FontForge uses is the one we bundle, right? So are we going to start bundling pip as well?

[skef]

That's already a problem for any "third party" script with python dependencies -- the storage and retrieval mechanism doesn't change that. Windows is likely a problem in that way unless and until we bind against the "native" one. (I believe switching to CMake was supposed to bring that closer to possible, but I don't know what further issues we face in compiling with Visual Studio.)

(In case anyone has envisioned a repository that bundles all the dependencies, that would end up being a much bigger hornets nest in the end.)

[ctrlcctrlv]

My idea which will work fine on Windows and for users not tech savvy enough to touch command lines:

• We make a repo named fontforge/contrib
• Each contrib script is in a single directory, users add more scripts through PRs
• We do some QA to make sure each script bundles all its pure Python dependencies and bails out if it can't find a C dependency
• We pack all contrib scripts by default, but make a UI for enabling/disabling them, with only a select few enabled

This is similar to how the WeeChat project does things.

[skef]

We do some QA to make sure each script bundles all its pure Python dependencies and bails out if it can't find a C dependency

Most obvious dependency for people looking to do more advanced spline stuff: numpy. Are we going to ask folks going to "bundle" numpy?

[skef]

I'll be stronger: Dependencies have their own dependencies. Code changes all the time in ways requiring updates. There are potential security problems. On systems where we're using the system python the dependencies we install will clash with the dependencies installed by the user

It's a non-starter, not going to happen.

[ctrlcctrlv]

Aren't we coming up with conventions in this thread? I think that a system that has no plans for being user-friendly or accessible to Windows users (or other platforms where we bundle the interpreter) is a "non-starter".

I don't think it's a big deal to add a well-defined subdirectory to the Python module search path on select platforms. Just tell developers if they can put any Python module they want into a folder named e.g. vendor. If the system Python is in use just don't put these vendor directories into the Python path.

[jtanx]

Apologies, I haven't been following this very closely, and I still don't have too much time on my hands right now.

Here's my 2c

contrib had become a place to put "fragile" projects that mixed C and python interfaces

It's also (if history is anything to go by), the place where code goes to die, because no one looks after it; the same too for pycontrib.

Open a new fontforge/contributions repo for this stuff and give all contributors edit permissions on it.

Out of the three options presented, I like this one the best.

The problem is, as always, #3663

I don't think so, I think that's conflating this issue. As to the whole Tk vs. GTK (vs Qt?) argument, I don't see that as being a problem; correct me if I'm wrong but given the existing limitations on how the GUI can be influenced through scripting, it's unlikely that whatever will be done in the future will cause any pre-existing functionality that uses some other toolkit to break. If it's looking like it will, well there's nothing preventing re-evaluating at that stage.

. We need to be able to install real python-y files in addition to our C-API modules

At a guess I'd think this would be a breaking change. I've thought about this before; basically we'd need to install a package like:

site-packages/fontforge/__init__.py # could potentially pull in the C modules by default or somehting
site-packages/fontforge/fontforge.pyd
site-packages/fontforge/psMat.pyd
site-packages/fontforge/other_garbage.py

But I don't know how we'd do that and retain backwards compatibility.

. On Windows for example the Python FontForge uses is the one we bundle, right? So are we going to start bundling pip as well?

We bundle a whole Python distribution, same as the mac bundle. It already has pip installed, iirc, although it's not that usable because we ship the msys2/mingw-w64 Python, which means there aren't binary wheels for things like numpy/pandas.

Finally some further thoughts:

• I'm generally ok with the idea proposed, but I would like to keep it lean/be wary of what's included by default (and as you mention @skef, be careful about what the tertiary dependencies are)
• Prove me wrong, but I'm not convinced there will be a strong uptake in contributions to whatever we end up doing. So I'd be hesitant to spend a bunch of effort and over-engineer this when no one will use it.

[ctrlcctrlv]

Prove me wrong, but I'm not convinced there will be a strong uptake in contributions to whatever we end up doing. So I'd be hesitant to spend a bunch of effort and over-engineer this when no one will use it.

I think there would be. I already mentioned the SVGinOTlayersCompositor thing...as the manager of the @FontForge Twitter account I see a lot of what people in the community are doing and there are a lot of scripts floating out there, people are disappointed there's no way to contribute them. We'd probably have around 10 scripts by mid-year 2020, I'll be contributing some of those of course.

it's unlikely that whatever will be done in the future will cause any pre-existing functionality that uses some other toolkit to break.

Well so long as we commit to not breaking Tkinter, which works right now, in the formal write up of the conventions, I'd be happy to support them.

@skef also wrote:

It would also be nice to have a place in 'Prefs' where you can turn a given discovered plugin off. None of this is very difficult to code up.

If we had this combined with a vendor PYTHONPATH we'd have a very powerful system. :-)

[skef]

Open a new fontforge/contributions repo for this stuff and give all contributors edit permissions on it.

Out of the three options presented, I like this one the best.

How much of a problem for end-user builds is that likely to cause? Would we need a step like the old build tools download or can CMake just deal with that itself? (Otherwise this is my preference as well.)

At a guess I'd think this would be a breaking change. I've thought about this before; basically we'd need to install a package like ...

Well, wait, @jtanx there are two "levels" to this. For now I'm just talking about shipping a python file under the fontforge namespace. (Or, if absolutely necessary, a different namespace. We just want to be sure that if fontforge is installed with python the discovery plumbing is there too.) I thought it was possible to ship myapp in one package and myapp.subclass in another, but maybe I'm remembering wrong.

If we're talking about the second level we'd start by changing our classes to "fontforge_int" (or whatever) and then have the python files import and inherit from the renamed C-API modules. That might break some really subtle stuff, but with a little care almost everything should be fine. (Famous last words.)

(I'd be curious to play around with that a bit if and when we get a chance. Just make "empty" python files that inherit, change the names around, and see if things work and if not what breaks. Some of the internal plumbing might need to change a bit but I'm guessing it would be doable.)

[skef]

I'm generally ok with the idea proposed, but I would like to keep it lean/be wary of what's included by default (and as you mention @skef, be careful about what the tertiary dependencies are)

I initiated this discussion because I think the already discussed Curvatura tool is close to meeting a bar for shipping with FontForge and therefore it would be good to have a model for that. But I agree the generality bar for contrib (or I guess pycontrib ...) should be high and the dependencies should be modest at most.

Subjectively I think the best indication is whether the tool seems like something that we would contemplate building into FontForge itself.

---

Anyway, if our "custom" python implementations have pip, and we have the same trouble compiling C or other non-python sources for them regardless, then there's even fewer arguments against the PyPI route for the broader set of tools. What we need for that are:

1. A "namespace class" to inherit from (which I can write) and a way to get it installed (which @jtanx can help with
2. The code to handle discovery (which I would also prefer to write in python and just invoke with the C-API, but that I can write in C if absolutely necessary.
3. A simple convention to import and call a function on discovered modules. (Basically make the interpreter aware of the module and call a method on it that means "initialize yourself as a plugin, and btw we're running with/without a UI.")
4. Preferably a screen in Preferences with a scrolling list of discovered plugins that allows you to disable them, which the discovery code takes into account.

Not a huge project.

[ctrlcctrlv]

Why not just install to a new directory like /usr/share/fontforge/python and tell FontForge to look there, like it already looks at ~/.config/fontforge/python?

[skef]

Install which files?

[ctrlcctrlv]

The contents of fontforge/contrib

[skef]

Oh, where the files end up isn't the problem, we'd presumably put them where pycontrib/excepthook.py ends up now. The only tricky bit is, if we maintain them in a separate repository, getting the plumbing that pulls them in to work well. We presumably want builds made by end-users to have the files too, which means they're build will have to pull the files down.

[ctrlcctrlv]

Cool. I'm glad we're allowing Tkinter. Do you want me to start writing contribution guidelines, or do you have that covered? If I finish #4066 would that meet the bar for inclusion you think?

[skef]

(I'm assuming you're talking about tools that are the subject of this issue narrowly. If you're talking more generally then pip should already put modules in the right place.

The problem we have right now is that there's no good way of building a "plugin-like thing" as just a python module because there's nothing to do an initial import to cause menu item registration. The current convention is to put files in a directory where each gets executed.

[skef]

Cool. I'm glad we're allowing Tkinter. Do you want me to start writing contribution guidelines, or do you have that covered?

If we pursue the discovery/plugin model I'm advocating then there are no contribution guidelines because the project has no control anyway. The only thing we might offer is a page with links, and maybe a searchable database later if the number of contributions gets large enough. (I suppose we might want to have some basic guidelines for what we'll "advertise" but I think an "at your own risk" warning at the top and the usual no-hate-or-pr0n conventions would be enough.)

[ctrlcctrlv]

Surely we want to distribute scripts that are disabled by default but accessible to the user in the plugin preferences pane you talked about, right?

[skef]

Surely we want to distribute scripts that are disabled by default but accessible to the user in the plugin preferences pane you talked about, right?

We don't for technical reasons: we don't want our installation scripts to have to deal with the dependencies.

[ctrlcctrlv]

Why not both Skef? If the system Python is in use they can install modules, but we also distribute a bunch. If it won't be accessible in the default distribution people won't be as excited to contribute.

[skef]

If we want to contemplate shipping scripts with literally zero non-core dependencies that way I have no particular objection. We are not bundling third-party dependencies with our code. They will both conflict with installed packages and rot between our releases.

[skef]

Any dependency we ship isn't going to have the right package metadata and therefore won't be pip-upgradable. It's a nightmare.

[jtanx]

Agree with @skef on that.

For now I'm just talking about shipping a python file under the fontforge namespace

To clarify, you want to do something like

from fontforge.contrib import something

?

The only tricky bit is, if we maintain them in a separate repository, getting the plumbing that pulls them in to work well. We presumably want builds made by end-users to have the files too, which means they're build will have to pull the files down.

We already do something to download additional test fonts when the tests are run and/or when making the dist package. We could hook it so that they're bundled as part of the dist package I guess.

[skef]

@jtanx, yes, more or less. We need to add a package similar to myapp.plugins discussed at https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages , so that other packages can put themselves in that namespace and FontForge can "discover" them on that basis.

[skef]

We already do something to download additional test fonts when the tests are run and/or when making the dist package. We could hook it so that they're bundled as part of the dist package I guess.

OK, I'll think about the mechanics of the new repository and we can go from there. I would prefer having separate directories for each tool, so that each can have a README.md file we might link to, and then maybe some cmake or just make stuff to coalesce the scripts into a single directory.

[skef]

And I guess it won't be that hard to do the equivalent of

discovered_plugins = {
    name: importlib.import_module(name)
    for finder, name, ispkg
    in iter_namespace(myapp.plugins)
}

from the C-API so I I suppose I can just write that stuff in C.

[skef]

(Hmm. Let me read up on this properly before doing anything -- I'm going off a pretty brief skim. The important thing is that it's workable.)

[skef]

Oh, this is super easy.

@jtanx Shipping actual python code remains among my strong desires, but we don't need that for this. I'll make a prototype (probably without prefs-disabling to start with) to play around with.

[ctrlcctrlv]

We already do something to download additional test fonts when the tests are run and/or when making the dist package. We could hook it so that they're bundled as part of the dist package I guess.

You could do the same to bundle non-core Python dependencies for packages that don't use the system's Python.

For each installed script:

pip install -r requirements.txt --prefix=vendor

[skef]

The proposal is a system that allows FontForge to be extended through normal python packages. How is the independence from the project itself not a feature rather than a bug? I understand the "excitement" point but I don't see how it outweighs all the downsides.

[ctrlcctrlv]

It certainly is a bug. People want their code in the main release. People want end users to be able to use their scripts without a lot of hassle on Windows and OS X, where a suitable system Python/Pip is unlikely to exist.

I'm against such a system if it's our only answer to plugins. Implement it, fine, but I strongly believe we need to promise to distribute scripts if we're going to get developers interested in contributing them. Only enable a few of them by default. Make them pip packages and at the time of building for Windows have Pip install the requirements into vendor.

[skef]

People want end users to be able to use their scripts without a lot of hassle on Windows and OS X, where a suitable system Python/Pip is unlikely to exist.

@jtanx already clarified:

We bundle a whole Python distribution [on Windows], same as the mac bundle. It already has pip installed, iirc, although it's not that usable because we ship the msys2/mingw-w64 Python, which means there aren't binary wheels for things like numpy/pandas.

So there's already a pip for windows and mac that can do what your alternative could do at most: install pure python packages.

[ctrlcctrlv]

So there's already a pip for windows and mac that can do what your alternative could do at most: install pure python packages.

Why should we expect users to bust out the command line to install the dependencies for a script we distribute? If we're bundling a whole Python distribution, why can't we install Python dependencies we know we need into the Windows/Mac bundles?

[skef]

I've said my piece.

[ctrlcctrlv]

So scripts accepted into contrib should have their dependencies installed on the Windows/OS X bundles. Glad we agree. If it's anything like Virtualenv it should be enough just to run pip install -r requirements.txt for each module; you can drop the --prefix and the idea of having vendor folders entirely.