[ufo3k] Moving pens out of Robofab

[moyogo]

There’s some interest in moving pens out of Robofab, maybe to fonttools or a separate library. @adrientetar @anthrotype @behdad @brawer @graphicore @jamesgk Robofab-devs, could this happen?

[graphicore]

:+1:

I wrote recently in https://github.com/robofab-developers/robofab/issues/46#issuecomment-140086483:

Well, I did not make up my mind completely yet. But with ufoJS I think it's far easier for me to move things around and rearrange them. So I'm thinking of a JavaScript library let's call it "penBox" where all the generally useful pens that are defined somewhere would be collected. That includes pens from fontTools, RoboFab and ufoLib. From RoboFab that would be all pens that don't require a direct RoboFab dependency i.e. a module from robofab.objects or so. The rationale is that I think the pen APIs are a great way to work with outlines and thus can be very useful for people even if they don't use ufoLib or RoboFab. I did so already in some occasions. For the plan of splitting ufoLib and RoboFab I also think it could be a good idea to make a "penBox"` that contains all pens that are neither dependent on RoboFab nor on ufoLib. Instead RoboFab and ufoLib would import penBox.

[adrientetar]

Also putting ufoLib into a separate repo would be a thing too. Nothing we can't do in our forks, of course.

[moyogo]

@adrientetar That’s addressed in #51. Sorry I should have pinged you guys.

[typesupply]

The pen protocol management has moved to ufoLib and ufoLib should be moved to be owned by the UFO spec (see my comment on #51). Thus, the protocol would become standardized or at least quasi-standardized until someone has time to fully document it. The only open question for me is: what pens should go along with it? As @graphicore wrote, there are some very good reasons to move any general use pens out of RoboFab. Which ones those are and where they should move is the real issue. I can kind of see them moving along with ufoLib, but it also seems kind of weird to make them officially part of the UFO spec. So, hm.

[adrientetar]

What do you think about putting them in fontTools? fontTools is basically, the things that you need to work with fonts regardless of whether you use robofab or defcon or sth. else. So I think it makes sense to put them in fontTools. Also, fontTools is already a dependency of robofab and some robofab pens subclass the base pens contained in fontTools so it makes all the more sense to me.

[typesupply]

fontTools has pens, but not pointPens. If fontTools wants to support pointPens too, then, yeah, that might be a good home for some of the more versatile pens. I still think that the pointPen protocol (any maybe the pen?) should be owned by the UFO spec.

[twardoch]

I'm in favor of aggregating the pens (including pointPens) into a separate package. The Pen API is great but I think the current dispersion across packages is somewhat confusing. A separate penBox package could be oragized a bit like the robofab "worlds", i.e. additional formats or apps could contribute their code there.

[adrientetar]

fontTools has pens, but not pointPens.

I had not realized that. Is there any reason why pointPens are not fitting for fontTools?

[twardoch]

Also, a unified package would make it easier for people to decide where to contribute code. For example, if someone writes a "remove-overlap-pen" or a "cubic-to-quadratic-pen" or some other "effect" pens, currently it is not obvious where such contributions should be made.

[twardoch]

By "effect" I meant "filterPen". I really like the principle behind filterPen, and I think boolean ops (at least remove overlaps), cubic-to-quadratic (including some clever optimizations) and other such operations could be well-implemented as pens or filterPens, as a rule.

I think even an autohinter should use the pen protocol (possibly extended). I think the pen protocol is the single best invention of the RoboFab team, and I think it should have a life of its own.

[anthrotype]

Thank you Tal. I also think it would be good to keep the "concrete" pens in a unified package, e.g. penLib, penTools or whatever. Except for the AbstractPointPen class, they should be kept separate from the UFO spec reference implementation (which will become ufoLib).

Is there any reason why pointPens are not fitting for fontTools?

It's not that the pointPen protocol isn't "fitting" for fontTools; simply it hasn't been "fitted" yet. But it shouldn't be hard to add a drawPoints method to fonttools' Glyph objects. I'll see what I can do.

[adrientetar]

Cool, Cosimo. With this I think we can move the pens to fontTools, which to me makes most sense.

[twardoch]

I don't think it makes sense to move pens to fontTools. fontTools is 98% related to native font formats, primarily SFNT (.otf, .ttf), and most of the fontTools functionality is relatively low-level, and is not in any way tied to UFO, RoboFab or other such higher-level formats. Pens are pretty much at the opposite end of the spectrum where fontTools mostly operates in.

[adrientetar]

It uses quite some pens already to work with font data.

[graphicore]

Like @twardoch I'd vote for a separate package, a penBox. I think fontTools should be depending on that.

… move the pens to fontTools, which to me makes most sense.

Not to me. Simply because pens can be useful even if you don't work with fonts. Just to draw vector graphics.

[anthrotype]

I think Adam is right. We could add the drawPoint methods to fonttools glyphs, so that they can work with point pens as well as segment pens. But that doesn't mean all the robofab pens should become part of fonttools.

[twardoch]

@adrientetar fontTools does use pens, but it also uses XML, Python strings and lists, and other things. I feel having pens in fontTools is like having a color management library sub-packaged into an HTML parser simply because at the time of the original writing it was a convenient place to put it.

In the 1930s, the English actress Edith Evans bought a Renoir painting. A friend was having tea with her and said, “Edith, have they delivered the Renoir?”. “Yes. Yes, it’s here,” she said. “Well, can I see it?”, asked the friend. Edith Evans pointed to a corner spot on the wall, quite low and half behind a curtain and said “It’s over there!”. The friend lifted the curtain a bit, looked at the painting and said, “Well, it’s lovely, Edith, but why did you hang it there?”. She said, “There was a hook!”

To me, having pens in fontTools is a bit like that :)

[benkiel]

Another vote to put pens into it's own thing.

[twardoch]

In other words, I'd prefer to add an external dependency to fontTools so that the new 'glyphPens' (or whatever) package is needed for fontTools to build, possibly leaving a minimal fallback implementation inside fontTools, and have 'glyphPens' to live in a separate package. The pens conveptually don't really belong to fontTools or roboFab completely, but are a shared protocol, one that hopefully will enjoy a new set of contributions if refactored. Of course some backwards-compatible hooks ("redirects" of sort) could be added so that existing code doesn't break.

[twardoch]

'roboPens" might be a sensible package name to underline their heritage. Or 'glyphPens', which what they really are.

[anthrotype]

There's no need to add any new dependency to fontTools. If one wants to implement a pen that subscribes to the PointPen protocol, one does not need to from roboPens import AbstractPointPen. One just needs to create a new class which has the four basic methods of any point pen: beginPath, endPath, addPoint and addComponent. If it walks like a PointPen, and quacks like a PointPen, it is a PointPen.

[twardoch]

Also from the organizational point of view: the pens have a different lifecycle that the rest of fontTools, and while @behdad is a great maintainer of fontTools, I don't think he's the right person to maintain the pens. The pens have indeed a tighter link to UFO. Perhaps one of the "active newcomers" (not in a negative way) like Cosimo or Adrien would be willing to maintain it? Someone who's closer to the "Robo/UFO" community, and who actually understands/uses pens extensively. I imagine that Tal wouldn't really jump up with joy at the prospect of maintaining yet another package. :)

[twardoch]

@anthrotype Good point :)

[behdad]

Not in fontmath?

[behdad]

I think PointToSegmentPen and SegmentToPointPen should live in fonttools, and the rest outside.

[graphicore]

I think PointToSegmentPen and SegmentToPointPen should live in fonttools, and the rest outside.

If we have a library that hosts both, point pens and segment pens, why should the adapters between these live somewhere else?

[behdad]

I think PointToSegmentPen and SegmentToPointPen should live in fonttools, and the rest outside.

If we have a library that hosts both, point pens and segment pens, why should the adapters between these live somewhere else?

If we do, agreed.

[anthrotype]

maybe because the AbstractPen (the protocol for all segment pens) was originally defined in fontTools.pens.basePen module. But in theory the latter could also become part of a shared pens library in which both segment and point pens coexist. Again, with duck typing one wouldn't need to depend on other packages if the only requirement is an abstract class.

[anthrotype]

@typesupply @adrientetar any updates on migrating robofab.pens to roboPens/penBox?

[adrientetar]

as I said here, my current concern would be to move SegmentToPointPen and PointToSegmentPen to ufoLib, since ufoLib "owns" the pointPen protocol per what Tal said (which I think makes a lot of sense), PointToSegmentPen is the only remaining dependency of ufoLib to robofab.

[anthrotype]

if ufoLib doesn't want to maintain any dependency from penBox (or whatever), then it could implement its own converter pens (there's nothing fancy about them). Otherwise, it could simply import them from penBox once it goes live, instead of robofab.

$ git grep robofab
Lib/ufoLib/glifLib.py:78:               from robofab.pens.adapterPens import PointToSegmentPen

[typesupply]

any updates on migrating robofab.pens to roboPens/penBox?

I think a new repository for non-environment specific pens is a very good idea.

I've been thinking about the dependency issues. I think the converters should live inside of ufoLib. That makes sense to me as it isolates the implementation of the PointPen infrastructure and protocol. As you pointed out, this wouldn't necessarily require a dependency on ufoLib for penBox since the PointPen protocol is pretty simple and doesn't require any abstract infrastructure. (Whereas the Pen protocol does for handling quad <-> cubic conversion.) So, that's easy enough. The UFO spec should officially document the PointPen protocol somewhere to make this non-dependency clear. Where that documentation goes, I'm not sure.

[anthrotype]

any news on moving robofab.pens to a standalone package? It's been a while now.

[behdad]

any news on moving robofab.pens to a standalone package? It's been a while now.

I've thought about this a bit as well; let's talk about it in person next week.

[davelab6]

Any news? :)

[jackjennings]

I'd be willing to tackle creating a fontPens library (not particularly correct but in line with fontTools and fontParts), if there's a clear list of which pens should be extracted from where. (@typesupply)

[typesupply]

I'm all for this after working on fontParts. Here's a list off the top of my head:

From RoboFab's pens:

• FlattenPen
• AngledMarginPen This looks useful, but I don't know if it is used much.
• MarginPen Same comment as above.
• ReverseContourPointPen

From defcon's pens:

• ClockwiseTestPointPen See this for an improvement.
• DecomposeComponentPointPen

[twardoch]

Please see the discussion on https://github.com/behdad/fonttools/pull/539 I've provided a list of pens there that I could find on Github.

I've recently used AngledMarginPen :)

[benkiel]

From yesterday's work on defcon, MarginPen is what provided glyph.box

[jackjennings]

Here's an exhaustive list of pens in the ufoLib, fonttools, defcon and robofab packages: https://gist.github.com/jackjennings/074c87230a25b8939428

I checked through @twardoch's very complete list of pens, and for this initial effort it seems more manageable to untangle the mutual dependencies of these 4 libraries rather than try to consolidate all of the pens written.

I'd like to get an up or down on all of pens on the linked checklist.

Pens that should be excluded are those that have hard dependencies on their parent package, or pull in any additional dependency—the fontPens package shouldn't have any additional dependencies so that it's easy to pull in to any project. At this point, I'm less interested in degree of usefulness as an eliminating factor, though it could be in the future.

The "abstract" pens are duplicated in a few places, and which of these should be the extracted implementation (if they're not identical) will need to be determined.

It would be helpful for some of the maintainers of these repositories to chime in with any information in regards to which pens from their package wouldn't be useful (ever) outside the context that their used in.

[anthrotype]

The "abstract" pens are duplicated in a few places, and which of these should be the extracted implementation (if they're not identical) will need to be determined.

The AbstractPen belongs to fontTools and AFAIK is not duplicated anywhere else. The AbstractPointPen, along with the converters between the two protocols, should belong to ufoLib package, as I think Tal has suggested.

the fontPens package shouldn't have any additional dependencies

So, you want to move all the pens, both the "abstract" and "concrete" ones. I thought we would keep the abstract pens inside the original packages, and only move their concrete implementations to the new package. Requiring fonttools or ufoLib doesn't seem too much (that's already the case), and plus one doesn't really need to import/subclass these abstract pens to implement a pen that works like a pen.

@graphicore above suggested to call it "penBox" instead of font-something

because pens can be useful even if you don't work with fonts. Just to draw vector graphics.

I personally don't have a strong opinion. I'm happy with "fontPens" if you prefer that, as long as this move happens and the dependency on robofab is lifted.

[anthrotype]

Let's not forget that the main reason we initiated this move was to take "the good part" of robofab out of robofab to a place where it could be maintained and developed further (UFO3, Python3, new font editors, etc). This has already happened for the UFO read/write stuff (ufoLib), and it's an ongoing effort with fontParts as regards the font objects and the scripting API. We could also consolidate both the robofab and fonttools pens to a separate package if you like, but IMO the priority should be given to the robofab pens.

[jackjennings]

The motivation for moving the abstract pens outside of their original libraries is that we'd be able to eliminate a sibling dependency, replacing it with a hierarchical dependency which is easier to reason about and manage.

If anything, I think that there is more value in moving the abstract classes into a new dependency.

If AbstractPen is left in ufoLib and I wanted to create a library that implemented the pen protocol, I'd have to depend on ufoLib, which is a much heavier dependency than a hypothetical library containing only pens. If the library is for something other than fonts, it would seem counterintuitive (to me) to include a UFO-centric library. ufoLib seems like it should focus on dealing with UFO attributes, and delegate most of the generic pen protocols to a separate package.

I'm also not sure that the pen protocol really belongs as a part of the UFO spec. The UFO spec is a file format spec that doesn't dictate how to manipulate the underlying data-structure at a high level. I think that this is good, since other libraries (possibly in other languages)should be able to be UFO compliant without implementing the pen protocol, even if that's a convenient part of how the current robofab/fonttools/ufoLib ecosystem works.

The pens that are duplicated between different packages are:

• AbstractPointPen
• BasePointToSegmentPen
• PointToSegmentPen
• SegmentToPointPen
• GuessSmoothPointPen
• Bounds Pen

Maybe a few others…

[anthrotype]

if I wanted to create a library that implemented the pen protocol, I'd have to depend on ufoLib

you don't. Your can define a class MyPen(object) which implements all the lineTo, moveTo etc. and no one should bother checking isinstance(pen, AbstractPen), as Python will duly raise AttributeError if that's not the case.

I guess the "ownership" of the abstract protocol has more to do with who should be responsible for designing and possibly updating those APIs.

The pens that are duplicated between different packages are: ...

All the pens you mention were moved from robofab/ufo3k branch to ufoLib, and slightly updated to make them work with python 2 and 3. So the ufoLib version should ideally be taken as reference. The BoundsPen was originally (and still is) inside fontTools and was later copied to robofab, modified to ignore contours with single points which robofab (UFO2) treats as anchors.

[jackjennings]

Yes, you can create a generic object subclass, but in practice nearly no one has actually done this.

[anthrotype]

all user-defined Python classes are, directly or not, children of the object type.

EDIT: or at least, all "new style" classes.

[jackjennings]

Yes, I understand that, and I understand how duck typing works. I mean that most of the pens that I just reviewed do not subclass object, but do subclass one of the abstract or base classes that have been provided by various libraries.

[anthrotype]

If they don't explicitly subclass object, then they will use "old-style" classes in Python 2, and (implicitly) "new-style" classes in Python 3, which will have some impact on e.g. super() and the like.

But what I meant was that, for the regular pen usage, it doesn't really matter whether the new pen you define is old-style, new-style or inherits from any of the abstract classes. It simply needs to define all the required methods that are expected from such a pen.

[jackjennings]

Again, I understand that. My point is that if you do want to subclass the abstract or base versions of Pen or PointPen, does it really make sense to have to require a library that is not specifically and narrowly related to creating pens?

[anthrotype]

I guess not. If do not want to depend on them, you don't subclass them.