Expand the scope of this project

Hi. First of all thank you for putting the time into developing this plugin.

Would you be interested in expanding the scope of this project and turn this into something similar to typeshed, but for packages relying on zope.interface?

My main pain point with this plugin has been that it ships with outdated and incomplete stubs, without any automation to help you maintain them. I have created complete up-to-date stubs for zope.interface that I would be happy to contribute back. I also found a small bug in the mypy plugin which I am happy to contribute a fix for.

My vision would be to add a lot more automation and decouple the plugin from the stubs, so you eventually have the option to use an older version of the stubs matching your installed version of zope.inteface/zope.schema etc. rather than having to hope that everything matches. This would also easily allow us to provide stubs for additional packages. E.g. I have some stubs for zope.sqlalchemy lying around I could contribute.

It would still be a monorepo like typeshed, but it will deploy multiple distributions and crucially the stubs will automatically be updated and deployed when new versions of zope.interface are published and stubtest still passes.

Similar automation could be added for the plugin to update the requirement pins if the tests still pass with a newer version of mypy.

Ideally we would also move to an organization and invite additional maintainers/collaborators, so it doesn't all depend on one person keeping up with things.

I would also like to see if we can significantly speed up the mypy plugin by using mypyc to compile it. Since the plugin can slow down things significantly in large code bases, compared to mypy without the plugin. mypy itself is also compiled with mypyc, so I feel like it should at least help a little bit.

I also have some ideas for supporting other type checkers by providing "lite" versions of the stubs. Which automatically replace all interface definitions with equivalent protocols and all @implementer decorators with explicit sub-classing of the protocol. That should model the behavior of interfaces sufficiently in the existing type system for basic interactions with zope.interface (i.e. you can use interface types in type hints and it mostly features the correct subtyping rules, unless it happens to be a marker interface without any attributes, but you can't define new interfaces). This would be easy to automate, but it might be more challenging to properly test.

/cc @icemac In case you'd be interested in hosting such a project as part of the zopefoundation organisation. Although in terms of the licence, I would stipulate that it must remain MIT or at most switch to Apache 2.0 (to match typeshed), so it's easier for third parties to contribute.

Hi @Daverball, sorry for late reply. Trying to respond to all your points here.

You are right, the stubs that this plugin is shipped with are indeed incomplete and possibly very outdated. There was an attempt to autogenerate new stubs from the latest versions of the lib, but it stalled because zope uses too much python runtime magic for static analysis tools like stubgen to comprehend. So current stubs are kind of MVP for plugin to be useful. I think any improvement in this area would definitely be beneficial for the users.

I think it might make sense to separate stubs from the plugin if that makes maintenance easier, but I believe the plugin itself makes certain assumptions about the contents of zope.interface and zope.schema, so the changes to both needs to be coordinated. In general the typeshed-like repo sounds like a right way to maintain stubs

Compiling the plugin with mypyc also sounds interesting. The codebase that I myself run mypy on is fairly large and we ended up using mypyd, which gives adequate performance during development, but improving the cold start time would definitely improve the quality of life.

Regarding treating interfaces as protocols - well, I tried exactly that and it didn't work :) That was my first idea in fact. I don't remember all the details, but I think I was stuck around class inheritance. You can get very complicated inheritance hierarchies when you combine interface inheritance trees with implementation inheritance trees. Mypy itself also treats protocols in a very special way and have a very hard opinions on how protocols work and that makes this approach even harder.

In general, the project is mostly in maintenance mode right now, mainly because it is reached the "good enough" state and covers all the basic needs for my project and the team, but if there's an interest to improve things, I'm not gonna resist :)

I think it might make sense to separate stubs from the plugin if that makes maintenance easier, but I believe the plugin itself makes certain assumptions about the contents of zope.interface and zope.schema, so the changes to both needs to be coordinated. In general the typeshed-like repo sounds like a right way to maintain stubs

Yes, as long as you always unit test the stubs together with the plugin I see no issue with shipping them separately. This will also allow us to make informed decisions about when to add a minimal version to the plugin/stubs install requirements. Since the plugin will have to require at minimum the zope.interface stubs and every other stub will at minimum require the plugin in order to work.

Regarding treating interfaces as protocols - well, I tried exactly that and it didn't work :) That was my first idea in fact. I don't remember all the details, but I think I was stuck around class inheritance. You can get very complicated inheritance hierarchies when you combine interface inheritance trees with implementation inheritance trees. Mypy itself also treats protocols in a very special way and have a very hard opinions on how protocols work and that makes this approach even harder.

Yes, I don't expect that to work for every use-case. But I've used this approach with moderate success in a couple of projects that make only light use of zope.interface i.e. they only use fairly specific interfaces where the runtime difference between a protocol and an actual interface becomes negligible. So I thought it would be an interesting experiment to provide auto-generated protocols for these types of projects, both for projects using mypy to get free speedups and for other type checkers, which currently have no typing support at all. But I wouldn't expect it to work for frameworks like pyramid, which makes very heavy use of zope.interface and uses marker interfaces amongst other things.

I have two versions of my zope.sqlalchemy and transaction stubs. The Protocol version works just fine in a relatively large codebase that doesn't rely on zope.interface too much. transaction in particular is something I've seen used in many codebases, including custom data managers. Whether or not they use interfaces anywhere else.

The main danger I see with shipping two versions of every stub is that people might install the wrong version or have both versions of a stub installed, so it depends on installation order which one you get. But it should be possible to mitigate that to some degree by being very explicit in the package description about when to use which version.

In general, the project is mostly in maintenance mode right now, mainly because it is reached the "good enough" state and covers all the basic needs for my project and the team, but if there's an interest to improve things, I'm not gonna resist :)

Happy to hear it. With enough automation and test coverage I think it's possible to keep the quality high with minimal maintenance effort. If we add one or two additional maintainers there should be fewer cases where people are waiting for the plugin or stubs to update, since most zope packages themselves are in maintenance mode and don't change much these days, most updates should be handled automatically by something like stubsabot and stub_uploader, although we would probably either need to contribute more configuration options or maintain our own fork if we wanted to use it. I assume it is a little too specific to typeshed right now to be usable out of the box, but I haven't done a deep dive into the code yet to verify that.

Shoobx / mypy-zope

Expand the scope of this project #122