The Future of GrokProject

[clozinski]

First zope.org went down and broke things. Then PyPi switched from http to https. and broke things. What will go wrong next?

@prsephton and I are still on the older release. Is that 1.5.5? Over the slow summer, I worked on upgrading to the current release. It is a lot of work, because I have touched so many Zope libraries. Nor is there any huge motivation to upgrade.

Then the upgrade to Python 3 will be even more difficult. I believe that Zope works on Python 3. But the grok and megrok libraries are still on Python 2. Google Cloud is still on Python 2.

Then there is the whole problem with zope.publisher. Grok, Pyramid and Plone(?) use a permissions on views model. Zope.pulisher uses a permissions on objects model. That whole section needs to be cleaned up. Evidently Cromlech did that clean up, but I have still not figured out what that code does. https://github.com/Cromlech

One last thought. This one is really out there. This whole word is really a huge tangle of complexity. Github is hard to understand. It does not preserve history. PyPi is easier. The setuptools docs say that they have a ton of options. Then there are eggs. And buildout. And then we have all these versions files which tell what works with what.

I understand that systems evolve in certain directions, think Microsoft Windows, but certainly the whole thing could be easier. Of course the first problem is that we are storing everything on 50 year old file systems, instead of in a proper object database (the ZODB).

Oh well. I wonder if anyone has a clear vision of how to make this whole huge stack simpler. There is an interesting plone product which merges all of the zope namespace packages back into a single tree. Is that a step in the right direction? Do we really need a whole bunch of different grok libraries, with all their different versions? Is anyone else using any of the grok libraries in a a separate project?

And a huge thanks to @prsephton for everything he has done.

At least we are not working in Javascript!

[prsephton]

Hey, Chris, perhaps things are not as dire as they look!

First zope.org went down and broke things. Then PyPi switched from http to https. and broke things. What will go wrong next?

What is life without a bit of excitement, eh? You are right that what development requires is change, but what users require is stabiity. The two needs are in dichotomy. Basically, nothing can improve without change, but stability cannot by definition be achived if things are in continuous flux. That is why we have repositories full of versioned eggs, and why we have facilities for pinning versions. That way we achieve both goals.

Which is why I was so inflamed by the idea of PyPI unilaterally dropping support for HTTP. So many older projects would still be referring to http://pypi... and this would break stable production code. The motivation was sound enough: the potential for man in the middle attacks. The remedy though was ridiculous, and wholly from the perspective of developers while ignoring the need for stability. A better remedy would have been to leave the decision (HTTP vs HTTPS) in the hands of the project maintainers, and gradual migration.

Python 3 will happen. The good news is that porting from 2.7 to 3.6 is not as hard as it used to be, as Python 3.6 made several allowances which aid the task. Mainly, its a matter of knowing when you are dealing with byte strings and when it is unicode so you know when to call .encode() and .decode() to convert, or where to use from futures import xxx. Its a bit more cumbersome than Python 2 code, but not by much.

Then there is the whole problem with zope.publisher. Grok, Pyramid and Plone(?) use a permissions on views model. Zope.pulisher uses a permissions on objects model

Regards permissions, from what I understand, permissions are always on objects; it's just that the objects the publisher accesses happen to be view objects. zope.security.management.checkPermission(permission, object, interaction=None) does not care whether the object is a view object or a model. 'interaction' is a thing that normally represents the principal (or current user) while serving a request. If interaction is none, it uses the current one. Seems rather simple to me.

The setuptools docs say that they have a ton of options. Then there are eggs. And buildout. And then we have all these versions files which tell what works with what.

I have to admit that this remains one of the most frustrating things about the Python ecosystem (and I believe many out there would agree with me!). There was the whole setuptools - distutils - setuptools debacle, where the original setuptools maintainer refused to allow scope creep to destroy backwards compatibility. Distutils came long to add some features, but was limited in other ways, and eventually setuptools threw caution to the winds and now we have pip, easy_install, eggs that work everywhere except Windows, wheels (binary) that only really work on Windows, and have a hope in hell of working reliably elsewhere. Buildout 1.5 was amazing. Then because setuptools deprecated certain behaviour it depended on, it stopped being quite so amazing, and virtualenv, which IMO tried but never quite managed to do what buildout already did really well became a standard built into setuptools, which meant that bootstrap.py could no longer upgrade setuptools in place after version 33.1.1, effectively pinning that mechanism to an old setuptools version.

Does the above read like a disaster? It should.

Buildout still does some things really really well, despite being royally pissed on by the setuptools crowd, and I vastly prefer it over virtualenv.

One last thought. This one is really out there. This whole word is really a huge tangle of complexity. Github is hard to understand.

Actually, github does have history; you can easily view any snapshot that has been tagged, or retrieve a version as it was for a specific date. It's a bit more complicated because people are more comfortable using branches with git, where svn would often barf unless you were very very careful with branches. As with anything really, the more comfortable you become using it, the less intimidating it seems.

There is an interesting plone product which merges all of the zope namespace packages back into a single tree. Is that a step in the right direction?

Oh, good lord, no! That is what we had before the ZTK was broken up into manageable bits!

Peace!

[clozinski]

Does the above read like a disaster? It should. Yes. After the first read, it did not all stick in my brain.

Buildout still does some things really really well, despite being royally pissed on by the setuptools crowd, and I vastly prefer it over virtualenv.

Buildout makes a lot of sense to me. In contrast I barely understand setuptools. And am not that motivated to figure it out. Having buildout and setuptools and eggs and PiPy seems too confusing.

How about is we just used github and buildout and the setup.py files and a little bit of extra code. That would be a huge simplification and make grok administration much easier to understand. Any packages which needed to come from PyPi could be installed either in the Python environment or in a virtual environment. Maybe in a Docker container. In fact I would think that Docker changes this whole dynamic.

Would it be difficult to get rid of setuptools, eggs, and PyPi. I would think that setup.py tells us what packages and versions we need. Buildout.cfg tells us what to create. And what versions to use. And even where to get them. So why are we even using setuptools? And if it makes sense to get rid of it, then there must be a some product that has done so. It is not that important to me to actually do this. But it is important to me to understand how to simplify this huge software stack which I am happily teetering on.

I always joke that Grok needs to go on a Paleolithic diet. Where he looses lots of excess fat.

[prsephton]

Buildout still does some things really really well, despite being royally pissed on by the setuptools crowd, and I vastly prefer it over virtualenv.

Buildout makes a lot of sense to me. In contrast I barely understand setuptools. And am not that motivated to figure it out. Having buildout and setuptools and eggs and PiPy seems too confusing.

Oh boy!

Sorry for perpetrating a misunderstanding, but buildout uses setuptools. When I say buildout was "royally pissed on", I meant that setuptools development made decisions which negatively impacted what buildout was able to do and broke some existing functionality, in favor of things like virtualenv and pip.

Buildout cannot exist without setuptools (or at the very least distutils).

All these things work together in synergy. The egg format is "understood" by Python itself so that packages can nicely integrate with one another; Setuptools (pip/easy_install) locates eggs from PyPI, downloads and installs them. Buildout manages versioning and pluggable recipes that do intelligent things at install time.

So, when you say "Would it be difficult to get rid of setuptools, eggs, and PyPi" that makes no sense to me. Similarly, github is a source control repo, and cannot replace what PyPI does. Setup.py is only understood by setuptools/PyPI and buildout could not (should not) replace it.

Virtualenv OTOH is in my opinion a poor alternative vs. buildout for providing an isolated runtime environment.

We should not IMHO be trying to re-invent wheels, but rather invest our time in seeng how the existing cogs can better be made to fit together, and to run more smoothly. If I criticise, it is not to diss the project and devs, but to point out where I perceive flaws to be, and I am the first to acknowlege that my own criticism is quite likely missplaced, but that is the way we learn, not so?

[clozinski]

Thank you.

That raises the obvious question Why was Buildout 1.5 better than 2.0.

Rather than ask the question let me answer it. https://github.com/buildout/buildout/blob/master/HISTORY.rst Scroll down to the 2.0 release.

@prsephton I am curious which buildout bug you are wrestling with?

And of course my pet peeve, is that buildout tells you there is a version conflict, but does not tell you which requirements path let to that version conflict.

For beginners I ran across an interesting comparison of buildout and virtual env. Virtual Env just controls which Python version you use. Buildout can be used to configure a system for anything, not just Python software. Buildout really makes huge sense to me. I will read more on the buildout github page.

I still think that this whole software stack is way way too tall, and needs big simplifications.

[prsephton]

That raises the obvious question Why was Buildout 1.5 better than 2.0.

Exactly. Quoting from your linked page:

"Buildout no-longer tries to provide full or partial isolation from system Python installations. If you want isolation, use buildout with virtualenv, or use a clean build of Python to begin with."

Sorry to burst the bubble, but virtualenv + buildout == disaster. If you install anything at all in your virtualenv, the current buildout reports a version conflict unless egg versions between buildout and virtualenv are identical. For example, buildout insists on setuptools v33.1.1, as this is the last version with the ability to self-upgrade. The current setuptools version is 36.x.x. The moment you use virtualenv (eg. in Travis), setuptools version conflict is reported and buildout barfs.

You can run buildout again to complete the install, but that little hickup leads to silly things like (bin/buildout || bin/buildout) in the Travis script.

Consider for a moment; We have a buildout based Grok installation. Now, for production code, in order to achieve isolation from system Python, we are required to do the little virtualenv source bin/activate raindance every time we start the server. Who actually thought this to be a good idea?

Another place this raindance becomes cumbersome (to say the least) is when calling python-console (the interpreter) from within an IDE such as Eclipse or PyCharm for development/debugging.

Isolation should be something provided by buildout. The original motivation for isolation was here. It remains a very valid issue, and one only partially solved by virtualenv.

Another issue, is that buildout cannot install wheels (and as far as I can see never will). Due to the disparity in system library versions between Linux distros, and versions of any given distro, getting binary stuff to install reliably from wheels is probably an impossibility. So from my perspective, the people who dreamt up the wheel format as a replacement for eggs were probably smoking some rather potent stuff. I don't see the egg format going anywhere soon.

Interestingly, the egg format was born from a need, while a PEP gave birth to wheels. Assuming that for Linux at least, binary wheels are not an option, what advantage does a wheel have over the egg format? From what I understand, wheels have potentially more meta data and pip can install a wheel whereas pip cannot install an egg.

Anyway, we are not going to solve Pythons distribution woes today; many better minds have tried. I have to agree that things could and should be way simpler.

[jamadden]

buildout cannot install wheels (and as far as I can see never will).

@jimfulton added support for wheels back in March. See https://pypi.python.org/pypi/buildout.wheel/

[prsephton]

@jimfulton added support for wheels back in March

Nice. That must have been a non-trivial task. Good to hear I am behind the times!