Rewrite

[darvid]

So I have some thoughts floating around for how I'd like to do this, and a very primitive skeleton of what I think I'd like the structure of Hydrogen to look like is in next.

I'll try not to make this too exhaustive, but I have a few goals and related use cases that would be nice to have, and easier to implement in code that isn't 2 years old, undocumented, and only half working:

• I'd like a framework that allows us to add package managers and requirements specs for different platforms or package managers at any time. I have an initial sketch of this in the next branch, but I'd like to use stevedore and a plugin/driver pattern. This should be minimal effort, once we decide on structure of the BasePackageManager and BasePackageRequirement ABCs.
• I'd like pip-tools integration. I use pip-compile and pip-sync for nearly all my projects now, with what I'd like to think of as a very maintainable pattern consisting of:
  • Adding requirements/ and requirements/src/ directories.
  • Adding a couple of Makefile targets and variables for defining different requirements files and source files. See https://gist.github.com/darvid/3c51f6e67cee0eb8101141e143a683c6
  • Direct dependencies (1st-level) are pinned in the respective src/*.in files. make requirements is run whenever requirements change, or in CI. Both source and "compiled" requirements files are stored in VCS.
  • Obviously this is a very specific and opinionated workflow, and shouldn't be the standard or default way that Hydrogen manages pip requirements. But I'd like some semblance of support for calling the pip-tools API and resolving requirements from a .in file to a .txt file. Ideally the requirements/src/* directory would go away, and you'd just have main/qa/test/whatever arbitrary groups you want in the requirements.yml file, and Hydrogen would build separate requirements files, and optionally resolve dependencies recursively with pip-tools (and generate a sweet annotated requirements file as well).
• Actual Python 3 support, sane packaging, actual unit tests, documentation on RTD, flake8, CI, etc. I can set all these up, it's not a huge amount of effort, but worth mentioning.

In terms of package managers, we should probably add plain old NPM/package.json support in addition to Bower.

/thread

[bbenne10]

Basically, this all seems sound. I'm not 100% sure that adding arbitrary package support is ENTIRELY a great idea (scope creep, imo), but the possibility of tracking OS level packages based on target OS is really appealing, so I might just say we should try it out without spending undue time on it. I believe stevedore to be a pattern that we use pretty heavily at work (~130k line Python code base that does a lot for a lot of people. Plugins are necessary), but I haven't read a formalized proposal for this idea, so I need to read over that before I offer an opinion. In general though I find an ABC formalized plugin system and a "collector" pattern that iterates over __subclasses__ to be extremely easy to reason about and even easier to implement. Once we have defined what we want the interface to look like, it'll be trivial to implement and document it.

We are starting to make good use of pip-tools as well, but I'm not the lead on that project so my exposure to the concepts is limited. I like the ideas that fuel it, but that's all I know so far.

Defining arbitrary groups of dependencies is nice, but I would only really ever make use of two (production and dev). A third (test) may be forthcoming, but I'm not sure there're too many more. Should we get bogged down in implementing an arbitrary number of these, I might suggest to drop the requirement and support an opinionated workflow that consists of these three and no more.

Defining a style guideline is extremely important in a project like this, and I find pylama a good match for myself (without the pylint integration enabled, as I find pylint much too verbose about too many things). I also find myself avoiding double quotes if I can, but that's probably because of shell and ruby/puppet where that's preferable. Thoughts?

[darvid]

Re: package support, I agree it shouldn't be a high priority. But if we can come up with a sensible abstraction that works for the majority of use cases, I think it'd be better than having hard-coded logic for only pip, bower, and npm.

Re: arbitrary groups, also agreed that allowing users to define their own groups might get unwieldy. We also have to factor in support for groups that are native to package managers like bower and npm, which have a 'devDependencies' in addition to regular dependencies. We'll have to come to some sort of decision about whether or not we care about feature parity and compatibility with the package managers we wrap -- i.e., I can see someone wanting to install only QA related jS dependencies for a CI pipeline stage without installing all of devDependencies.

In terms of style guidelines and code quality, I tend to prefer flake8 because it's maintained by the PyCQA and has neat extensions like flake8-import-order and flake8-quotes. I could be convinced to use pylama, though. I use double quotes out of habit, but I'm completely fine with switching to single, except for docstrings (PEP 257).

Pre-commit is pretty cool, too.