Use as dependency in package?

[InfamousPlatypus]

I'm trying to build a wrapper for some haskell functions to be used in python, I would like to use this package as a dependency. Is there a way to do that? I assume I could make some .batch files for each OS, but I was wondering if there was a simpler way that may have been implemented without updating the build.md. thanks!

[tbarnetlamb]

Hi! Thanks for your interest in hyphen, and sorry for the delay in replying (I was on vacation!)

I'm afraid I'm not 100% sure that I understand your ask. When you say you'd like to use hyphen as a dependency, I guess you mean for some particular packaging system. Do you have a particular packaging system in mind? Currently, all users of hyphen seem to build it from source, so I guess the direct answer to "has this already been implemented" is "no", but adding packaging should definitely be possible...

[InfamousPlatypus]

Oh, that's fine. Completely understand. I'm building a wrapper that would be nice if we could use pip to install, In the current case we would need to pull this repo from github, check what OS the user has, then use the right build commands probably in a shell script. I totally respect the decision to not add packaging, it's not the easiest thing ever, but if you want a random person's two cents, I think it would be useful and good for the project in that it would make it easier for users. If you like I can help.

[tbarnetlamb]

OK. So, for your purposes, the ask is something like "pip install hyphen works". That way you would be able to declare hyphen as a dependency for your own pip-based project. That makes sense :)

(In particular, it sounds like the "particular packaging system" is pip, or anything else that works off the pypi.org system, rather than nix, package managers associated to linux distros, homebrew, or anything like that.)

I did think about registering hyphen in pypi back when I originally wrote it, but (at the time) I felt that there were big obstacles to doing things that way. Maybe I was wrong, or maybe technology has now developed so that the obstacles have gone away. Let me talk through what I see as the problems.

Do you envisage packaging it as an sdist or as built wheels for particular OSes? Of course, ideally one would have both, but for a first stab just one might be enough. I would think the most natural goal would be an sdist, since if no wheel is available, we'll fall back on using an sdist, but not vice versa, and since it's unlikely I'd be able to build wheels for all distributions/OSes. Do you agree? Assuming you do, let me talk about the difficulties we'd face in building a reliable sdist for hyphen on pypi.

[In fact, I worry that building wheels might be actually impossible to do in a reliable way. This is because the wheel naming system assumes that a wheel compiled for the same Python version on the same OS on the same architecture will always be compatible. But for hyphen this isn't enough---you also need to make sure that the GHC version used to compile hyphen matches the GHC version that the end user is actually using. I don't think the usual tools for keeping track of binary compatibility in python extensions have a good way of handling this. Probably you'd have to do some hack like creating multiple "extensions" called hyphen-ghc9.6, hyphen-ghc9.8 and so on, then the user would have to install the Right One.]

If we're going for a source distribution, the main problem seems to be that we would then need pip to actually be able to build the extension. Back when hyphen was written, this forced one to use the old setuptools-based ecosystem to actually build the extension. This seemed to be a very serious obstacle because setuptools's Extension class seems to only support extensions that are going to be compiled using C/C++/Cython. It doesn't support invoking other compilers like GHC, which is what we need. (See here.)

Nowadays, in the post PEP 517 world, we theoretically have a much wider range of build systems available, and it's therefore possible that one of them will allow us to use GHC to build. But I had trouble actually finding anything that really supported anything other than C/C++/Cython. I looked through the list of build tools listed here (namely flit, hatch, pdm, poetry, setuptools, trampolim, and whey), and none of them seemed to allow extensions built with 3rd-party compilers. Do you know of any PEP-517-compatible build system which is (a) good quality/reasonably widely used (i.e. not likely to just go away in the future) and (b) supports using arbitrary compilers to build? If you do, I'm happy to try to port build-extn.py so that it works with that build system, which would hopefully allow packaging an sdist that could be uploaded to pypi.

[InfamousPlatypus]

It looks like Hatch and PDM have some functions that could be used to get a third party compiler, but they appear to fall just short. It looks like the easiest at this point is to package a .batch file for each OS for setup. I'm working in the quantum space, which seems to have shifted from Haskell to Python as the Langua Franca, so I would guess that if this software can become easier to add to packages we can avoid doing extra work. If you like I can figure it out on my package, share what I did and go from there?

[tbarnetlamb]

Yes, I'm afraid that was my conclusion too. I suppose we could try to ask them to extend their systems, but I'm not sure how interested they would be in a PR for such a change.

What you suggest (looking at what's best for your software, then trying to extend so that others can use the same system) sounds excellent. I would love to help in any way I can.

[InfamousPlatypus]

It looked to me like Hatch may have that as a goal, but not quite there. I'll try to keep an eye on that since that would be a far better option, but until then I will do what I can. Thanks!