search

tbarnetlamb / hyphen

hyphen - access Haskell modules from Python
GNU General Public License v2.0
91 stars 9 forks source link

Python 3.9.6, ghc 8.10.7, NixOS 21.11 (79c7b6a353e): hyphen.HsException: Could not find module ‘Data.Hashable’ #15

Closed exarkun closed 2 years ago

exarkun commented 2 years ago

hyphen seems like a very cool project. I wanted to try it out just to get a feel for its behavior. I was unable to build a working version though. Here's a transcript of my effort:

[exarkun@magnon:~/Work/python/hyphen]$ nix-shell -p python3 'haskellPackages.ghcWithPackages (hs: with hs; [ hashable unordered-containers ghc-paths ])'

[nix-shell:~/Work/python/hyphen]$ python --version
Python 3.9.6

[nix-shell:~/Work/python/hyphen]$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.10.7

[nix-shell:~/Work/python/hyphen]$ python hyphen/build-extn.py 
[ 1 of 12] Compiling HyphenBase       ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HyphenBase.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HyphenBase.o )
[ 2 of 12] Compiling HyphenKinds      ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HyphenKinds.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HyphenKinds.o )
[ 3 of 12] Compiling HyphenTyCon      ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HyphenTyCon.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HyphenTyCon.o )
[ 4 of 12] Compiling HsType           ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HsType.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HsType.o )
[ 5 of 12] Compiling HyphenUnify      ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HyphenUnify.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HyphenUnify.o )
[ 6 of 12] Compiling PythonBase       ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/PythonBase.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/PythonBase.o )
[ 7 of 12] Compiling HsObjRaw         ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HsObjRaw.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HsObjRaw.o )
[ 8 of 12] Compiling HyphenWrapping   ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HyphenWrapping.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HyphenWrapping.o )
[ 9 of 12] Compiling Pythonate        ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/Pythonate.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/Pythonate.o )
[10 of 12] Compiling HyphenExceptions ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HyphenExceptions.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HyphenExceptions.o )
[11 of 12] Compiling HyphenGHC        ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/HyphenGHC.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/HyphenGHC.o )
[12 of 12] Compiling Hyphen           ( /home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/Hyphen.hs, /home/exarkun/Work/python/hyphen/hyphen/lowlevel_inter/Hyphen.o )

/home/exarkun/Work/python/hyphen/hyphen/lowlevel_src/Hyphen.hs:1030:5: warning: [-Wdeprecations]
    In the use of ‘defaultCleanupHandler’ (imported from GHC):
    Deprecated: "Cleanup is now done by runGhc/runGhcT"
     |
1030 |     GHC.defaultCleanupHandler flags $ return 0
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^
Linking /home/exarkun/Work/python/hyphen/hyphen/hslowlevel.cpython-39-x86_64-linux-gnu.so ...

[nix-shell:~/Work/python/hyphen]$ python
Python 3.9.6 (default, Jun 28 2021, 08:57:49) 
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hyphen
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/exarkun/Work/python/hyphen/hyphen/__init__.py", line 23, in <module>
    caches.precache_modules([
  File "/home/exarkun/Work/python/hyphen/hyphen/caches.py", line 40, in precache_modules
    module_cache.update(hslowlevel.import_lib(*names))
hyphen.HsException: Could not find module ‘Data.Hashable’
Use -v (or `:set -v` in ghci) to see a list of the files searched for.
exarkun commented 2 years ago

In an strace of the Python process doing the import, I do notice:

newfstatat(AT_FDCWD, "Data/Hashable.hs", 0x42006edc50, 0) = -1 ENOENT (No such file or directory)                                                           
newfstatat(AT_FDCWD, "Data/Hashable.lhs", 0x42006edd80, 0) = -1 ENOENT (No such file or directory)                                                          
newfstatat(AT_FDCWD, "Data/Hashable.hsig", 0x42006edec0, 0) = -1 ENOENT (No such file or directory)                                                         
newfstatat(AT_FDCWD, "Data/Hashable.lhsig", 0x42003d1010, 0) = -1 ENOENT (No such file or directory)
tbarnetlamb commented 2 years ago

Thank-you for your interest in hyphen! I'm sorry that building isn't going as smoothly for you as I would have hoped.

It looks like the problem is that Data.Hashable has not been built in a way that is visible in the environment to every running program. In particular, it seems as though it is visible to ghc (invoked via build-extn.py at build time), otherwise the build would fail. But it is not available to the python3 process you invoke which imports hyphen. Reading this page seems to suggest that the way haskellPackages.ghcWithPackages works is actually a little subtle: it wraps various haskell related executables (like ghc, ghci, runhaskell and so on) with new versions that set magic environment variables so that the packages nix has installed can be found. My guess is that the problem arises because those magic environment variables are not being set for the python3 process, so hyphen (living inside that process) can't find the binaries of the packages nix compiled.

To test my hypothesis, could you please try invoking a python3 process inside ghci, as follows:

$ ghci
GHCi, version X.Y.Z: http://www.haskell.org/ghc/  :? for help
Prelude> :! python3
Python 3.9.7 (default, Oct 25 2021, 01:04:21) 
[GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hyphen

If I'm right, then the magic environment variables will be set in the ghci process (because ghci is one of the program names that nix monkey-patches so that they pick up the magic environment vars), and thus, hopefully, in the python3 subprocess that I invoke inside it... which would mean that hyphen can be imported.

[The one other potential problem I can foresee is that you may not be building dynamic versions of the libraries, as described in BUILDING. I'm not sure how to get nix to do that... we can cross that bridge when we get to it though...]

[Also: I'm not sure how attached you are to using nix to build the background packages. If it's easy for you to switch to cabal (as per the BUILDING file) or stack then I can try to advise you to get that working. If you'd prefer to stick to nix, I can try my best to help with that too. (If it's helpful, the CI configuration contains example scripts to do the whole build through stack as well as cabal.)]

Thanks again for your interest in hyphen!

exarkun commented 2 years ago

Thank-you for your interest in hyphen! I'm sorry that building isn't going as smoothly for you as I would have hoped.

Thanks for the quick reply, and no worries about the build problems. I recognize this is a work in progress. :)

To test my hypothesis, could you please try invoking a python3 process inside ghci, as follows:

The evidence is consistent with your hypothesis:

$ ghci
GHCi, version 8.10.7: https://www.haskell.org/ghc/  :? for help
Prelude> :! python3
Python 3.9.6 (default, Jun 28 2021, 08:57:49)         
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hyphen      
>>> hyphen.hs.Data.ByteString.ByteString.append(b"foo", b"bar")                                                                                                                                                                                           
b'foobar'

And your explanation makes sense to me as well (based on my general understanding of the use of wrapping in NixOS, but I haven't tried to learn about the ghc-related wrappers specifically).

Speculating about how I might have helped myself here ... I could have tried to learn more about the environment ghcWithPackages gives me and then maybe made the leap to Python3 being missing the ghc-related wrapping. As I filed the ticket, a question I had in the back of my head was about how hyphen actually works ... what does it need to discover at runtime vs compile time? Now it occurs to me to grep for ghc-pkg and I see there's a great comment in hyphen/loading_source.py that mentions it. I'll have to take some time to digest it, probably (I'm more familiar with Python and Nix than I am with how ghc works).

Anyhow, for the immediate future, I am not blocked in my attempts to play with hyphen. Thank you again.

tbarnetlamb commented 2 years ago

I'm glad to be of assistance! I'll keep the issue open for a little while longer, in case you hit any other problems. Happy haskelling/pythoning!

tbarnetlamb commented 2 years ago

Closing this ticket. Please don't hesitate to open another one if you have further questions. I'm also interested to hear any suggestions for improvements that may have occurred to you!