Importing non Haskell2010 modules

sheepforce commented 9 months ago

First and foremost, thank you for this amazing piece of software! :+1: :slightly_smiling_face:

I'm currently trying to get it working for my project on NixOS. While I was successful at getting it running on NixOS can actually import many modules, some fail unexpectedly and I, by far, don't understand well enough whats actually going on inside hyphen. :smile:

So far something like this works:

Python 3.11.7 (main, Dec  4 2023, 18:10:11) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hyphen
>>> import hs.Data.Text
>>> import hs.Data.Vector.Unboxed

and I can also use these modules easily. Perfect!

However, some modules fail unexpectedly and I'm not so sure why. For example the RIO library:

>>> import hs.RIO
Traceback (most recent call last):
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/importing.py", line 228, in load_module
    marshall_module(data_ns, type_ns, module_to_write)
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/importing.py", line 171, in marshall_module
    data_namespace = dict(
                     ^^^^^
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/importing.py", line 172, in <genexpr>
    (k, _process_data_ns_item(k, v))  for (k, v) in data_namespace.items())
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/importing.py", line 136, in _process_data_ns_item
    return hyphen.marshall_ctor.get_marshalled_dacon(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/marshall_ctor.py", line 187, in get_marshalled_dacon
    process_constructors_from_module(vm)
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/marshall_ctor.py", line 614, in process_constructors_from_module
    data_ns, tycon_ns = fetch_lib_module(module_name)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/caches.py", line 28, in fetch_lib_module
    result = module_cache[name] = hslowlevel.import_lib(name)[name]
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyphen.HsException: evaluate ((((Prelude.id) :: (()) -> (()))) (GHC.IO.Handle.Types.haDevice))<interactive>:1:36: error:
    * Cannot use record selector `haDevice' as a function due to escaped type variables
      Probable fix: use pattern-matching syntax instead
    * In the first argument of `(Prelude.id) :: (()) -> (())', namely
        `(GHC.IO.Handle.Types.haDevice)'
      In the expression:
        (((Prelude.id) :: (()) -> (()))) (GHC.IO.Handle.Types.haDevice)
      In an equation for `_compileParsedExpr':
          _compileParsedExpr
            = ((((Prelude.id) :: (()) -> (()))) (GHC.IO.Handle.Types.haDevice))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/importing.py", line 232, in load_module
    raise ImportError(
ImportError: Failed to import haskell module 'hs.RIO'

>>> hyphen.hslowlevel.import_lib("RIO")
{'RIO': ({'displayException': <hsobjraw.HsObjRaw object at 0x7fffe9eb40f0>, 'bifoldl': <hsobjraw.HsObjRaw object at 0x7fffe9fc9bb0>, 'modifyMVarMasked_': <hsobjraw.HsObjRaw object at 0x7fffdef70cb0>, 'takeMVar': <hsobjraw.HsObjRaw object at 0x7fffdef70cf0>, 'toList': <hsobjraw.HsObjRaw object at 0x7fffdef70cd0>, 'newURef': <hsobjraw.HsObjRaw object at 0x7fffdef70c50>, 'either': <hsobjraw.HsObjRaw object at 0x7fffdef70d10>, 'hSetFileSize': <hsobjraw.HsObjRaw object at 0x7fffdef70d30
....

Or the Massiv array library

>>> import hs.Data.Massiv.Array
Traceback (most recent call last):
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/importing.py", line 219, in load_module
    data_ns, type_ns = fetch_lib_module(haskell_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/caches.py", line 28, in fetch_lib_module
    result = module_cache[name] = hslowlevel.import_lib(name)[name]
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
hyphen.HsException: <no location info>: error:
    Could not find module `Data.Massiv'
    Perhaps you meant
      Data.Basis (from vector-space-0.16)
      Data.Matrix (from matrices-0.5.0)
      Data.Matrix (from matrix-0.3.6.3)
    Use -v (or `:set -v` in ghci) to see a list of the files searched for.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/n9gx1dk31wdfqsgny7dnnkhmlj40vkn0-python3-3.11.7-env/lib/python3.11/site-packages/hyphen/importing.py", line 224, in load_module
    raise ImportError(
ImportError: Failed to import haskell module 'hs.Data.Massiv'

>>> hyphen.hslowlevel.import_lib("Data.Massiv.Array")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
hyphen.HsException: evaluate ((((Prelude.id) :: ((->) (Data.Massiv.Core.Index.Ix2) ([] ((,) (GHC.Types.Int) (GHC.Types.Int)))) -> ((->) (Data.Massiv.Core.Index.Ix2) ([] ((,) (GHC.Types.Int) (GHC.Types.Int)))))) ((\ x -> [(a0, a1) | Data.Massiv.Array.:. a0 a1 <- [x]])))<interactive>:1:204: error:
    parse error on input `Data.Massiv.Array.:.'

So both can be found but not loaded and I assume it has something to do with how the Haskell code is written? For example, Massiv uses a lot of GHC Language Extensions and RIO at least some, while Data.Text or Data.Vector are plain Haskell2010.

Am I doing something wrong, can I do something to get these modules working or are modules with fancy Post Haskell2010 features simply out of scope for hyphen? :)

tbarnetlamb commented 9 months ago

Hi!

First of all, thank you for your kind words about hyphen.

The short answer (I'm sorry to say) is that the intent of hyphen was originally that it handle Haskell98 modules (so not even Haskell2010 was in scope when I started it, let alone post-2010). So there's no particular expectation that modules using more advanced features will work. (That said, I think quite a few newer Haskell features do happen to work serendipitously.) In my own stuff, I've generally found that it's possible to work around this restriction pretty easily by building a small "simple-Haskell" wrapper layer (in Haskell) that re-exports the functionality that I need with an interface that doesn't use fancy compiler extensions. This seems to work well. I think the particular problem with RIO is the use of existential types, which is certainly a "big" thing that would be hard to support in hyphen.

That's the bad news. The good news:

I'm open to adding support for particular features that are blocking large number of modern modules, modules where one can make the case that they'd be particularly useful if available, etc. I'm not actually familiar with RIO. Do you think it falls into this category? It seems to mostly re-export various other libraries, some of which I've successfully used from hyphen in the past. Maybe you can just use those of the underlying libraries that work?
The issue with Data.Massiv seems like it could possibly be a minor one. There's an annoying mismatch between the way the module hierarchy works in Haskell vs in Python discussed here: https://github.com/tbarnetlamb/hyphen/blob/main/hyphen/importing.py#L33 and I think that could be the problem in the traceback above. Try doing hyphen.importing.EXPECTED_EMPTY.append("Data.Massiv") after import hyphen and before import hs.Data.Massiv to see if that helps with the issue. Of course, there may be other things blocking importing this module, especially if it uses lots of advanced Haskell features, but this could be the only problem.

sheepforce commented 9 months ago

Thank you so much for your fast and detailed response.

I'm open to adding support for particular features that are blocking large number of modern modules, modules where one can make the case that they'd be particularly useful if available, etc. I'm not actually familiar with RIO.

RIO is, as you said, mainly a re-export of simple libraries, but emphasizes the ReaderT IO pattern in its entire design and sanitises logging (there is a dedicated logging framework instead of putStr stuff). I think the ReaderT pattern is useful and popular but it doesn't necessarily require RIO.

Try doing hyphen.importing.EXPECTED_EMPTY.append("Data.Massiv") after import hyphen

Some progress made, but given your premise of Haskell2010 modules, it is not unexpected that we now encounter an other error

hyphen.HsException: evaluate ((((Prelude.id) :: ((->) (Data.Massiv.Core.Index.Ix2) ([] ((,) (GHC.Types.Int) (GHC.Types.Int)))) -> ((->) (Data.Massiv.Core.Index.Ix2) ([] ((,) (GHC.Types.Int) (GHC.Types.Int)))))) ((\ x -> [(a0, a1) | Data.Massiv.Array.:. a0 a1 <- [x]])))<interactive>:1:204: error:    parse error on input `Data.Massiv.Array.:.'

Massiv heavily uses Language extensions like TypeOperators, DataFamilies, MultiParamTypeClasses and many many more.

That being said my use case is probably very niche (I'm writing Haskell for computational chemistry on high performance computers) and I don't want to stretch your time with feature requests like adding new features too much. :) My own project uses the full set of GHC2021 extensions plus a few others. Even if I have to write a Haskell2010 wrapper around my modules, this is still a much better approach than manually exposing a C-API and writing python bindings to it.

If you wish we can close this issue, now that I am aware that I simply require a Haskell2010 compatibility layer :)

tbarnetlamb commented 9 months ago

As you wish! But if you do come to believe that there are extensions to hyphen that could be broadly useful, please open another issue and I will try to help...

I do find that the simple subset of Haskell that hyphen supports is slowly becoming more of a hindrance/annoyance. That said, for my own uses, I generally build Haskell components whose implementations might use powerful post-2010 features, but where the interface that needs to be shared to Python is very simple. So the issue has never become critical for me.

tbarnetlamb / hyphen

Importing non Haskell2010 modules #32