(Optionally) remove transitive dependencies on Template Haskell

wolfgangwalther commented 4 months ago

I am trying to cross-compile PostgREST to different architectures / OS. The biggest challenge I am facing is Template Haskell, which is notoriously hard to deal with when cross-compiling. Only build-time dependencies are relevant, because running tests when cross-compiling.. is equally hard anyway.

AFAICT, hasql currently depends on Template Haskell twice:

Once through contravariants-extra in hasql-transactions: https://github.com/nikita-volkov/hasql-transaction/blob/2a9a710bd6230cb0abae84f14f852ab14fc3fa85/library/Hasql/Transaction/Private/Statements.hs#L41-L43
And then by using network-ip in various packages (hasql, hasql-implicits, postgresql-binary, ... more?), which depends on data-dword, which is one big piece of Template Haskell. This seems to be related to encoders and decoders for postgres' inet type only.

The usage of contrazip2 could trivially be replaced with (fst >$< ...) <> (snd >$< ...).

Removing network-ip is unlikely to happen, I guess. But maybe we could add cabal flags to each package using it, to be able to disable this dependency and the related encoders/decoders? That should have no impact on other code, right? We are not using inet in PostgREST anyway.

I have not tried the above, yet, but would you consider accepting PRs to do the above and thus make hasql more friendly for cross-compiling?

nikita-volkov commented 4 months ago

What are the issues that you're facing with template-haskell?

Regarding removing the dependency on contravariant -extras, sure, I'll accept the PR.

Regarding network-ip we can consider moving to a different package to represent the types. I've actually been thinking about it for other reasons anyway.

wolfgangwalther commented 4 months ago

What are the issues that you're facing with template-haskell?

Template Haskell needs to be run (via the internal or external interpreter) at build time. This means that code for the target platform needs to run at build time. For some special cases this can work, i.e. when compiling on the same OS to a different architecture you can use qemu-user to run the external interpreter, or when compiling to windows you can use wine to do it. But it does not work in the general case - for example, I can't cross-compile any TH from Linux to FreeBSD, because I just can't run an external interpreter at compile time like that.

In theory, I could run a full VM to run the external interpreter on - but then the external interpreter runs in a totally different environment and the TH code might rely on that. For example, we currently use TH to fetch the current commit hash from the local git checkout. Such things won't work in that scenario.

Most Template Haskell is just used for code generation, like in the examples here. But unfortunately, this still prevents a generic solution. The only generic solution is... to avoid it! :)

Regarding removing the dependency on contravariant -extras, sure, I'll accept the PR.

I'll prepare one.

Regarding network-ip we can consider moving to a different package to represent the types. I've actually been thinking about it for other reasons anyway.

How would you split that? One package per type in a monorepo setup? Or something else?

nikita-volkov commented 4 months ago

I must admit I've learned something today thanks to your thorough answer :)

The hasql-transaction update should get autoreleased once the pipeline succeeds.

Regarding network-ip we can consider moving to a different package to represent the types. I've actually been thinking about it for other reasons anyway.

How would you split that? One package per type in a monorepo setup? Or something else?

I've meant that I've been considering moving to a different package to represent those types. "network-ip" hasn't been updated since 2019. "ip" seems like a viable alternative. What do you think?

wolfgangwalther commented 4 months ago

I've meant that I've been considering moving to a different package to represent those types. "network-ip" hasn't been updated since 2019. "ip" seems like a viable alternative. What do you think?

Ah, much simpler ;)

So ip seems to depend on bytebuild, which depends on haskell-src-meta, which depends on th-orphans. bytebuild itself seems to use Template Haskell, too, but my build failed before that.

So that wouldn't help with avoiding template haskell, yet.

I went through hackage, searching for ip ordering by "latest upload" and found hw-ip. This builds fine without Template Haskell. But it's certainly not as recent as ip, at least in terms of the last package update. The repo has some activity as of January this year.

Also note https://github.com/byteverse/haskell-ip/issues/68.

I also looked into the data types for ip and hw-ip. I'm not sure whether ip actually supports netmasks, too. hw-ip seems to do that. The upstream types do have netmask information, imho, so maybe that rules out ip already.

TLDR: maybe hw-ip is a better fit?

wolfgangwalther commented 4 months ago

The hasql-transaction update should get autoreleased once the pipeline succeeds.

Seems like the pipeline is broken because of some... other dependency on template-haskell and not high enough bounds there? Funny, kind of :D

nikita-volkov commented 4 months ago

The hasql-transaction update should get autoreleased once the pipeline succeeds.

Seems like the pipeline is broken because of some... other dependency on template-haskell and not high enough bounds there? Funny, kind of :D

Yeah. We have the centric packages depending on template-haskell:

Mostly to provide the instances of Lift.

It's likely that we've started something here based on some false assumptions.

wolfgangwalther commented 4 months ago

Yeah. We have the centric packages depending on template-haskell:
* https://hackage.haskell.org/package/aeson

* https://hackage.haskell.org/package/bytestring

* https://hackage.haskell.org/package/uuid-types

* https://hackage.haskell.org/package/text
Mostly to provide the instances of Lift.

It's likely that we've started something here based on some false assumptions.

No, not really. Those build fine without template haskell - they probably need it for their tests, not to build the code.

nikita-volkov commented 4 months ago

Check out the Lift instances on Value, UUID and ByteString.

wolfgangwalther commented 4 months ago

Check out the Lift instances on Value, UUID and ByteString.

So two of the three files have {-# LANGUAGE TemplateHaskellQuotes #-}, the last one does not have any. Notable, they don't have LANGUAGE TemplateHaskell. I haven't fully understood the difference here, but it seems that everything that can be done with TemplateHaskellQuotes works fine for cross-compiling.

aeson cross-compiles fine, I just confirmed it again. uuid-types, too. bytestring and text are built-in and work 100%.

Edit: The template-haskell library is built-in and is available when cross-compiling, too. That's why instance TH.Lift ByteString is not a problem. It only becomes a problem when the TH splices are actually run - so further down the stream.

wolfgangwalther commented 4 months ago

So to be more precise: Template Haskell's splices are the evil, not quotes.

nikita-volkov commented 4 months ago

So do we still need to fix anything here?

wolfgangwalther commented 4 months ago

Yeah, the situation is unchanged - my opening post is correct. All my observations (does X cross-compile?) where based on actually trying to build. A few months ago I tried replacing all those template haskell splices in data-dword (network-ip) and contravariant-extras with the their resultant haskell code. Pretty sure, that this made hasql and related libs cross-compile fine. But obviously that was not maintainable.

So I know this will work once we get rid of those dependencies.

nikita-volkov commented 4 months ago

Okay. The "hasql-transaction" update is now released. Gotta make a decision on the inet data-type. Does the issue with the "ip" package still stand?

wolfgangwalther commented 4 months ago

Yes, ip does not cross-compile, hw-ip does.

wolfgangwalther commented 4 months ago

iproute might also be an option: It has an IPRange type which should support ips with netmask. Seems to be even lighter on dependencies and cross-compiles fine.

nikita-volkov commented 4 months ago

Actually I've just come in to suggest that as well :)

wolfgangwalther commented 4 months ago

Should I give it a go for postgresql-binary or are you going to?

nikita-volkov commented 4 months ago

Sure! Please do

nikita-volkov commented 4 months ago

The ecosystem is updated. Thanks for your contribution!

wolfgangwalther commented 4 months ago

Thanks a lot!

nikita-volkov / hasql

(Optionally) remove transitive dependencies on Template Haskell #163