Korusuke commented 5 years ago

This can be tested now

Korusuke commented 5 years ago

@mojca can you please test this out a bit and see if it works ? upt-hackage-frontend

mojca commented 5 years ago

I would also suggest to also ask @essandess to test (if he's interested and if someone can quickly explain what needs to be done). He contributed https://github.com/macports/macports-ports/pull/4794 (and a few others), and testing hackage is definitely related to having a working Haskell available.

The ultimate test would be to get pandoc port working this way :) @essandess used a slightly different approach, but the goal would be very similar.

essandess commented 5 years ago

@Korusuke @mojca I’d be happy to test, but there are caveats:

Based on my own limited knowledge and experience, I don’t understand why it’s necessary or desirable for MacPorts to duplicate existing Haskell development tools like cabal (#4715) or stack (#4633).
It’s trivial to build pandoc from source (#4698) using stack, and both unnecessary and difficult/practically impossible to maintain using a large number of MacPorts hs-* dependencies.
I believe this is the reason that MacPorts ghc (#4794) and pandoc (#4698) are year/s out-of-date. There are too many moving parts and conflicting hackage versions. Just use the lts-* long-term support settings from stack, as specified by the Haskell packages themselves in their stack.yaml (or cabal.yaml) configurations, e.g. pandoc‘s stack.yaml.
I am not aware of a useful purpose of MacPorts maintaining its hs-* ports.

essandess commented 5 years ago

I am not aware of a useful purpose of MacPorts maintaining its hs-* ports.

That’s too strong. These are great for ghci session load :l commands, the same way that the py-* ports are in Python.

But for building Haskell packages, using the standard development tools is the way to go.

codecov[bot] commented 5 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@7eb75a1). Click here to learn what that means. The diff coverage is 92.85%.

@@          Coverage Diff           @@
##             master   #39   +/-   ##
======================================
  Coverage          ?   88%           
======================================
  Files             ?     1           
  Lines             ?   200           
  Branches          ?    10           
======================================
  Hits              ?   176           
  Misses            ?    24           
  Partials          ?     0

Impacted Files	Coverage Δ
upt_macports/upt_macports.py	`88% <92.85%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7eb75a1...a034656. Read the comment docs.

pmetzger commented 5 years ago

So the issue is this: for several languages (python, ocaml, haskell, etc.) there are language-specific package systems available, but they are all-encompassing/totalizing systems that don't fit well inside the MacPorts world. Meanwhile, we have individual applications some users want to run that have nothing to do with ocaml or haskell or what have you per se; they have no idea what the program is written in, but they want a macports-style build of the package.

So it makes some sense for MacPorts to maintain packages for a limited number of such language-specific packages in order to build such apps. I've been wanting to do such a thing myself for OCaml even though opam is the better way to build ocaml apps. In fact, I'd like a opam front end developed for upt-macports for just this reason.

reneeotten commented 5 years ago

I've been wanting to do such a thing myself for OCaml even though opam is the better way to build ocaml apps. In fact, I'd like a opam front end developed for upt-macports for just this reason.

@pmetzger can you please open a separate issue for this and provide a little more background information? I have personally no experience with ocaml/opam, so one would need to check whether all the metadata is easily available to use with upt. Can you point us to (i) the repository where typical packages are downloaded from, (ii) your favorite port/Portfile that you manually wrote and we can use as exptected/correct output?

Again, right now @Korusuke is working on some other things but we can certainly take a look at this at a slightly later point in time.

pmetzger commented 5 years ago

@reneeotten It might be easier if you were to get in touch with me by email; address is on my github home page. I believe that opam files contain essentially everything one would need in order to generate Portfiles automatically, and I'd be happy to work along with you guys making this happen. There's even an ocaml package for parsing opam files that could be twisted into extracting data without needing to rewrite the parser.

pmetzger commented 5 years ago

I have also opened an issue... #46

reneeotten commented 5 years ago

back to the hackage frontend, I tested a random selection of ports (not the installation, only the Portfile generation) that are currently present in MacPorts and noticed the following:

the upt-hackage frontend doesn't declare its dependencies (Issue opened on https://framagit.org/upt/upt-hackage)
for several packages it throws an IndexError in the parse function (e.g., vector-space, json, readline)
parsing of the descriptions doesn't always work probably due to HTML syntax or such... (e.g., base64-bytestring, blaze-html, data-accessor-template)
I think all these packages actually get compiled and if that's true (please double check), then we shouldn't add the stuff about "supported_archs" in the template/Portfile
there should be a newline between homepage and checksums and checksums and depends_lib

Korusuke commented 5 years ago

1. the `upt-hackage` frontend doesn't declare its dependencies (Issue opened on https://framagit.org/upt/upt-hackage)

Sorry, forgot to add it to the commit.

2. for several packages it throws an `IndexError` in the `parse` function (e.g., `vector-space`, `json`, `readline`)

Apparently few packages doesn't declare homepage...in that case now it sets homepage to the hackage url itself.....this is already done by haskell 1.0 portgroup too, so maybe we don't need to set the homepage.

3. parsing of the descriptions doesn't always work probably due to HTML syntax or such... (e.g., `base64-bytestring`, `blaze-html`, `data-accessor-template`)

Fixed!

4. I think all these packages actually get compiled and if that's true (please double check), then we shouldn't add the stuff about "supported_archs" in the template/Portfile

I checked a few existing haskell packages and they don't declare any of this so maybe it's not needed but I will check about this more.... @mojca can help here maybe.

5. there should be a newline between `homepage` and `checksums` and `checksums` and `depends_lib`

Fixed!

reneeotten commented 5 years ago

Apparently few packages doesn't declare homepage...in that case now it sets homepage to the hackage url itself.....this is already done by haskell 1.0 portgroup too, so maybe we don't need to set the homepage.

I think we should set the homepage to whatever upstream gives and otherwise set it to the hackage url itself. I see indeed that this is the default in the PortGroup; so if you can leave the homepage out of the template if it's not specified upstream.

Fixed!

Good, now there are however two newlines between long_description and homepage....

reneeotten commented 5 years ago

also as I said earlier, please remove the noarch part from the base template for hackage just as is done for rubygems

reneeotten commented 5 years ago

@Korusuke after you made the changes, please do verify that there are no errors reported anymore by port lint and check for a subset of haskell ports that they actually install correctly. Once that is done, we should be good to merge this PR and then it could be more easily tested by others as well.

@Korusuke okay, so you made the changes I suggested but didn't comment on how much testing you've actually done on installing the haskell ports. There are currently 145 hs- ports in MacPorts; can you please generate Portfiles and install a random selection of ~50-75 these ports and post somewhere the information on those that installed correctly (i.e., using port -dv installed | grep "hs-") and some logs for ports that failed (if applicable)? Additionally, please verify that port lint --nitpick <name> for these ports do not give any errors/warnings (perhaps with the exception of missing dependencies if you didn't to the recursive packaging).

Korusuke commented 5 years ago

Yeah few days back, I had tested packaging ghc itself....I did not hope it work but atleast was expecting some error but after a whole night(10+ hours) of compiling/installing and a zillion warning and debug log, I had no more patience left so I quit it and never tried it again till now.

This time as suggested by @reneeotten, I took normal hs-* ports which are already there and tried to package those and in the way found a few issues with upt-hackage and upt.

Following are the issues faced...I will be creating a issue if not already there for each oneof the following:

Firstly recursive ends mid way as described in #50 , I have the patch for this offline will push it today and create a PR at upt.
When packaging multiple package recursively at the same time it is possible that two packages have same dependency which results in sys.exit due to Portfile already exist error....so maybe we can replace that with just a error log message instead of exiting?
As @Steap is implementing for upt-cpan to convert pod-name into dist-name something similar is required in upt-hackage or maybe if viable in all frontends....basically for hackage there are some dependencies like "unbuildable", "integer", etc.. which are not present in hackage and thus when trying to install port, macports gives error of dependency not found.....this can be eliminated by only adding dependency which are present in the frontend....this would save a lot of time of the maintainer.
Last thing are hackage specifiers for some ports are bit complicated and have "&&" and "||" operators and that cause issues which I am yet not sure how to solve....more on this issue in a dedicated issue tomorrow.

After "kinda" solving or bypassing all the above issues finally a port hs-mmap is installing but I am not sure if it will install properly cause of old ghc version but yeah as of now from past 2 hours ghc is compiling/installing, so I am gonna sleep now and continue tomorrow morning.

essandess commented 5 years ago

@Korusuke There’s a working, bootstrapped ghc v. 8.6.5 compiler in https://github.com/macports/macports-ports/pull/4794

This uses straight-up make etc. command from the upstream district without any complicating MacPorts hs-* ports.

We’ve all been waiting on the merge for a while now. Not sure what’s causing the delay.

Korusuke commented 5 years ago

Thanks!!! @essandess , after many many hours finally it installed. The existing one in macports never installed even after a whole nights sleep, #4794 normally would have worked but after about 3 hours I was tired of waiting and so cancelled it installed with bootstrap variant which took more 3-4 hours...but yeah I am very happy that it finally installed.

I will be creating a issue if not already there for each oneof the following Done! please refer #52, #53, over at upt-hackage and the PR in upt for recusrion error.

Quickly trying to install hs-mmap seems to work after removing all it's dependency as they are the ones that #4794 ghc already installs so as mentioned in #53 we should not have them in packages again.

Trying a batch install as originally suggested by @reneeotten would be hard right now because of the manual dependency editing issue. So I would say first we will have to solve #52 #53 and then we can get back to testing this.

macports / upt-macports

[WIP]: Add Hackage frontend support #39

Codecov Report