search

haskell-infra / hackage-trustees

Issue tracker for Hackage maintainance and trustee operations
https://hackage.haskell.org/packages/trustees/
42 stars 7 forks source link

Missing GHC 8.8.1 boot libraries #240

Closed tfausak closed 4 years ago

tfausak commented 5 years ago

GHC 8.8.1 was released more than two weeks ago. Many of the boot libraries are not on Hackage yet. Someone made a Reddit thread about it, which is how I noticed.

Based on the output of stack --resolver ghc-8.8.1 exec ghc-pkg list, the following packages are not on Hackage:

Whereas these packages are currently available: Cabal-3.0.0.0, array-0.5.4.0, binary-0.8.7.0, containers-0.6.2.1, deepseq-1.4.4.0, directory-1.3.3.2, filepath-1.4.2.1, ghc-boot-8.8.1, ghc-boot-th-8.8.1, ghc-compact-0.1.0.0, ghc-prim-0.5.3, haskeline-0.7.5.0, hpc-0.6.0.3, integer-gmp-1.0.2.0, mtl-2.2.2, parsec-3.1.14.0, pretty-1.1.3.6, process-1.6.5.1, stm-2.5.0.0, template-haskell-2.15.0.0, terminfo-0.4.1.4, text-1.2.4.0, time-1.9.3, transformers-0.5.6.2, unix-2.7.2.2, xhtml-3000.2.2.1.

Can these packages be uploaded? And since this seems to happen a lot (#203, #192, #179, #160, #153, #141), is there anything I can do to help automate this in the future? Thanks!

RyanGlScott commented 5 years ago

FWIW, I don't think the rts library was ever intended to be uploaded to Hackage, as it's only used internally by GHC and doesn't expose any modules (i.e., no Haddocks).

I seem to recall that there is a bug that makes it difficult to upload ghc-heap to Hackage, although I can't recall what it is off the top of my head. Perhaps someone in the know can fill in the details.

RyanGlScott commented 5 years ago

As far as bytestring-0.10.9.0 goes, see GHC #17110.

tfausak commented 5 years ago

Oh, good catch! I produced that list of packages pretty mechanically. Taking a closer look at them ...

hvr commented 5 years ago

is there anything I can do to help automate this in the future?

No.

domenkozar commented 5 years ago

Please? :)

sjakobi commented 4 years ago

Ben's mail on Haskell Cafe shines some light on the situation:

https://mail.haskell.org/pipermail/haskell-cafe/2019-October/131624.html

domenkozar commented 4 years ago

So in summary, there's a huge backlog of work, but while people are stepping up to help, there's no interest in delegation. Looks like a superhero syndrome to me.

bgamari commented 4 years ago

This issue has been mentioned on haskell-cafe where I describe the current state-of-play.

tfausak commented 4 years ago

Based on your explanation, this is a manpower problem. Yet the help of many people (including @bgamari, @RyanGlScott, and me) has been rejected. What can we do? Looking at the hackage-server repository, I can't find any work in progress (issue, project, branch, or pull request) to help with.

gbaz commented 4 years ago

The problem to be clear is not about helping to "upload the libraries" -- that's the easy part! The problem is updating hackage-server to the new Cabal library, which is a fair amount of work, and tricky. As far as I know, ben has started to do some stuff in this direction. As in all such cases what we need is not "drive by" help, but at the end of the day more core maintainers of hackage-server.

bgamari commented 4 years ago

In the interest of making the hackage-server upgrade task more accessible to contributors I have opened https://github.com/haskell/hackage-server/issues/852.

codygman commented 4 years ago

context: I landed here after following ghcide issues for ghc 8.8.1 and why ghcide-nix didn't support it yet and I found this very interesting comment section and have no idea what's going on.

The problem is updating hackage-server to the new Cabal library

Did anyone ever figure out why upgrading to Cabal v3 was necessary?

There was never a response to this query on Haskell cafe or anywhere else I could find.

Also, I see that more core maintainers were and are needed. Then I go on to read the quote below and see:

For what it is worth, I have asked several times whether I can help with the upload process but have been turned down repeatedly. I can't claim that I fully understand the reasoning behind this, but I can nevertheless respect the decision. - @bgamari

This makes me feel that I too could be turned down with no explanation if I offered my help and I'm sure it leaves others with this impression. I'm left feeling confused and kind of put off. Some clarity here would help extinguish that for me and others I think.

Could this lack of clarity be part of the problem for getting more core maintainers?

codygman commented 4 years ago

Also... I was tempted to avoid addressing but how in the world is this a constructive answer or acceptable behavior?

image

Is what seems to be an earnest request to help shutdown with "No" seen as acceptable by the Hackage trustees?

It seems I'm out of the loop and there is something personal going on here or another source of frustration I don't know about. I'd also like to help if possible, but I fear simply by commenting on these issues I could doomed to face answers like above in the future if I try to contribute.

gbaz commented 4 years ago

It was a yes/no question. So a yes/no answer is acceptable. There's plenty of ways you can help contribute to the ecosystem, including to hackage-server, and also to cabal-install.

Let me give a little more, but not all the context: For complicated reasons, uploading "base" is not like uploading a normal library, not least related to the fact that it isn't reinstallable. Some of the plumbing changes going on have a long term goal of making it more installable. But to upload base-as-it-stands means that we need to use the cabal library that understands those intermediate changes. Context-dumping all the details is difficult. There are a lot of moving parts. A lot of updating hackage-server to the new cabal lib isn't just "chasing types" but working out how to interact with all the new semantic changes.

I know you're not happy with the answer given above, hopefully this helps clear things up.

And a final plea: please don't be rude to the people doing the work if sometimes they give short answers -- they, like all of us, are busy, and only have so many cycles to spare.

domenkozar commented 4 years ago

I'm still puzzled about where the dependency comes from and how we can help to remove it.

But to upload base-as-it-stands means that we need to use the cabal library that understands those intermediate changes.

Is it about the upload process itself or the post-processing that happens once base lands in hackage server?

It would make sense to me to decouple this dependency and split the work so others can help, that way we can prevent future blockers like this.

Context-dumping all the details is difficult.

That's understandable, but maybe post-mortem we can take a look at changes and figure out if there are ways to avoid that in the future.

phadej commented 4 years ago

To upload package which uses cabal-version: 3.0 https://github.com/ghc/ghc/blob/02f97bc24d11ef0a1c7f8bd06dc752263a89bdd3/libraries/base/base.cabal#L1 (base.cabal in ghc-8.8 branch) you have to teach hackage-server about cabal-version: 3.0, which requires porting hackage-server to use Cabal-3.0. The library version which knows about cabal-version: 3.0.

domenkozar commented 4 years ago

To upload package which uses cabal-version: 3.0

Why does base need to require 3.0?

you have to teach hackage-server about cabal-version: 3.0

Would it make sense to extract cabal-version: 3.0 parsing into a separate package, so that the amount of work is reduced to only parsing when cabal is bumped?

phadej commented 4 years ago

And the reason for cabal-version: 3.0 is the cmm-sources used in above mentioned base.cabal. Something which is actually still not satisfactory fixed in Cabal the library, https://github.com/haskell/cabal/issues/6377. The feature was pushed through, but never finished. There are still no tests, and few corner cases which won't work. If you have time to help, the 6377 issue is nicely boring one. Invisible work, which have to be done.

It's the example of "drive-by contribution", which IMHO should been rejected, until it meets the standards.

That's the reason why ghc-heap isn't on Hackage, as it uses cmm-sources too: https://github.com/ghc/ghc/blob/7179b968c709eb0f5413c8e3847a8a593c596185/libraries/ghc-heap/ghc-heap.cabal.in#L30. As far as I know that library would be normal, reinstallable library once uploaded to Hackage, but Cabal won't be able to build it.

And if you check who works or watches over these leftovers: those are always the same people. I'm glad that Herbert is pedantic and does care about the details.

If the only thing you need are haddocks, they are at https://downloads.haskell.org/ghc/latest/docs/html/libraries/index.html Not as convenient as Hackage, but they are available.

phadej commented 4 years ago

Would it make sense to extract cabal-version: 3.0 parsing into a separate package, so that the amount of work is reduced to only parsing when cabal is bumped?

Thank you for your recommendation. But it won't work. Over the half of Cabal library is the one which deals with .cabal file. And these changes are still would be need to to dealt with on hackage-server side. My understanding is that, hackage-server actually doesn't care about other half of Cabal library, but here my knowledge ends.

domenkozar commented 4 years ago

And the reason for cabal-version: 3.0 is the cmm-sources used in above mentioned base.cabal.

Those seems to be there for 2 years, since GHC 8.4.1: https://github.com/ghc/ghc/commit/ba2ae2c8729d5aef2aeb7fb32d6c0ea2a465ea25

Sorry to be a pain, but I still don't see why bumping cabal-version couldn't be avoided.

As far as I know that library would be normal, reinstallable library once uploaded to Hackage, but Cabal won't be able to build it.

OK, but that's more about the cabal library the user has, I presume hackage-server doesn't install anything? Isn't the package served as-is?

And if you check who works or watches over these leftovers: those are always the same people. I'm glad that Herbert is pedantic and does care about the details.

I very much appreciate all the work that's put into. I'm not here to judge, just trying to dismantle what's going on. I think a few people also expressed the same, so digging into what's where seems beneficial for the next time cabal version is bumped.

I quite disagree with @gbaz & @hvr that there's no room for help, we've identified a few cases where others could have stepped in if communication was clearer.

If the only thing you need are haddocks, they are at https://downloads.haskell.org/ghc/latest/docs/html/libraries/index.html Not as convenient as Hackage, but they are available.

I need these libraries to support ghcide for 8.8 via https://github.com/input-output-hk/haskell.nix

And these changes are still would be need to to dealt with on hackage-server side. My understanding is that, hackage-server actually doesn't care about other half of Cabal library, but here my knowledge ends.

I wasn't specific enough, I meant splitting up hackage-server so that cabal parsing happens independently, separating concerns of parsing user supplied cabals and the rest of hackage-server that wouldn't need the cabal bump. Reducing the scope and code that needs to be looked at.

phadej commented 4 years ago

@domenkozar because the cmm-sources where added 2 years ago, in hurry, and without properly implementing them in Cabal. Follow the links in

While those buildinfo fields were added to the parser some time ago via 57d7f28 and 4a28765 that work was never completed by implementing the necessary build/sdist logic in Cabal.

OK, but that's more about the cabal library the user has, I presume hackage-server doesn't install anything? Isn't the package served as-is?

It (e.g. ghc-heap for example) couldn't be uploaded as cabal-version: 2.4 package, and therefore the ghc-heap-8.6.1 is still not uploaded (https://github.com/haskell-infra/hackage-trustees/issues/179), because it was pending on Cabal bug.

The only solution I really can think about is to not permit any "non perfect" patches to Cabal, but then people would complain about that too. So please, just give us time. People are working on this difficult issue.

I'm 100% sure, you don't need these libraries for ghcide for 8.8, as they are bundled with GHC-8.8. Something wrong in the nix setup if it requires the package to be in the Hackage index too.

I'm unsubscribing from this issue.

tfausak commented 4 years ago

Are people working on this? It’s been almost four months since GHC 8.8.1 was released. It’s been more than a month since any changes have been made (or updates given) to hackage-server. Last I heard @dwijnand was going to sync up with @hvr. I don’t know if that’s happened or not.

As I said when I opened this issue, I would be happy to help. However I’m not going to do so without an explicit go-ahead from someone in charge. Last time I tried to contribute my changes were made obsolete by other changes that were quietly happening at the same time.

phadej commented 4 years ago

@tfausak yes. @hvr spend whole last weekend deploying hackage-server to reduce the delta of upgrade from Cabal-2.4 to 3.0. You can see that from the commit activity in master and central-server branch.

domenkozar commented 4 years ago

Any updates?

tfausak commented 4 years ago

GHC 8.8.2 was released last week: https://www.haskell.org/ghc/download_ghc_8_8_2.html

Many of the included libraries are missing: https://downloads.haskell.org/~ghc/8.8.2/docs/html/users_guide/8.8.2-notes.html#included-libraries

I assume it's the same root cause as the missing GHC 8.8.1 packages.

tfausak commented 4 years ago

GHC 8.8.3 was released last week: https://www.haskell.org/ghc/download_ghc_8_8_3.html

Many of the included libraries are missing: https://downloads.haskell.org/ghc/8.8.3/docs/html/users_guide/8.8.3-notes.html#included-libraries

I assume it's the same root cause as the missing GHC 8.8.1 and 8.8.2 packages.

tfausak commented 4 years ago

Even though these packages are not yet available on Hackage, they are now available on Stackage.

You can read more about things from Stackage's perspective here: https://tech.fpcomplete.com/blog/base-on-stackage

tfausak commented 4 years ago

You know the drill: GHC 8.10 was released and the libraries aren't available on Hackage. I think they also won't be on Stackage until the nightly resolvers switch over from GHC 8.8.3.

https://downloads.haskell.org/~ghc/8.10.1/docs/html/users_guide/8.10.1-notes.html#included-libraries

Ailrun commented 4 years ago

Now hackage server has been updated and base-4.14.0.0 was also uploaded. Is there any news for GHC 8.8 packages?

sjakobi commented 4 years ago

Is there any news for GHC 8.8 packages?

I see that @hvr has uploaded base-4.13.0.0.

At least ghc-8.8.* and integer-gmp-1.0.3.0 still seem to be missing though.

BTW many thanks to everyone who has helped to get this issue fixed! :)

Ailrun commented 4 years ago

It looks like all the packages except ghc/ghci/ghc-boot/ghc-boot-th have been uploaded. Thank you all for the work!

sjakobi commented 4 years ago

ghc-boot-8.10.1 is the last (interesting) one missing, AFAICT.

rwlock404 commented 4 years ago

And a final plea: please don't be rude to the people doing the work if sometimes they give short answers -- they, like all of us, are busy, and only have so many cycles to spare.

I'd like to second that! These people are doing an important but apparently thankless job and don't deserve getting all this flak. If I ever meet one of you Trustees at a conference at least one beer for each of you is on me! Keep up the good work!

phadej commented 4 years ago

Only ghc-heap is remaining.

The test file I used is simple

module Main (main) where

import GHC.Exts.Heap.Closures

main :: IO ()
main = do
    let x = True
    eq <- areBoxesEqual (asBox x) (asBox x)

    if eq == True
    then putStrLn "OK"
    else fail "Doesn't work"

which exercises custom cmm code. I think it could be even added as test-suite to ghc-heap so release manager (i.e. @bgamari) could test it works when built from source (Hackage). I'd recommend moving source files under src/ (i.e. not use the default hs-source-dirs: .).

I'm not opening GHC ticket about this, as I'm not interested in driving this to completion.

phadej commented 4 years ago

FWIW I'd also argue that ghc-heap should have some stricter dependency declaration on GHC, maybe even

if !impl(ghc ==8.8.1)
  build-depends: unbuildable <0

Or better, we should start version rts package, as it's now always rts-1.0 which renders its version unusable for dependency declarations.

codygman commented 4 years ago

And a final plea: please don't be rude to the people doing the work if sometimes they give short answers -- they, like all of us, are busy, and only have so many cycles to spare.

I'd like to second that! These people are doing an important but apparently thankless job and don't deserve getting all this flak. If I ever meet one of you Trustees at a conference at least one beer for each of you is on me! Keep up the good work!

I second your thanks and wouldn't mind buying beers for all either 😁.

I cannot see where there was any rudeness in this thread to contributors though. Accountability shouldn't be considered rude.

The problem with "it's fine for people doing the work to give short answers" is that they"ll likely never get any help because others offering to help don't know where to start.

This is concerning because we don't know what happens when they are no longer interested in doing this very important work.