search

commercialhaskell / hi-file-parser

Other
4 stars 4 forks source link

Support coming GHC 9.8.1 (GHC 9.8.1-rc1 is available) #16

Closed mpilgrem closed 1 year ago

mpilgrem commented 1 year ago

@hsyl20, may I once again call on your assistance with the above? Based on my experiments with GHC 9.8.1-rc1, it looks to me that the *.hi format has changed again, after GHC 9.6.3.

hsyl20 commented 1 year ago

I'll have a look at this week.

hsyl20 commented 1 year ago

That was a minor change. Fixed in #17

mpilgrem commented 1 year ago

@hsyl20, I am trying to build pantry with the official release of GHC 9.8.1 and a version of Stack that depends on hi-file-parser-0.1.5.0, and I'm getting these messages:

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry\Internal\Stackage.hi Decoding failure:
         Invalid dictionary index: 4168842332

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry.hi Decoding failure:
         Invalid dictionary index: 2574345280

I am wondering: could there have been a further change to the *.hi file format that is somehow evading the CI of this library (which is reporting 'green' for GHC 9.8.1).

mpilgrem commented 1 year ago

@hsyl20, I built Stack against a version of hi-file-parser-0.1.5.0 with enableDebug = True, and got this sort of debub output (extracts only, gaps represented by ...). Hope that helps:

Magic: 1face64
Version: 9081
Ways: ""
Dict ptr: 6248
Dictionary size: 478
Dictionary: ["pantry-0.9.2-JkhjEPHMYEj8HkTp1e9qOb", ...
...
Module: "Prelude"
Usage type: 0
Unit type: 0
Dict index: 5
Dict index: 250
Module: "Pantry.Types"
Usage type: 1
Dict index: 2
Home module: "Pantry.Hackage"
Dict index: 0
Dict index: 331
Dict index: 331
Dict index: 332
Dict index: 333
Dict index: 331
Dict index: 4168842332

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry\Internal\Stackage.hi Decoding failure:
         Invalid dictionary index: 4168842332
...
Magic: 1face64
Version: 9081
Ways: ""
Dict ptr: 88102
Dictionary size: 1412
Dictionary: ["pantry-0.9.2-JkhjEPHMYEj8HkTp1e9qOb", ...
...
Home module: "Pantry.HTTP"
Dict index: 0
Dict index: 396
Usage type: 1
Dict index: 5
Home module: "Pantry.Hackage"
Dict index: 0
Dict index: 397
Dict index: 398
Dict index: 399
Dict index: 400
Dict index: 401
Dict index: 402
Dict index: 403
Dict index: 404
Dict index: 405
Dict index: 406
Dict index: 407
Dict index: 408
Dict index: 409
Dict index: 2574345280

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry.hi Decoding failure:
         Invalid dictionary index: 2574345280
mpilgrem commented 1 year ago

I suppose it could be possible that GHC 9.8.1 can produce corrupt *.hi files. The corresponding parts of the debug output when building pantry with GHC 9.6.3 are:

...
Usage type: 1
Dict index: 2
Home module: "Pantry.Hackage"
Dict index: 0
Dict index: 334
Dict index: 334
Dict index: 335
Dict index: 336
Dict index: 337
Dict index: 338
...
Usage type: 1
Dict index: 5
Home module: "Pantry.Hackage"
Dict index: 0
Dict index: 400
Dict index: 401
Dict index: 402
Dict index: 403
Dict index: 404
Dict index: 405
Dict index: 406
Dict index: 406
Dict index: 407
Dict index: 408
Dict index: 409
Dict index: 410
Dict index: 411
Dict index: 412
Dict index: 413
Dict index: 414
Dict index: 415
Dict index: 416
mpilgrem commented 1 year ago

I think I can rule out the (unlikely) possibility that it is GHC 9.8.1 that is producing malformed *.hi files. I used this test (built against package ghc-9.8.1):

module Main (main) where

import Control.Monad ( forM )
import GHC.Iface.Binary ( CheckHiWay (..), TraceBinIFace (..), readBinIface )
import GHC.Platform
         ( Platform (..), ArchOS (..), Arch (..), OS (..), genericPlatform )
import GHC.Platform.Profile ( Profile (..) )
import GHC.Types.Name.Cache ( NameCache (..), initNameCache )
import GHC.Unit.Module.ModIface ( ModIface, ModIface_ (..) )
import GHC.Unit.Types ( GenModule (..) )
import GHC.Utils.Outputable ( SDoc, printSDocLn, traceSDocContext )
import GHC.Utils.Ppr ( Mode (..) )
import System.IO ( stdout )

main :: IO ()
main = do
  let fps =
        [ "D:\\Users\\mike\\Code\\GitHub\\commercialhaskell\\pantry\\.stack-work\\dist\\2b10cf2c\\build\\Pantry\\Internal\\Stackage.hi"
        , "D:\\Users\\mike\\Code\\GitHub\\commercialhaskell\\pantry\\.stack-work\\dist\\2b10cf2c\\build\\Pantry.hi"
        ]
  nc <- initNameCache 'A' []
  modIfaces <- forM fps (readBinIface' nc)
  let ns = map (moduleName . mi_module) modIfaces
  print ns

readBinIface' :: NameCache -> FilePath -> IO ModIface
readBinIface' nc fp =
  readBinIface profile nc CheckHiWay (TraceBinIFace traceBinIFace) fp

profile :: Profile
profile = Profile platform mempty

platform :: Platform
platform = genericPlatform
  { platformArchOS = archOS }

archOS :: ArchOS
archOS = ArchOS ArchX86_64 OSMinGW32

traceBinIFace :: SDoc -> IO ()
traceBinIFace = printSDocLn traceSDocContext (PageMode True) stdout

The output of the exectuable is (that is, it does not throw an error):

❯ stack exec -- hiTest-exe

Warning: Stack has not been tested with GHC versions 9.8 and above, and using 9.8.1, this may fail.
Magic: Wanted 33214052,
       got    33214052
Version: Wanted 9081,
         got    9081
Way: Wanted ,
     got
Magic: Wanted 33214052,
       got    33214052
Version: Wanted 9081,
         got    9081
Way: Wanted ,
     got
[ModuleName "Pantry.Internal.Stackage",ModuleName "Pantry"]
mpilgrem commented 1 year ago

Modifying my test to provide more information, the first instance gives me this:

UsageHomeModule ModuleName "Pantry.Hackage"
[ (HackageTarballResult,e1b26ec099457bfa71b89d24afac1cae)
, (HackageTarballResult,3130b9a905b4346529568b9a8f954565)
, (forceUpdateHackageIndex,50666e761ee4bf6215ce10c4cec342ce)
, (getHackageTarball,b53840208942067b1c88be8c9c67906f)
, (htrFreshPackageInfo,c4b8e8e27696394290fddd7c3dbe2e76)
, (htrPackage,fb6b3fab39801649ab2cb84cb8b22078)
]

and the elements of the directory are:

331: , "HackageTarballResult"
332: , "forceUpdateHackageIndex"
333: , "getHackageTarball"
334: , "htrFreshPackageInfo"
335: , "htrPackage"

However, it seems 334 is being read as 331 and 335 is being read as 4168842332???

mpilgrem commented 1 year ago

To rule out that I had introduced a bug since #17 was merged, I built Stack against hi-file-parser commit 4797274ec699dc60ffde106e7fba1d7cea957f51, and experienced the same problem.

mpilgrem commented 1 year ago

The first and second problem with added debug information from hi-file-parser:

Usage type: 1
Dict index: 2
Home module: "Pantry.Hackage"
Dict index: 0
Fingerprint:  "8048068b3e99c1d8832be56c1865429f"
Ignoring:  ()
Fingerprint:  "73d1081b486cee23657eb1c107af3f4d"
Ignoring:  ()
List (with LEB128): 6
Dict index: 331
Fingerprint:  "e1b26ec099457bfa71b89d24afac1cae"
Ignoring:  ()
Dict index: 331
Fingerprint:  "3130b9a905b4346529568b9a8f954565"
Ignoring:  ()
Dict index: 332
Fingerprint:  "50666e761ee4bf6215ce10c4cec342ce"
Ignoring:  ()
Dict index: 333
Fingerprint:  "b53840208942067b1c88be8c9c67906f"
Ignoring:  ()
Dict index: 331
Fingerprint:  "14ec4b8e8e276963942"
Ignoring:  ()
Dict index: 4168842332

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry\Internal\Stackage.hi Decoding failure:
         Invalid dictionary index: 4168842332
Usage type: 1
Dict index: 5
Home module: "Pantry.Hackage"
Dict index: 0
Fingerprint:  "8048068b3e99c1d8832be56c1865429f"
Ignoring:  ()
Fingerprint:  "73d1081b486cee23657eb1c107af3f4d"
Ignoring:  ()
List (with LEB128): 18
Dict index: 397
Fingerprint:  "fa9c7e16290dad7418a8252b836a5e75"
Ignoring:  ()
Dict index: 398
Fingerprint:  "e6879306c0e103c727c3120b40a50720"
Ignoring:  ()
Dict index: 399
Fingerprint:  "7c8e1a69a972a661437c87821b9415bd"
Ignoring:  ()
Dict index: 400
Fingerprint:  "b0de1a49a54c7e324c788f164f5dc731"
Ignoring:  ()
Dict index: 401
Fingerprint:  "f937d72150e494f225c0e3c8f2e945c2"
Ignoring:  ()
Dict index: 402
Fingerprint:  "8f82da48126adb635270b2f99deca1e3"
Ignoring:  ()
Dict index: 403
Fingerprint:  "c611f3918cfdefa95ce07f5776b7de3d"
Ignoring:  ()
Dict index: 403
Fingerprint:  "6f3b376f3d495477589fe4ee05c816e8"
Ignoring:  ()
Dict index: 404
Fingerprint:  "7c67e6444f5f388efe9d5fa6aa80b60d"
Ignoring:  ()
Dict index: 405
Fingerprint:  "167f6ae9113d1783150c807aee25cd8e"
Ignoring:  ()
Dict index: 406
Fingerprint:  "42818d6ba53c3e7f3a366d97e496343c"
Ignoring:  ()
Dict index: 407
Fingerprint:  "c4b08bfa666623eeda4df0a69c5f124e"
Ignoring:  ()
Dict index: 408
Fingerprint:  "b53840208942067b1c88be8c9c67906f"
Ignoring:  ()
Dict index: 409
Fingerprint:  "af5100401a891d5e209f87f4b46bbbf3"
Ignoring:  ()
Dict index: 410
Fingerprint:  "be37359c6d8be7093ffcf6567db3a46b"
Ignoring:  ()
Dict index: 411
Fingerprint:  "e40dacb6b6364edff90a529aa3a929cf"
Ignoring:  ()
Dict index: 412
Fingerprint:  "19dfb6b3fab39801649"
Ignoring:  ()
Dict index: 2574345280

Warning: Failed to decode module interface: D:\Users\mike\Code\GitHub\commercialhaskell\pantry\.stack-work\dist\2b10cf2c\build\Pantry.hi Decoding failure:
         Invalid dictionary index: 2574345280

It seems that something is going wrong with the penultimate Fingerprint in the lists.

mpilgrem commented 1 year ago

@hsyl20, I think I have it! The instance of Binary for NameSpace has changed; it is no longer guaranteed to be a single byte:

GHC 9.6.3:

instance Binary NameSpace where
    put_ bh VarName =
            putByte bh 0
    put_ bh DataName =
            putByte bh 1
    put_ bh TvName =
            putByte bh 2
    put_ bh TcClsName =
            putByte bh 3
    get bh = do
            h <- getByte bh
            case h of
              0 -> return VarName
              1 -> return DataName
              2 -> return TvName
              _ -> return TcClsName

GHC 9.8.1:

instance Binary NameSpace where
    put_ bh VarName =
            putByte bh 0
    put_ bh DataName =
            putByte bh 1
    put_ bh TvName =
            putByte bh 2
    put_ bh TcClsName =
            putByte bh 3
    put_ bh (FldName parent) = do
            putByte bh 4
            put_ bh parent
    get bh = do
            h <- getByte bh
            case h of
              0 -> return VarName
              1 -> return DataName
              2 -> return TvName
              3 -> return TcClsName
              _ -> do
                parent <- get bh
                return $ FldName { fldParent = parent }

I think that is throwing off this line in hi-file-parser, which assumes getWord8:

void (getList (getTuple (getWord8 *> getFastString) getFP)) -- usg_entities
mpilgrem commented 1 year ago

I am going to attempt a fix ...

hsyl20 commented 1 year ago

Good work @mpilgrem! Sorry for the delay, I was on vacation.