search

haskell-nix / hnix

A Haskell re-implementation of the Nix expression language
https://hackage.haskell.org/package/hnix
BSD 3-Clause "New" or "Revised" License
741 stars 114 forks source link

Implement `getURL` of `instance MonadHttp IO` #1051

Closed soulomoon closed 1 year ago

soulomoon commented 2 years ago

Now the builtins.fetchurl would be done Also I unmask the test files It might close #258 as real implementation related #1034

soulomoon commented 2 years ago

It seems a lot of name conflict between Effects.hs and hnix store

soulomoon commented 2 years ago

I am now able to compute the correct hash and output path using makeFixedOutputPath. But I do not understand why using addTextToStore' returns a output path with different hash.

Evaluating:

--eval -E 'builtins.fetchurl 
  https://raw.githubusercontent.com/haskell-nix/hnix/master/default.nix'

Some digging, I try to trace addTextToStore' to hnix store, but it seems it is just proxy to remote store

soulomoon commented 2 years ago

Considering if it is the padding diff the hash

https://github.com/haskell-nix/hnix-store/blob/792c76b0af71c8bd76534ac879cf9355041a94f4/hnix-store-remote/src/System/Nix/Store/Remote/Binary.hs#L34

the remote store line is here https://github.com/NixOS/nix/blob/e34fe47d0ce19fc7657970fb0e610bffbc3e43f0/src/libstore/daemon.cc#L351

soulomoon commented 2 years ago

It's my mistake, the reason is that the addTextToStore and makefixouput use different hash object for hashing a name:

where 37268335dd6931045bdcdf92623ff819a64244b53d0e746d438797349d4da578 is hash of the content. Here is the problem, how do I add the text along with fixed output through hnix store

Anton-Latukha commented 2 years ago

You can open hnix-store reports during work.

It is much more malleable, we can shape API there to fit with hnix.

Also would thank if details as in https://github.com/haskell-nix/hnix/pull/1051#issuecomment-1031380804 would end-up noted in the source code.

soulomoon commented 2 years ago

You can open hnix-store reports during work.

It is much more malleable, we can shape API there to fit with hnix.

Also would thank if details as in #1051 (comment) would end-up noted in the source code.

Sure, I am still getting familiar with the interaction between hnix-store, nix-daemon and hnix.

soulomoon commented 2 years ago

How nix implement fetchurl break down into(source) :

soulomoon commented 2 years ago

hnix-store api for addToStore does seems a bit off comparing to nix, we should shape API as in nix.

soulomoon commented 2 years ago 
baseNameOf :: Text -> Text
baseNameOf a = Text.takeWhileEnd (/='/') $ Text.dropWhileEnd (=='/') a

str :: ByteString -> ByteString
str t =
  let
    len = B.length t
  in
    int len <> padBS len t

-- | Distance to the next multiple of 8
padLen :: Int -> Int
padLen n = (8 - n) `mod` 8

padBS :: Int -> ByteString -> ByteString
padBS strSize bs = bs <> B.replicate (padLen strSize) 0

int :: Integral a => a -> ByteString
int n = Serial.runPut $ Serial.putInt64le $ fromIntegral n

-- | dumpString
-- dump a string to nar
dumpString ::  ByteString -> NarSource Store.Remote.MonadStore
dumpString text yield = do
  yield $ str "nix-archive-1"
  yield $ str "("
  yield $ str "type" 
  yield $ str "regular"
  yield $ str "contents"
  yield $ str text
  yield $ str ")"

addToFile :: System.Nix.StorePath.StorePathName -> ByteString -> IO (Either ErrorCall Store.StorePath)
addToFile name content = do 
    res <- Store.Remote.runStore $ addToStore1 name (dumpString content) False (const True) False
    either
        Left -- err
        pure  -- path
        <$> parseStoreResult "addToFile" res

type NarSource m =  (ByteString -> m ()) -> m ()

-- | Pack `Nar` and add it to the store.
addToStore1
  :: Store.StorePathName        -- ^ Name part of the newly created `StorePath`
  -> NarSource Store.Remote.MonadStore -- ^ provide nar stream
  -> Bool                 -- ^ Add target directory recursively
  -> (FilePath -> Bool)   -- ^ Path filter function
  -> RepairFlag           -- ^ Only used by local store backend
  -> Store.Remote.MonadStore Store.StorePath
addToStore1 name source recursive _pathFilter _repair = do
  Store.runOpArgsIO AddToStore $ \yield -> do
    yield $ toStrict $ Data.Binary.Put.runPut $ do
      putText $ System.Nix.StorePath.unStorePathName name
      putBool $ not $ System.Nix.Hash.algoName @Hash.SHA256 == "sha256" && recursive
      putBool recursive
      putText $ System.Nix.Hash.algoName @Hash.SHA256
    source yield
  sockGetPath

-- ** Instances

instance MonadHttp IO where
  getURL url =
    do
      let urlstr = toString url
      traceM $ "fetching HTTP URL: " <> urlstr
      req     <- parseRequest urlstr
      manager <-
        bool
          (newManager defaultManagerSettings)
          newTlsManager
          (secure req)
      response <- httpLbs (req { method = "GET" }) manager
      let status = statusCode $ responseStatus response
      let body = (B.concat . BL.toChunks) $ responseBody response
      let digest::Hash.Digest Hash.SHA256 = Hash.hash body
      let name = baseNameOf url
      bool
        (pure $ Left $ ErrorCall $ "fail, got " <> show status <> " when fetching url = " <> urlstr)
        (either (\ err -> pure $ Left $ ErrorCall $ "name: '" <> toString name <> "' is not a valid path name: " <> err)
              (\x -> do
                res <- addToFile (System.Nix.StorePath.StorePathName name) body 
                either print print res
                print $ Store.makeFixedOutputPath "/nix/store" False digest x
                pure $ Right. toStorePath . Store.makeFixedOutputPath "/nix/store" False digest $ x)
              (Store.makeStorePathName name))
        (status == 200)

hand revising the hnix store addToStore api is working. But it should be put to hnix-store I have open a draft to shape the api haskell-nix/hnix-store#177.

soulomoon commented 2 years ago

The work should continue after haskell-nix/hnix-store#177 is merged or haskell-nix/hnix-store#176 is solved.