aws 0.19.0 build failure with http-conduit 2.3.0

[ilovezfs]

Hackage matrix shows aws is now broken across the board: https://matrix.hackage.haskell.org/package/aws#GHC-8.2/aws-0.19

I am seeing the same failure locally.

Configuring library for aws-0.19..
Preprocessing library for aws-0.19..
Building library for aws-0.19..
[ 1 of 80] Compiling Aws.Ec2.InstanceMetadata ( Aws/Ec2/InstanceMetadata.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Ec2/InstanceMetadata.o )
[ 2 of 80] Compiling Aws.Network      ( Aws/Network.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Network.o )
[ 3 of 80] Compiling Aws.Core         ( Aws/Core.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Core.o )

Aws/Core.hs:185:5: warning: [-Wnoncanonical-monad-instances]
    Noncanonical ‘return’ definition detected
    in the instance declaration for ‘Monad (Response m)’.
    Either remove definition for ‘return’ or define as ‘return = pure’
    |
185 |     return x = Response mempty (Right x)
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 4 of 80] Compiling Aws.Iam.Core     ( Aws/Iam/Core.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Core.o )
[ 5 of 80] Compiling Aws.Iam.Internal ( Aws/Iam/Internal.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Internal.o )
[ 6 of 80] Compiling Aws.Iam.Commands.UpdateUser ( Aws/Iam/Commands/UpdateUser.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/UpdateUser.o )
[ 7 of 80] Compiling Aws.Iam.Commands.UpdateAccessKey ( Aws/Iam/Commands/UpdateAccessKey.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/UpdateAccessKey.o )
[ 8 of 80] Compiling Aws.Iam.Commands.PutUserPolicy ( Aws/Iam/Commands/PutUserPolicy.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/PutUserPolicy.o )
[ 9 of 80] Compiling Aws.Iam.Commands.ListUsers ( Aws/Iam/Commands/ListUsers.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/ListUsers.o )
[10 of 80] Compiling Aws.Iam.Commands.ListUserPolicies ( Aws/Iam/Commands/ListUserPolicies.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/ListUserPolicies.o )
[11 of 80] Compiling Aws.Iam.Commands.ListMfaDevices ( Aws/Iam/Commands/ListMfaDevices.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/ListMfaDevices.o )
[12 of 80] Compiling Aws.Iam.Commands.ListAccessKeys ( Aws/Iam/Commands/ListAccessKeys.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/ListAccessKeys.o )
[13 of 80] Compiling Aws.Iam.Commands.GetUserPolicy ( Aws/Iam/Commands/GetUserPolicy.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/GetUserPolicy.o )
[14 of 80] Compiling Aws.Iam.Commands.GetUser ( Aws/Iam/Commands/GetUser.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/GetUser.o )
[15 of 80] Compiling Aws.Iam.Commands.DeleteUserPolicy ( Aws/Iam/Commands/DeleteUserPolicy.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/DeleteUserPolicy.o )
[16 of 80] Compiling Aws.Iam.Commands.DeleteUser ( Aws/Iam/Commands/DeleteUser.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/DeleteUser.o )
[17 of 80] Compiling Aws.Iam.Commands.DeleteAccessKey ( Aws/Iam/Commands/DeleteAccessKey.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/DeleteAccessKey.o )
[18 of 80] Compiling Aws.Iam.Commands.CreateUser ( Aws/Iam/Commands/CreateUser.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/CreateUser.o )
[19 of 80] Compiling Aws.Iam.Commands.CreateAccessKey ( Aws/Iam/Commands/CreateAccessKey.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands/CreateAccessKey.o )
[20 of 80] Compiling Aws.Iam.Commands ( Aws/Iam/Commands.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam/Commands.o )
[21 of 80] Compiling Aws.Iam          ( Aws/Iam.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Iam.o )
[22 of 80] Compiling Aws.DynamoDb.Core ( Aws/DynamoDb/Core.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Core.o )

Aws/DynamoDb/Core.hs:1247:5: warning: [-Wnoncanonical-monad-instances]
    Noncanonical ‘return’ definition detected
    in the instance declaration for ‘Monad Parser’.
    Either remove definition for ‘return’ or define as ‘return = pure’
     |
1247 |     return a = Parser $ \_kf ks -> ks a
     |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Aws/DynamoDb/Core.hs:1258:5: warning: [-Wnoncanonical-monad-instances]
    Noncanonical ‘pure = return’ definition detected
    in the instance declaration for ‘Applicative Parser’.
    Move definition from ‘return’ to ‘pure’
     |
1258 |     pure  = return
     |     ^^^^^^^^^^^^^^
[23 of 80] Compiling Aws.DynamoDb.Commands.UpdateItem ( Aws/DynamoDb/Commands/UpdateItem.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/UpdateItem.o )
[24 of 80] Compiling Aws.DynamoDb.Commands.Table ( Aws/DynamoDb/Commands/Table.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/Table.o )
[25 of 80] Compiling Aws.DynamoDb.Commands.Scan ( Aws/DynamoDb/Commands/Scan.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/Scan.o )
[26 of 80] Compiling Aws.DynamoDb.Commands.Query ( Aws/DynamoDb/Commands/Query.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/Query.o )
[27 of 80] Compiling Aws.DynamoDb.Commands.PutItem ( Aws/DynamoDb/Commands/PutItem.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/PutItem.o )
[28 of 80] Compiling Aws.DynamoDb.Commands.GetItem ( Aws/DynamoDb/Commands/GetItem.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/GetItem.o )
[29 of 80] Compiling Aws.DynamoDb.Commands.DeleteItem ( Aws/DynamoDb/Commands/DeleteItem.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/DeleteItem.o )
[30 of 80] Compiling Aws.DynamoDb.Commands.BatchWriteItem ( Aws/DynamoDb/Commands/BatchWriteItem.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/BatchWriteItem.o )
[31 of 80] Compiling Aws.DynamoDb.Commands.BatchGetItem ( Aws/DynamoDb/Commands/BatchGetItem.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands/BatchGetItem.o )
[32 of 80] Compiling Aws.DynamoDb.Commands ( Aws/DynamoDb/Commands.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb/Commands.o )
[33 of 80] Compiling Aws.DynamoDb     ( Aws/DynamoDb.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/DynamoDb.o )
[34 of 80] Compiling Aws.Aws          ( Aws/Aws.hs, /tmp/matrix-worker/1518373364/dist-newstyle/build/x86_64-linux/ghc-8.2.2/aws-0.19/build/Aws/Aws.o )

Aws/Aws.hs:254:96: error:
    • Couldn't match type ‘C.ConduitM
                             i0 B.ByteString (ResourceT IO) ()’
                     with ‘C.ResumableSource (ResourceT IO) B.ByteString’
      Expected type: HTTP.Response
                       (C.ResumableSource (ResourceT IO) B.ByteString)
        Actual type: HTTP.Response
                       (C.ConduitM i0 B.ByteString (ResourceT IO) ())
    • In the fourth argument of ‘responseConsumer’, namely ‘hresp’
      In the expression:
        responseConsumer httpRequest request metadataRef hresp
      In a stmt of a 'do' block:
        {-# SCC "unsafeAwsRef:responseConsumer" #-}
        responseConsumer httpRequest request metadataRef hresp
    • Relevant bindings include
        hresp :: HTTP.Response
                   (C.ConduitM i0 B.ByteString (ResourceT IO) ())
          (bound at Aws/Aws.hs:250:3)
    |
254 |   {-# SCC "unsafeAwsRef:responseConsumer" #-} responseConsumer httpRequest request metadataRef hresp
    |                                                                                                ^^^^^
<<ghc: 49363463200 bytes, 1410 GCs, 94653460/262510064 avg/max bytes residency (18 samples), 675M in use, 0.001 INIT (0.001 elapsed), 63.641 MUT (66.476 elapsed), 14.875 GC (14.885 elapsed) :ghc>>

[DanBurton]

Came here to report the same. Here's how I reproduced locally:

stack unpack aws-0.19 && cd aws-0.19
edit stack.yaml # add the following stack.yaml
stack build

# stack.yaml
resolver: nightly-2018-03-10
extra-deps:
- http-conduit-2.3.0

[ilovezfs]

@aristidb any ideas here?

[joeyh]

@aristidb, I've made some progress on this in https://github.com/joeyh/aws -- that fork successfully builds with conduit-1.3.

I'm hesitant to make a pull request, because I don't understand conduit well enough to know if my changes may have caused a subtle bug along the lines of a http resource not being cleaned up as promptly as it was before. See 3758da0850575dba66fbac9a2ca45ee5ef7fc3c3 for details.

Also, I've not run the test suite at all. Anyway, I hope this will help.

[DanBurton]

To run the test suite:

diff --git a/aws.cabal b/aws.cabal
index 7c7bc09..13d3e67 100644
--- a/aws.cabal
+++ b/aws.cabal
@@ -431,7 +431,7 @@ test-suite s3-tests
         QuickCheck >= 2.7,
         aeson >= 0.7,
         bytestring,
-        conduit-combinators,
+        conduit,
         errors >= 2.0,
         lifted-base >= 0.2,
         monad-control >= 0.3,

diff --git a/tests/S3/Main.hs b/tests/S3/Main.hs
index 6b8c8d9..5888151 100644
--- a/tests/S3/Main.hs
+++ b/tests/S3/Main.hs
@@ -236,7 +236,7 @@ test_versioning = askOption $ \(BucketOption bucket) ->
         let Just vid = cmurVersionId resp
         bs <- runResourceT $ do
             gor <- pureAws cfg s3cfg mgr (getObject bucket k) { goVersionId = Just vid }
-            responseBody (gorResponse gor) $$+- sinkLazy
+            sealConduitT (responseBody (gorResponse gor)) $$+- sinkLazy

         assertEqual "data do not match" testStr bs
     ]

Using these changes, I was able to "run" the test suite locally against lts-11.6. Of course it didn't really run the test suite since I didn't provide aws creds, but this at least compiles.

[joeyh]

Thanks Dan, I've updated my fork with that.

Investigating the consequences of the removed closeResumableSource line, I see that was added as part of a memory leak in S3 multipart upload fix in 6b9c070e2ce5ea7e967c3efec9ceb0791fd21d9f. So, it may be that dealing with conduit's dropping of finalization may have led to my fork reintroducing issue #142.

The thing to test would thus be Examples/MultipartUpload.hs with a 500 mb file and see if it leaks memory.

-- see shy jo

[joeyh]

Tested /usr/bin/time dist/build/MultipartUpload/MultipartUpload s3.amazonaws.com joeyh-test-23-apr 1gbtest 1gbtest 100

It grew to 240 mb memory, which is the same as what it used once #142 was fixed. The 1gb memory leak is not reintroduced by my fork.

[ilovezfs]

@joeyh I'm trying to get git-annex to build with GHC 8.4 using your aws fork and hit this:

Preprocessing library for aws-0.18..
Building library for aws-0.18..
[ 4 of 80] Compiling Aws.Iam.Core     ( Aws/Iam/Core.hs, dist/dist-sandbox-215793fc/build/Aws/Iam/Core.o )

Aws/Iam/Core.hs:63:10: error:
    • No instance for (Semigroup IamMetadata)
        arising from the superclasses of an instance declaration
    • In the instance declaration for ‘Monoid IamMetadata’
   |
63 | instance Monoid IamMetadata where
   |          ^^^^^^^^^^^^^^^^^^
Failed to install aws-0.18
cabal: Error: some packages failed to install:
aws-0.18-AFSHEQTGVYdDavj3oZcjYf failed during the building phase. The
exception was:
ExitFailure 1

[joeyh]

@ilovezfs https://prime.haskell.org/wiki/Libraries/Proposals/SemigroupMonoid

-- see shy jo

[ilovezfs]

@joeyh so I guess that will require another fix in aws?

[aristidb]

The semigroup issue at least should be fixed with a PR that I just merged.

[ilovezfs]

Nice! @aristidb did you have a chance to review @joeyh's fork?

[aristidb]

@ilovezfs I don't understand the new conduit stuff well enough to know if it's good or not, I'm afraid. But once @joeyh is more confident in its stability, I'm sure he will make a PR so we can merge it. :+1:

[ilovezfs]

@joeyh any news here?

[joeyh]

https://github.com/aristidb/aws/pull/247

[aristidb]

@joeyh PR is merged, will release 0.20 shortly

[mat8913]

Could you please make a metadata revision on hackage? cabal still tries to build aws-0.19 with http-conduit-2.3.2 on my computer.

[hvr]

@mat8913 et al.; I've performed the metadata revision as part of the effort to to repair git-annex's install-plans (see https://github.com/haskell-infra/hackage-trustees/issues/178) at https://hackage.haskell.org/package/aws-0.19/revisions/

In the future, please file an issue in the Trustee issue tracker to help make sure the metadata is corrected asap via metadata revisions to help users have a good experience with Hackage!