search

jcaddel / maven-s3-wagon

Multi-threaded wagon to connect Maven with Amazon S3
123 stars 50 forks source link

Upload failure: Input stream cannot be reset #2

Closed drsgoodall closed 12 years ago

drsgoodall commented 12 years ago

I've used various version of this plugin and they all exhibit the same occassion al problem. Uploads to s3 can fail with the following exception; j ava.io.IOException: Input stream cannot be reset as 18290688 bytes have been written, exceeding the available buffer size of 131072

After some investigation it seems the error is caused by TransferProgressFileInputStream not really being repeatable. Should the upload fail, the retry can fail as the buffer is not properly set up.

One solution should be to make TransferProgressFileInputStream extend the aws Re peatableFileInputStream instead.

drsgoodall commented 12 years ago

Stack trace;

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project PROJECT_NAME: Failed to deploy artifacts: Could not transfer artifact ARTIFACT_GAV from/to REPO_ID (s3://S3_BUCKET/PATH): Transfer of resource ARTIFACT.zipfailed: Unable to execute HTTP request: Input stream cannot be reset as 18290688 bytes have been written, exceeding the available buffer size of 131072 -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project PROJECT_NAME: Failed to deploy artifacts: Could not transfer artifact ARTIFACT_GAV from/to REPO_ID (s3://S3-BUCKET/PATH): Transfer of resource ARTIFACT.zipfailed at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to deploy artifacts: Could not transfer artifact ARTIFACT_GAV from/to REPO_ID (s3://S3-BUCKET/PATH): Transfer of resource ARTIFACT.zipfailed at org.apache.maven.plugin.deploy.DeployMojo.execute(DeployMojo.java:193) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) ... 19 more Caused by: org.apache.maven.artifact.deployer.ArtifactDeploymentException: Failed to deploy artifacts: Could not transfer artifact ARTIFACT_GAV from/to REPO_ID (s3://S3_BUCKET/PATH): Transfer of resource ARTIFACT.zipfailed at org.apache.maven.artifact.deployer.DefaultArtifactDeployer.deploy(DefaultArtifactDeployer.java:141) at org.apache.maven.plugin.deploy.AbstractDeployMojo.deploy(AbstractDeployMojo.java:167) at org.apache.maven.plugin.deploy.DeployMojo.execute(DeployMojo.java:188) ... 21 more Caused by: org.sonatype.aether.deployment.DeploymentException: Failed to deploy artifacts: Could not transfer artifact ARTIFACT_GAV from/to REPO_ID (s3://S3_BUCKET/PATH): Transfer of resource ARTIFACT.zipfailed at org.sonatype.aether.impl.internal.DefaultDeployer.deploy(DefaultDeployer.java:284) at org.sonatype.aether.impl.internal.DefaultDeployer.deploy(DefaultDeployer.java:215) at org.sonatype.aether.impl.internal.DefaultRepositorySystem.deploy(DefaultRepositorySystem.java:480) at org.apache.maven.artifact.deployer.DefaultArtifactDeployer.deploy(DefaultArtifactDeployer.java:137) ... 23 more Caused by: org.sonatype.aether.transfer.ArtifactTransferException: Could not transfer artifact ARTIFACT_GAV from/to REPO_ID (s3://S3_BUCKET/PATH): Transfer of resource ARTIFACT.zipfailed at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:949) at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:940) at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$PutTask.flush(WagonRepositoryConnector.java:864) at org.sonatype.aether.connector.wagon.WagonRepositoryConnector.put(WagonRepositoryConnector.java:466) at org.sonatype.aether.impl.internal.DefaultDeployer.deploy(DefaultDeployer.java:278) ... 26 more Caused by: org.apache.maven.wagon.TransferFailedException: Transfer of resource ARTIFACT.zipfailed at org.kuali.maven.wagon.AbstractWagon.handleException(AbstractWagon.java:264) at org.kuali.maven.wagon.AbstractWagon.put(AbstractWagon.java:248) at org.sonatype.aether.connector.wagon.WagonRepositoryConnector$PutTask.run(WagonRepositoryConnector.java:825) at org.sonatype.aether.connector.wagon.WagonRepositoryConnector.put(WagonRepositoryConnector.java:465) ... 27 more Caused by: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Input stream cannot be reset as 18290688 bytes have been written, exceeding the available buffer size of 131072 at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:293) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:168) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2502) at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1038) at org.kuali.maven.wagon.S3Wagon.putResource(S3Wagon.java:358) at org.kuali.maven.wagon.AbstractWagon.put(AbstractWagon.java:245) ... 29 more Caused by: java.io.IOException: Input stream cannot be reset as 18290688 bytes have been written, exceeding the available buffer size of 131072 at com.amazonaws.services.s3.internal.RepeatableInputStream.reset(RepeatableInputStream.java:84) at com.amazonaws.services.s3.internal.MD5DigestCalculatingInputStream.reset(MD5DigestCalculatingInputStream.java:60) at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:123) at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:96) at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108) at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:120) at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:263) at org.apache.http.impl.conn.AbstractClientConnAdapter.sendRequestEntity(AbstractClientConnAdapter.java:227) at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:255) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:645) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:464) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:261) ... 34 more

drsgoodall commented 12 years ago

Some discussion around issue;

https://forums.aws.amazon.com/thread.jspa?threadID=66753

jcaddel commented 12 years ago

Thanks for the feedback. I've noticed these same errors in our deployments from time to time but hadn't investigated them much. They occurred infrequently enough for me, that I just assumed they were network issues that nothing could be done about. I'll take a look at the info you've pointed out, and attempt to work that in. Sounds like the heavy lifting related to re-trying is done already, just need to take advantage of things correctly.

jcaddel commented 12 years ago

Version 1.1.10 (released today) should fix this. All input streams now go through Amazon's RepeatableFileInputStream.

There was one spot where a FileInputStream was being used and one spot where TransferProgressFileInputStream was being used.

The first spot now uses RepeatableFileInputStream and TransferProgressFileInputStream now extends RepeatableFileInputStream