When migrating data from The Hive 3.5.1 to 4.1.16 and using AWS S3 as the storage backend for the 4.1.16 cluster, file uploads fail with a 400 Bad Request due to missing uploadId.
Steps to Reproduce
Setup The Hive 4 with S3 storage backend. Relevant config section:
Migration starts warning on each S3 upload with the following:
2022-01-09 19:01:22,218 [WARN] from org.thp.thehive.migration.th4.Output in TheHiveMigration-akka.actor.default-dispatcher-16 [|55ea1a97] Unable to set avatar to user xxx.xxx@xxx.com: akka.stream.alpakka.s3.FailedUpload: Upload part 1 request failed. Response header: (HttpResponse(400 Bad Request,List(x-amz-request-id: 1PWGZ6FZ51TC6PQE, x-amz-id-2: dgDz0+nxd6CkG3jtwgeoOhxrm2YlUhUhp3PQLzaS7NhsGvkcnF/Ps7H4zNZ22XQP7bdNEu1sF3g=, Date: Sun, 09 Jan 2022 19:01:21 GMT, Server: AmazonS3, Connection: close),HttpEntity.Chunked(application/xml),HttpProtocol(HTTP/1.1))), response body: (<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidArgument</Code><Message>This operation does not accept partNumber without uploadId</Message><ArgumentName>partNumber</ArgumentName><ArgumentValue>partNumber</ArgumentValue><RequestId>1PWGZ6FZ51TC6PQE</RequestId><HostId>xxx</HostId></Error>).
No files are uploaded to S3 bucket
Possible Solutions
The Hive is currently using alpakka 2.0.2, but It seems that alpakka 3.0.4 introduces quite substantial changes to the AWS S3 library, which can possibly remedy the situation.
Since the core problem seems to be a missing uploadId value for the MultiPartUpload and there seem to be references to it in alpakkas 2.0.2 version here then possibly setting the uploadId when calling the alpakka MultiPartUpload function could also fix it.
Request Type
Bug
Work Environment
Problem Description
When migrating data from The Hive 3.5.1 to 4.1.16 and using AWS S3 as the storage backend for the 4.1.16 cluster, file uploads fail with a
400 Bad Request
due to missinguploadId
.Steps to Reproduce
Setup The Hive 4 with S3 storage backend. Relevant config section:
NB! Using IAM role for auth, hence setting alpakka credentials provider to default
Start migration with:
/opt/thehive/bin/migrate -d --output /etc/thehive/application.conf --main-organisation OrgName --input /etc/thehive/hive3.conf
Migration starts warning on each S3 upload with the following:
No files are uploaded to S3 bucket
Possible Solutions
uploadId
value for the MultiPartUpload and there seem to be references to it in alpakkas 2.0.2 version here then possibly setting theuploadId
when calling the alpakka MultiPartUpload function could also fix it.