$export and $import bulk data transmission using s3 bucket

[sridhar-shivraj]

1. When I do $import i am able to import the resource into the FHIR server but while do an $export i am facing The request signature we calculated does not match the signature you provided

2.$Import how to import a bundle from s3 bucket. As of now when I imported a bundle , the resource that are imported cannot be accessed individually , it is mostly looking like a contained resource not a bundle

[prb112]

@sridhar-shivraj I would like to confirm what you are asking.
1 - The Import supports files/content formatted in the NDJSON format. Are you asking if we support raw bundles? Please confirm. If so, we do not support resource type Bundle, we support NDJSON, no.
2 - To understand your issue further, please provide a copy of your request without the secrets.

[sridhar-shivraj]

Stack Dump = com.ibm.jbatch.container.exception.BatchContainerRuntimeException: com.ibm.cloud.objectstorage.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 1618ADC361D43EC3), S3 Extended Request ID: e0aa5714-0808-4cb2-80bf-5a58e9b224af at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.writeChunk(ChunkStepControllerImpl.java:503) at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeChunk(ChunkStepControllerImpl.java:636) at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.invokeCoreStep(ChunkStepControllerImpl.java:795) at com.ibm.jbatch.container.controller.impl.BaseStepControllerImpl.execute(BaseStepControllerImpl.java:295) at com.ibm.jbatch.container.controller.impl.ExecutionTransitioner.doExecutionLoop(ExecutionTransitioner.java:118) at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.executeCoreTransitionLoop(WorkUnitThreadControllerImpl.java:96) at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.executeWorkUnit(WorkUnitThreadControllerImpl.java:178) at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl$AbstractControllerHelper.runExecutionOnThread(WorkUnitThreadControllerImpl.java:503) at com.ibm.jbatch.container.controller.impl.WorkUnitThreadControllerImpl.runExecutionOnThread(WorkUnitThreadControllerImpl.java:92) at com.ibm.jbatch.container.util.BatchWorkUnit.run(BatchWorkUnit.java:113) at com.ibm.ws.context.service.serializable.ContextualRunnable.run(ContextualRunnable.java:79) at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) at java.base/java.lang.Thread.run(Thread.java:832) Caused by: com.ibm.cloud.objectstorage.services.s3.model.AmazonS3Exception: The request signature we calculated does not match the signature you provided. Check your key and signing method. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 1618ADC361D43EC3), S3 Extended Request ID: e0aa5714-0808-4cb2-80bf-5a58e9b224af at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712) at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367) at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113) at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770) at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744) at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726) at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686) at com.ibm.cloud.objectstorage.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668) at com.ibm.cloud.objectstorage.http.AmazonHttpClient.execute(AmazonHttpClient.java:532) at com.ibm.cloud.objectstorage.http.AmazonHttpClient.execute(AmazonHttpClient.java:512) at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3985) at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3932) at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:3080) at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1141) at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1131) at com.ibm.cloud.objectstorage.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1271) at com.ibm.fhir.bulkexport.system.ChunkWriter.writeItems(ChunkWriter.java:188) at com.ibm.jbatch.container.artifact.proxy.ItemWriterProxy.writeItems(ItemWriterProxy.java:80) at com.ibm.jbatch.container.controller.impl.ChunkStepControllerImpl.writeChunk(ChunkStepControllerImpl.java:481)

[sridhar-shivraj]

"jobParameters": { "cos.bucket.name": "fhir-r4-connectathon", "cos.location": "us", "cos.endpointurl": "https://ostore.****.tech/", "cos.credential.ibm": "N", "cos.api.key": "*", "cos.srvinst.id": "**" }

[sridhar-shivraj]

1)The Import supports files/content formatted in the NDJSON format. Are you asking if we support raw bundles? Please confirm. If so, we do not support resource type Bundle, we support NDJSON, no.

a)ya i was asking if you provide support to raw bundle, ya i see NDJSON is working fine b)is there any validation supported while importing the data

[prb112]

Re: b) Not by default. The code that does the import does support validation, however, the BulkData does not currently expose it.

Please note the FHIR Bundle request is supported directly with the IBM FHIR Server.

I think we've answered the questions. Please let us know if you need anything else.

Transitioned to Review/QA, if no further questions/actions, we'll close on 19-JUN.

[albertwang-ibm]

@sridhar-shivraj I have reproduced the "The request signature we calculated does not match the signature you provided" issue, and it can still be reproduced even after I have upgraded my ibm cos sdk from 2.6.1 to 2.6.2. After spending hours trying to figure out what happens but still without too much clue, then I have just created a new user with new secrets, and then it works well now.

So, Could you try a new user and secrets? or maybe just new secrets?

[sridhar-shivraj]

@albertwang-ibm we are using minio(s3 compatiable) so below is the log message in s3 ,while performing $export operation 04:30:28.279 [403 Forbidden] s3.GetBucketACL ostore.314ecorp.tech/fhir-r4-connectathon/?acl 14.141.179.130 560µs ↑ 229 B ↓ 657 B 04:30:28.290 [403 Forbidden] s3.GetBucketACL ostore.314ecorp.tech/fhir-r4-connectathon/?acl 14.141.179.130 352µs ↑ 229 B ↓ 657 B

[albertwang-ibm]

we don't officially support minio even though it declares as s3 compatible, IBM COS is also s3 compatible, and the codes work with IBM COS without any issue. I'm going to create an new issue for supporting minio.

[albertwang-ibm]

issue #1242 is created for supporting minio.

[kesavkolla]

why is bulk operation checking for ACL? It seems bizzare that it's performing ACL checks for bucket

[albertwang-ibm]

it's the S3(IBM COS) SDK behavior, we just simply call the S3 API doesBucketExistV2 with the bucket name to check if the bucket exists or or not before writing to it.