S3 UploadTask fails with some files

[salva116ario]

Describe the bug

I use aws-android-sdk-s3 in an Android mobile application to upload a text file from the mobile device to my s3 bucket. The text file i want to upload has always the same path on mobile device: /data/data/{{my_app_package}}/files/atlasmobile.log

Using aws-android-sdk-s3 version 2.22.1, i noticed that upload systematically fails with some text files. It seems I encountered the same problem as described in #717: ETag is null in the returned object metadata, then in putObject method, the line

final byte[] serverSideHash = BinaryUtils.fromHex(returnedMetadata.getETag()); 

throws NPE (but contrary to what was indicated in #717 , in my case, file is not uploaded correctly...)

With aws-android-sdk-s3 updated in version 2.74.0, I still have the same problem of upload failure for some files, even though no exceptions are thrown.

To Reproduce

Here is an example of file content for which uploadTask fails systematically : (call him FAIL_CONTENT)

java.lang.NullPointerException: Attempt to invoke virtual method 'java.util.List fr.xxx.atlasmobile.shared.model.relation.Vehicle.getRelations()' on a null object reference
    at fr.xxx.atlasmobile.shared.tools.DepotUtil.getUnknownPostalDepots$lambda$4(DepotUtil.kt:49)
    at fr.xxx.atlasmobile.shared.tools.DepotUtil.$r8$lambda$kwnNekirNUSQvlQCF_HvrLHPlZc(Unknown Source:0)
    at fr.xxx.atlasmobile.shared.tools.DepotUtil$$ExternalSyntheticLambda1.accept(Unknown Source:4)
    at java.util.ArrayList.forEach(ArrayList.java:1262)
    at fr.xxx.atlasmobile.shared.tools.DepotUtil.getUnknownPostalDepots(DepotUtil.kt:48)
    at fr.xxx.atlasmobile.n2truck.storage.RepositoryN2Truck.lambda$getAtlasRelations$16(RepositoryN2Truck.java:288)
    at fr.xxx.atlasmobile.n2truck.storage.RepositoryN2Truck.$r8$lambda$65bRNpBOsErQurdvryfyw3Gr74I(Unknown Source:0)
    at fr.xxx.atlasmobile.n2truck.storage.RepositoryN2Truck$$ExternalSyntheticLambda12.accept(Unknown Source:10)
    at com.xxx.bscc.http.apim.impl.ApimProcessor.lambda$sendCallerSuccess$1(ApimProcessor.java:403)
    at com.xxx.bscc.http.apim.impl.ApimProcessor.$r8$lambda$o8EFSfFZO6s6II4y0-qiKbfHx-M(Unknown Source:0)
    at com.xxx.bscc.http.apim.impl.ApimProcessor$$ExternalSyntheticLambda4.run(Unknown Source:6)
    at java.lang.Thread.run(Thread.java:764)

2024.01.18 at 11:05:17:430 GMT+01:00 Error : CRASH :2024.02.23 at 12:00:32:145 GMT+01:00 Debug : POP-UP : DialogPassword Rapport d'erreur Entrez le mot de passe pour envoyer le rapport d'erreur
2024.02.23 at 12:00:50:351 GMT+01:00 Debug : POP-UP : 'ENVOYER' clicked

Now, an example of file content for which uploadTask is OK: (call him CONTENT_OK : it looks like first content, minus some lines in the error stacktrace...)

java.lang.NullPointerException: Attempt to invoke virtual method 'java.util.List fr.xxx.atlasmobile.shared.model.relation.Vehicle.getRelations()' on a null object reference
    at fr.xxx.atlasmobile.shared.tools.DepotUtil.getUnknownPostalDepots$lambda$4(DepotUtil.kt:49)
    at fr.xxx.atlasmobile.shared.tools.DepotUtil.$r8$lambda$kwnNekirNUSQvlQCF_HvrLHPlZc(Unknown Source:0)
    at fr.xxx.atlasmobile.shared.tools.DepotUtil$$ExternalSyntheticLambda1.accept(Unknown Source:4)
    at java.util.ArrayList.forEach(ArrayList.java:1262)

2024.01.18 at 11:05:17:430 GMT+01:00 Error : CRASH : 2024.02.23 at 11:58:43:080 GMT+01:00 Debug : POP-UP : DialogPassword Rapport d'erreur Entrez le mot de passe pour envoyer le rapport d'erreur
2024.02.23 at 11:58:46:954 GMT+01:00 Debug : POP-UP : 'ENVOYER' clicked

Here is the method called in my app when i want to upload file :

    @JvmStatic
    fun uploadFile(
        filename: String,
        fileContent: InputStream,
        metadata: ObjectMetadata,
        context: Context
    ): MutableLiveData<Boolean> {
        val liveData = MutableLiveData<Boolean>()
        val transfert = TransferUtility.builder().context(context).s3Client(amazonS3Client).build()
        val options: UploadOptions = UploadOptions.builder().bucket(panamaBucket).objectMetadata(metadata).build()
        val transfertObserver: TransferObserver = transfert.upload(filename, fileContent, options)

        TransferNetworkLossHandler.getInstance(context)

        transfertObserver.setTransferListener(object : TransferListener {
            override fun onStateChanged(id: Int, state: TransferState) {
                if (state == TransferState.COMPLETED) {
                    liveData.postValue(true)
                    Log.d(TAG, "s3 upload success")
                } else if (state == TransferState.FAILED) {
                    liveData.postValue(false)
                    Log.d(TAG, "s3 upload failed")
                }
            }

            override fun onProgressChanged(id: Int, bytesCurrent: Long, bytesTotal: Long) {
                //Not implemented
            }

            override fun onError(id: Int, ex: Exception) {
                Log.e(TAG, ex.toString())
                liveData.postValue(false)
            }
        })
        return liveData
    }

To summarize :

• If on my device, atlasmobile.log file content equals FAIL_CONTENT, then upload fails ( = file is not uploaded on my s3 bucket...)

1. with version 2.22.1 of aws-android-sdk-s3, there is a NPE on BinaryUtils.fromHex(returnedMetadata.getETag()); and in my app code i'm on the onError method inside uploadFile...

2. with version 2.74.0 of aws-android-sdk-s3, without exception (once uploadTask is finished, in my app code, i'm in the onStateChanged method with status == TransferState.COMPLETED)

Note that, with these 2 versions, as described in #717 returned object metadata etag is null. You can see it in SceenShots section.

• If on my device, atlasmobile.log file content equals CONTENT_OK, upload is correctly done.

Which AWS service(s) are affected? S3 upload

Expected behavior Successful upload ! :)

Screenshots

• With CONTENT_OK file, using v2.74.0, result of this call :

returnedMetadata = invoke(request, new S3MetadataResponseHandler(), bucketName, key);

is returnedMetadata.metadata =

• With FAIL_CONTENT file, using v2.74.0, result of this call :

returnedMetadata = invoke(request, new S3MetadataResponseHandler(), bucketName, key);

is returnedMetadata.metadata =

As you can see, there seems to be some formatting differences and "etag" field is missing here (as in #717 ).

Environment Information (please complete the following information):

• AWS Android SDK Versions: 2.74.0 & 2.22.1 : different behaviours but same problem (file not uploaded on bucket)
• Devices + Android Version: Zebra TC52 / Android 8.1.0 Samsung G8 / Android 10
• Specific to simulators: no

Additional context Add any other context about the problem here.

[ankpshah]

Hello thank you for creating this issue. Someone from our team will take look at this issue.

[ankpshah]

I believe this may be related to the specific contents of a file being uploaded. To help narrow down the potential cause of the issue, would it be possible for you to verify the integrity of the file contents? Here are a few steps that could help us diagnose the issue further:

Manual Upload Test: Could you attempt to manually upload the file using the S3 console? This can help us determine if the issue lies with the file itself or the way the SDK is handling the upload.

File Integrity Check: Before uploading, could we perform a checksum comparison (e.g., using MD5) of the file on the client side and then again after uploading (if possible) to ensure the file has not been corrupted during the process?

SDK Configuration: Please let me know if there are any specific SDK configurations or settings that I should double-check on my end that might affect the upload process.

[ankpshah]

Hello @salva116ario,

1. If the file is not private could you share that so we are able to replicate the issue.
2. What is the content type of the FAIL_CONTENT file? is it any different from CONTENT_OK file?
3. Whats the difference between contents of CONTENT_OK and FAIL_CONTENT file?

[salva116ario]

Hi, @ankpshah thank you for considering my request.

One way to reproduce my problem is, by example, to use these two files :

ko.log (... a file for which upload task fails) ok.log (... with this file, all it's OK !)

I can uplaod ok.log from my Android mobile app to my s3 bucket using your lib (so, upload from mobile device storage to s3 bucket) And I can uplaod ok.log from my PC (Windows) to my s3 bucket using WinSCP (so, from my PC storage to s3 bucket...)

But, with ko.log, the upload fails using both WinSCP (throws Internal Error) and from mobile app (but no error thrown).

It's tricky but it seems there is something wrong with this line in ko.log : at java.lang.Thread.run(Thread.java:764)

Now, note that the same ko.log file can be uploaded on another bucket from a SpringBoot app. So, if there is a problem with the content file, it seems that it happens only in some cases and then, we're currently trying to check our bucket configuration...

And, at last, i precise that if ko.log file is zipped before the upload, it works fine...

I'll keep you posted on the progress of our research as soon as possible.

[salva116ario]

Hi, @ankpshah,

we have finally understood why uploads fail for some files such as ko.log: it's actually because of the configuration of the waf (Web Application Firewall), which considers that these files present a potential security risk. The waf therefore blocks the upload but returns a 200 response, which explains why the upload is considered finalised in your code.

I can therefore close this issue.

Thank you.