search

aws / aws-iot-device-sdk-embedded-C

SDK for connecting to AWS IoT from a device using embedded C.
MIT License
974 stars 622 forks source link

s3 bucket download size #1905

Closed WeiWei00123 closed 2 months ago

WeiWei00123 commented 3 months ago

Currently using the http_demo_s3_download.c, I'm attempting to download files from the cloud to the device. However, I've encountered errors whenever the file size exceeds 200KB. Are there any parameters that need adjustment? image Additionally, I'd like to know how to download a tar.gz file from the cloud to the device. Would I still use http_demo_s3_download.c for this purpose?

ActoryOu commented 3 months ago

Hello, thank you for bringing this to our attention! We will be looking further into this issue as soon as we can, thank you for your patience!

ActoryOu commented 3 months ago

Hi @WeiWei00123, I tried to download >600KB file via this demo on s3, and that works on my environment without any change (only Detail Steps configured). Is it working while file size lower than 200KB in your environment? Could you share the entire log?

Additionally, I'd like to know how to download a tar.gz file from the cloud to the device. Would I still use http_demo_s3_download.c for this purpose?

Certainly, this demo should work no matter what kind of file you're downloading. So, yes, it should work to download tar.gz files. However, this particular demo only prints all responses from S3 on the console and does not store the file to the file system. You might need to modify that part of the code if you want to store the downloaded file.

Thank you.

WeiWei00123 commented 3 months ago

Thank you for your response. In my environment, the program works fine as long as the file size is less than 200KB. Below are the results of running two different files. Also, I'd like to inquire if there is support for uploading JPG files from the device to the cloud.

File size is 362KB file1_log.txt

File size is 161KB file2_log.txt

ActoryOu commented 3 months ago

Hi @WeiWei00123, Sorry for my late response. I had initially misunderstood your question, but upon further investigation, I realized that I have encountered a similar issue as you described.

After some research, it appears that the remote peer (AWS cloud) is closing the connection after several requests and responses by Encrypted Alert & FIN packet. My current suspicion is that this behavior might be related to the maxkeepaliverequests configuration setting in Apache on the server-side. This setting limits the number of requests that can be served through a single TCP connection.

There are a few potential solutions we could explore:

  1. Increasing the buffer size for 'RANGE_REQUEST_LENGTH' and 'USER_BUFFER_LENGTH'. This might help reduce the number of iterations required for a single connection. Please note that 'USER_BUFFER_LENGTH' should be larger than 'RANGE_REQUEST_LENGTH'.
  2. Implementing a reconnection mechanism when a disconnection occurs. I plan to work on developing this solution shortly.

I have reached out to the S3 team to verify my understanding of the issue and will keep you updated once I receive their feedback.

Please feel free to let me know if you have any further questions or concerns. I appreciate your patience, and I'm committed to resolving this matter as efficiently as possible.

Thank you.

WeiWei00123 commented 3 months ago

Hi @ActoryOu Adjusting 'RANGE_REQUEST_LENGTH' and 'USER_BUFFER_LENGTH' allowed successful downloads. Thank you for your assistance. Additionally, I would like to inquire if there is support for uploading JPG files from the device to an S3 bucket?

ActoryOu commented 3 months ago

I want to ensure I fully comprehend the situation so that I can provide the most appropriate assistance. Would you please elaborate further on the specific issue or question you have? Are you encountering any challenges or problems while uploading JPG files? I'll be happy to address your concerns once I have a clearer understanding of the context.

Thank you.

WeiWei00123 commented 3 months ago

Currently encountering some issues during upload: 1.When uploading text files, if the file size exceeds 65KB, an error message will be received. Attempts to increase the USER_BUFFER_LENGTH did not resolve the issue. however text files smaller than 65KB can be successfully uploaded. 2.When uploading text files using http_demo_s3_upload.c, it is successful by simply reading out the string. However, prior to uploading jpg files, they need to be converted into what data type? file_64KB.txt file_65KB.txt