Verify that the SFTP component can handle files of 1GB with default memory config

[jhorbulyk]

Other Feature Request

Description

Currently, the max supported file size is 100MB. The component can read & write files under this size without having to increase the required memory for the component beyond the default allocated memory of 250 MB. We should check (on a staging environment) if the component can handle files of up to 1GB in size without the need to increase the allocated component memory beyond 250 MB.

Definition of Done

One of:

• The component can manipulate files of 1 GB without needing to alter the component code or the available memory (only change the max file size env var.)
• We make the needed code changes so that the component can manipulate files of 1 GB without needing to increase the available component memory
• We identify a reason why files of 1 GB can not be manipulated without increasing the component memory.

[if0s]

Hi @jhorbulyk Can we transfer MAX_FILE_SIZE to config instead of using it as env variable?

[jhorbulyk]

Can we transfer MAX_FILE_SIZE to config instead of using it as env variable?

Not before https://github.com/elasticio/elasticio/issues/6172 is implemented.