search

elasticio / sftp-component

SFTP component for the elastic.io platform
Apache License 2.0
2 stars 3 forks source link

Lookup Files fails when it tries to download many files #68

Closed jhorbulyk closed 4 years ago

jhorbulyk commented 4 years ago

Steps to reproduce

  1. Create 1000 thousand files of size ~75KB and place them in a single folder on an sftp server. Some comands to help you are:
    • Create a file of 75 KB with dd if=/dev/urandom of=sampleFile.txt bs=75KB count=1
    • Duplicate that file 1000 times with for i in {1..1000}; do cp sampleFile.txt "sampleFile$i.txt"; done
  2. Create a flow of the form Simple Trigger -> SFTP:DownloadFiles with Behavior Type: Emit Individually Use {} for both samples. Set the flow to run once per hour.
  3. Run the flow.

Expected Result

Step 2 produces 1000 records without any errors.

Actual result

Get an error timeout of 10000ms exceeded