files.list and files.delete are limited to 5000 files

[amol-anand]

Expected Behaviour

await files.delete('/') should delete all files

Actual Behaviour

files.delete('/') only deletes 5000 files

Reproduce Scenario (including but not limited to)

Run await files.delete('/') on an account that has more than 5000 files. Run files.list('/') before and after the delete command to see how many files were deleted.

Steps to Reproduce

Platform and Version

aio-lib-files version 1.3.1

Sample Code that illustrates the problem

  //Initialize aio-lib-files  
  const files = await filesLib.init();
  LOG.debug('aio-lib-files initialized');
  const allFiles = await files.list('/');
  LOG.debug('Listing all files. Number of files are: '+allFiles.length);
  const deletedFiles = await files.delete('/');
  LOG.debug('Number of deleted files: '+deletedFiles.length);

Logs taken while reproducing problem

No useful logging. "response": { "result": { "error": "The action did not produce a valid response and exited unexpectedly." }, "status": "action developer error", "success": false }

[meryllblanchet]

Thanks for raising the issue, @amol-anand ! This looks like a default limit of the underlying storage, but ideally the SDK should handle the operation in the most appropriate manner.

cc @purplecabbage @sandeep-paliwal @moritzraho for feedback

[aiojbot]

JIRA issue created: https://jira.corp.adobe.com/browse/ACNA-946

[moritzraho]

After some investigation I can conclude that there are two issues here:

• 1. Azure returns max 5000 files when listing, list is internally used by delete and copy which explains the bug with delete. We need to support storage with more than 5000 files, we should expose some sort of pagination and need to fix the returned number of files in list based on the memory constraints in runtime actions. files.delete('/') should still delete all files but internally list with pagination once we support this. Some changes to handle > 5000 files will be required by files.copy too.
• 2. Runtime action crashes when handling many files (5000) because execution hits resource limits both for memory (default is 256 mb) and number of opened file descriptors for open requests to Azure. To solve this, delete and copy should handle files in fix size batches (like 200 or so).