Internal storage cannot handle more than a thousand files

[TjerkNan]

Describe the bug When uploading more than 999 files, the interface will only show a 1000 files. I've tested this with 2000 files of 10K. In addition, when downloading the files, only a 1000 of the 2000 files are within the zip file on safari. On firefox the download fails.

If only 950 files are uploaded, the file count is OK in zips downloaded with safari and firefox. Also the download on firefox doesn't fail.

@emielvdveen I will provide you with a to-be-created tool to help you upload files to S3 for testing purposes.

To Reproduce Steps to reproduce the behavior:

1. Upload 1000+ files into the appropriate project location
2. Browse to the storage component of the data donation
3. Observe that the file count is 1000 even though more files have been uploaded
4. Observe that the download on firefox fails
5. Observe that the download on safari succeeds but does not contain the correct amount of files.

Expected behavior

• Correct number of files are shown in interface
• Correct number of files are present in succesfull downloaded zip
• Download on firefox succeeds.

Screenshots Firefox download fails: https://github.com/eyra/mono/assets/88683839/78367dec-d6fc-4891-9c11-b970f56bb6e5

Desktop (please complete the following information):

• safari
• firefox

[mellelieuwes]

Ik denk dat ik weet waarom dit is. S3 doet file listing per page van 1000 en de huidige code vraagt niet door.

[TjerkNan]

@emielvdveen script om te testen zit nu in devops master (git pull) zie ook link op basecamp naar notion pagina voor beetje context

[TjerkNan]

Solved, just dealt with 1100 files perfectly