When uploading big files the number of total parts gets bigger than the maximum (1000) and makes the upload fail. This happens probably because s3fs doesn't use the file size to calculate the chunk size and it rather uses the default value (5Mb) if no other value is provided.
Maybe we can assume that the file size is passed as a parameter (somehow making it mandatory) and when instantiating the s3fs object pass the chunk size, default_block_size, which should be the max of 5Mb and file_size/1000 (5Mb is the smallest we can go). This works on my head, but I haven't tried it yet, and there might be some issues ☺️
When uploading big files the number of total parts gets bigger than the maximum (1000) and makes the upload fail. This happens probably because s3fs doesn't use the file size to calculate the chunk size and it rather uses the default value (5Mb) if no other value is provided.
Maybe we can assume that the file size is passed as a parameter (somehow making it mandatory) and when instantiating the s3fs object pass the chunk size,
default_block_size, which should be the max of 5Mb andfile_size/1000(5Mb is the smallest we can go). This works on my head, but I haven't tried it yet, and there might be some issues ☺️Other alternatives are welcome.