search

peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.65k stars 236 forks source link

LastModified filter on s3 listing #388

Open tooptoop4 opened 2 years ago

tooptoop4 commented 2 years ago

as per https://stackoverflow.com/questions/45429556/how-list-amazon-s3-bucket-contents-by-modified-date

can s5cmd list only files modified after certain date (server side filtering not client side which is slow) ?

igungor commented 1 year ago

The link above does not mention there's a last-modified filtering on S3 side. ListObjectsV2 API has no server-side filtering based on LastModified date of objects.

If we implement this, filtering would be on the client side.

denizsurmeli commented 1 year ago

Can you give your use case for this feature? I'm asking this to understand whether your problem can be solved by a combination of the current features or not.

tooptoop4 commented 1 year ago

I need to check new logs but don't want to pay to list millions of old files in the same folder

denizsurmeli commented 1 year ago

Well, the implementation will list everything then filter the ones that are after the date you give, since there is no parameter in the request that will only list the items modified after some certain date. Hence, the same cost will occur. Maybe structuring your objects by date would help your problem, but the implementation you ask won't help you to reduce costs unfortunately.