search

versity / versitygw

versity s3 gateway
https://www.versity.com/products/versitygw/
Apache License 2.0
180 stars 23 forks source link

[Bug] - os.PathSeparator incorrect for io/fs #903

Open lunixbochs opened 4 days ago

lunixbochs commented 4 days ago

Describe the bug backend.Walk constructs paths using os.PathSeparator

This has two problems:

  1. io/fs uses forward slash on all platforms, including windows "Note that paths are slash-separated on all systems, even Windows", so os.PathSeparator is incorrect for the io/fs abstraction.
  2. The prefix passed in by the caller probably wants to work with either / or delimiter, not os.PathSeparator

I'm looking at speeding up backend.Walk. I'll see about improving this while I'm in there.

benmcclelland commented 4 days ago

@lunixbochs Thanks for taking a look at this. I think you are probably right, not much thought was put into using os.PathSeparator here.

One optimization that I was trying to figure out if it would be possible is somehow storing the directory and readdir position in the continuation marker. Ideally we would like to continue the walk without having to visit all the previous entries. The current walk really restricts the bucket scalability to be able to support large bucket list-objects.