search

noobaa / noobaa-core

High-performance S3 application gateway to any backend - file / s3-compatible / multi-clouds / caching / replication ...
https://www.noobaa.io
Apache License 2.0
268 stars 78 forks source link

MD5 etag retrieval for list objects and other lists #6878

Closed jeniawhite closed 6 months ago

jeniawhite commented 2 years ago

Environment info

Actual behavior

  1. Currently the MD5 etag is not getting queried upon listing of objects (we always return inode etag)
  2. We can uncomment the below code and have it (with a performance hit, since it is not optimized) 
                // r.fs_xattr = config.NSFS_CALCULATE_MD5 ? await this._get_fs_xattr_from_path(fs_account_config, entry_path) : undefined;
  3. We can maybe use some FS caching for performance
  4. Another option is not to use the same method for retrieval of xattrs (list and then get), and instead have a separate method for retrieval of md5 xattr (using single getxattr operation since we know the exact key that we want to query)

Expected behavior

  1. The list should return MD5 etag if it is chosen to be calculated by the system

Steps to reproduce

1.

More information - Screenshots / Logs / Other output

nimrod-becker commented 2 years ago

I think gving a dedicated API (within the FS lib in this case) to retrieve the list with all the needed data for the response would be the way to go

guymguym commented 2 years ago

@romayalon you meant this one?

romayalon commented 2 years ago

@guymguym yes

nimrod-becker commented 6 months ago

@jackyalbo fixed by you?

jackyalbo commented 6 months ago

retrieving xattr was pushed into stat as part of this PR: https://github.com/noobaa/noobaa-core/pull/6973 It applies for both list_objects and read_object_md.