streamingfast / dstore

Blob stores abstractions. Supports AWS S3, Google Storage, Azure Blob File Storage, and local FS
https://dfuse.io
Apache License 2.0
11 stars 10 forks source link

dstore should add microsecond timestamp when uploading files to s3 bucket #10

Open matthewdarwin opened 1 year ago

matthewdarwin commented 1 year ago

ceph s3 buckets support microseconds timestamps, as seen with this example:

$ echo "hello" | rclone rcat march:test/xxxx; rclone lsl march:test/xxxx
        6 2023-08-21 13:04:18.116438610 xxxx

However, when dstore uploads files to s3, all the microseconds are 0000000.

$ rclone lsl march:bsc-sf-v4-one-blocks | head
   100915 2023-08-20 22:37:15.000000000 0031030100-26450c12d1f5f2bd-59cc49ee4a18b3ce-31030097-x209.dbin.zst
   100917 2023-08-20 22:37:16.000000000 0031030100-26450c12d1f5f2bd-59cc49ee4a18b3ce-31030097-x37.dbin.zst
   101048 2023-08-20 22:37:16.000000000 0031030100-26450c12d1f5f2bd-59cc49ee4a18b3ce-31030097-x38.dbin.zst
    66760 2023-08-20 22:37:19.000000000 0031030101-d25a0ceb678bd587-26450c12d1f5f2bd-31030098-x209.dbin.zst
    66842 2023-08-20 22:37:18.000000000 0031030101-d25a0ceb678bd587-26450c12d1f5f2bd-31030098-x37.dbin.zst
    66868 2023-08-20 22:37:20.000000000 0031030101-d25a0ceb678bd587-26450c12d1f5f2bd-31030098-x38.dbin.zst
   128592 2023-08-20 22:37:22.000000000 0031030102-b72d39842237d55a-d25a0ceb678bd587-31030099-x209.dbin.zst
   128547 2023-08-20 22:37:22.000000000 0031030102-b72d39842237d55a-d25a0ceb678bd587-31030099-x37.dbin.zst
   128693 2023-08-20 22:37:22.000000000 0031030102-b72d39842237d55a-d25a0ceb678bd587-31030099-x38.dbin.zst
   129610 2023-08-20 22:37:25.000000000 0031030103-4d20a18ccdb62d3e-b72d39842237d55a-31030100-x209.dbin.zst