Data in the area mounted with mountpoint for amazon s3 (mount-s3) cannot be viewed

mura890 commented 3 weeks ago

Mountpoint for Amazon S3 version

mount-s3 version：1.7.2

AWS Region

ap-northeast-1

Describe the running environment

I am running an S3 bucket in the same account mounted on EC2 (OS: CentOS7.9)

Mountpoint options

mount-s3 test-S3-bucket --prefix archives/ /mnt/applications/classtream/archives --uid 1006 --gid 1006 --file-mode 0660 --dir-mode 0770 --allow-delete --allow-other --allow-overwrite --cache /mnt/s3-cache --metadata-ttl 1

What happened?

Video files are stored in the "archives" folder, and the behavior and verification of the mount-s3 cache function is being carried out. The verification flow is to investigate when the stored video A is replaced with video B while playing video A, and when the video A being played changes, the video being played changes. During the verification, the video could not be viewed at all, and when I checked the mount destination folder with the following ls command,

ls /mnt/applications/classtream/archives

The S3 data could not be displayed.

ls: cannot open directory /mnt/applications/classtream/archives: Transport endpoint is not connected

When I checked the OS messages log, I found a large number of warnings and errors like the following. (Excerpt)

Relevant log output

00:42:20 tes-sv mount-s3[1370]: [WARN] get_object{id=99 bucket=test-S3-bucket key=archives/30301013/6.mp4 range=Some(15728640..16777216) if_match=Some(ETag("\"1893234a3bc4a68053e9c351232e8c8a-40\""))}: mountpoint_s3_client::s3_crt_client:  duration=35.130092ms request_id=GP8VGFZEKVY1JXCQ error=ServiceError(PreconditionFailed)meta request failed
00:42:20 tes-sv mount-s3[1370]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
00:42:20 tes-sv mount-s3[1370]: [WARN] read{req=102660 ino=32 fh=42 offset=15757312 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102661 ino=32 fh=42 offset=15757312 size=4096 name=6.mp4}: mountpoint_s3::logging: panic on Thread { id: ThreadId(8), name: Some("fuse-worker-3"), .. } at /workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9: cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102661 ino=32 fh=42 offset=15757312 size=4096 name=6.mp4}: mountpoint_s3::logging: backtrace:#012   0: tracing_panic_hook#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:65:21#012   1: {closure#0}#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:74:9#012   2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9#012   3: std::panicking::rust_panic_with_hook#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13#012   4: std::panicking::begin_panic_handler::{{closure}}#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:649:13#012   5: std::sys_common::backtrace::__rust_end_short_backtrace#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18#012   6: rust_begin_unwind#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5#012   7: core::panicking::panic_fmt#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14#012   8: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9#012   9: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/task.rs:47:49#012  10: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:507:66#012  11: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:450:43#012  12: {async_block#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:269:38#012  13: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<mountpoint_s3::checksums::ChecksummedBytes, mountpoint_s3::prefetch::PrefetchReadError<mountpoint_s3_client::s3_crt_client::S3RequestError>>> + core::marker::Send), alloc::alloc::Global>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9#012  14: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fs.rs:857:58#012  15: poll<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9#012  16: {closure#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:23#012  17: {closure#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:90:37#012  18: try_with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16#012  19: with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9#012  20: run_executor<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:86:27#012  21: block_on<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:5#012  22: read<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fuse.rs:145:15#012  23: dispatch_req<mountpoint_s3::fuse::S3FuseFilesystem<mountpoint_s3_client::s3_crt_client::S3CrtClient, mou
00:42:20 tes-sv mount-s3[1370]: [WARN] fuser::reply: Reply not sent for operation 102661, replying with I/O error
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102662 ino=32 fh=42 offset=15761408 size=4096 name=6.mp4}: mountpoint_s3::logging: panic on Thread { id: ThreadId(6), name: Some("fuse-worker-1"), .. } at /workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9: cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102662 ino=32 fh=42 offset=15761408 size=4096 name=6.mp4}: mountpoint_s3::logging: backtrace:#012   0: tracing_panic_hook#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:65:21#012   1: {closure#0}#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:74:9#012   2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9#012   3: std::panicking::rust_panic_with_hook#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13#012   4: std::panicking::begin_panic_handler::{{closure}}#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:649:13#012   5: std::sys_common::backtrace::__rust_end_short_backtrace#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18#012   6: rust_begin_unwind#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5#012   7: core::panicking::panic_fmt#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14#012   8: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9#012   9: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/task.rs:47:49#012  10: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:507:66#012  11: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:450:43#012  12: {async_block#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:269:38#012  13: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<mountpoint_s3::checksums::ChecksummedBytes, mountpoint_s3::prefetch::PrefetchReadError<mountpoint_s3_client::s3_crt_client::S3RequestError>>> + core::marker::Send), alloc::alloc::Global>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9#012  14: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fs.rs:857:58#012  15: poll<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9#012  16: {closure#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:23#012  17: {closure#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:90:37#012  18: try_with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16#012  19: with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9#012  20: run_executor<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:86:27#012  21: block_on<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:5#012  22: read<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fuse.rs:145:15#012  23: dispatch_req<mountpoint_s3::fuse::S3FuseFilesystem<mountpoint_s3_client::s3_crt_client::S3CrtClient, mou
・・・
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-2 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-3 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-4 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-5 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-6 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [WARN] get_object{id=100 bucket=test-S3-bucket key=archives/30301013/6.mp4 range=Some(15728640..16777216) if_match=Some(ETag("\"1893234a3bc4a68053e9c351232e8c8a-40\""))}: mountpoint_s3_client::s3_crt_client:  duration=46.14418ms request_id=GP8YWT0JPYZZF8FJ error=ServiceError(PreconditionFailed)meta request failed
00:42:20 tes-sv mount-s3[1370]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
00:42:20 tes-sv mount-s3[1370]: [WARN] read{req=102666 ino=32 fh=42 offset=15761408 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102667 ino=32 fh=42 offset=15761408 size=4096 name=6.mp4}: mountpoint_s3::logging: panic on Thread { id: ThreadId(13), name: Some("fuse-worker-8"), .. } at /workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9: cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102667 ino=32 fh=42 offset=15761408 size=4096 name=6.mp4}: mountpoint_s3::logging: backtrace:#012   0: tracing_panic_hook#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:65:21#012   1: {closure#0}#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:74:9#012   2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9#012   3: std::panicking::rust_panic_with_hook#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13#012   4: std::panicking::begin_panic_handler::{{closure}}#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:649:13#012   5: std::sys_common::backtrace::__rust_end_short_backtrace#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18#012   6: rust_begin_unwind#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5#012   7: core::panicking::panic_fmt#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14#012   8: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9#012   9: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/task.rs:47:49#012  10: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:507:66#012  11: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:450:43#012  12: {async_block#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:269:38#012  13: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<mountpoint_s3::checksums::ChecksummedBytes, mountpoint_s3::prefetch::PrefetchReadError<mountpoint_s3_client::s3_crt_client::S3RequestError>>> + core::marker::Send), alloc::alloc::Global>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9#012  14: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fs.rs:857:58#012  15: poll<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9#012  16: {closure#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:23#012  17: {closure#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:90:37#012  18: try_with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16#012  19: with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9#012  20: run_executor<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:86:27#012  21: block_on<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:5#012  22: read<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fuse.rs:145:15#012  23: dispatch_req<mountpoint_s3::fuse::S3FuseFilesystem<mountpoint_s3_client::s3_crt_client::S3CrtClient, mou
・・・
00:42:20 tes-sv mount-s3[1370]: [WARN] fuser::reply: Reply not sent for operation 102672, replying with I/O error
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-12 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102673 ino=32 fh=42 offset=15761408 size=4096 name=6.mp4}: mountpoint_s3::logging: panic on Thread { id: ThreadId(18), name: Some("fuse-worker-13"), .. } at /workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9: cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102673 ino=32 fh=42 offset=15761408 size=4096 name=6.mp4}: mountpoint_s3::logging: backtrace:#012   0: tracing_panic_hook#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:65:21#012   1: {closure#0}#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:74:9#012   2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9#012   3: std::panicking::rust_panic_with_hook#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13#012   4: std::panicking::begin_panic_handler::{{closure}}#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:649:13#012   5: std::sys_common::backtrace::__rust_end_short_backtrace#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18#012   6: rust_begin_unwind#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5#012   7: core::panicking::panic_fmt#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14#012   8: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9#012   9: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/task.rs:47:49#012  10: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:507:66#012  11: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:450:43#012  12: {async_block#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:269:38#012  13: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<mountpoint_s3::checksums::ChecksummedBytes, mountpoint_s3::prefetch::PrefetchReadError<mountpoint_s3_client::s3_crt_client::S3RequestError>>> + core::marker::Send), alloc::alloc::Global>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9#012  14: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fs.rs:857:58#012  15: poll<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9#012  16: {closure#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:23#012  17: {closure#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:90:37#012  18: try_with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16#012  19: with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9#012  20: run_executor<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:86:27#012  21: block_on<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:5#012  22: read<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fuse.rs:145:15#012  23: dispatch_req<mountpoint_s3::fuse::S3FuseFilesystem<mountpoint_s3_client::s3_crt_client::S3CrtClient, mou
00:42:20 tes-sv mount-s3[1370]: [WARN] fuser::reply: Reply not sent for operation 102673, replying with I/O error
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-13 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102674 ino=32 fh=42 offset=15728640 size=4096 name=6.mp4}: mountpoint_s3::logging: panic on Thread { id: ThreadId(19), name: Some("fuse-worker-14"), .. } at /workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9: cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102674 ino=32 fh=42 offset=15728640 size=4096 name=6.mp4}: mountpoint_s3::logging: backtrace:#012   0: tracing_panic_hook#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:65:21#012   1: {closure#0}#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:74:9#012   2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9#012   3: std::panicking::rust_panic_with_hook#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13#012   4: std::panicking::begin_panic_handler::{{closure}}#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:649:13#012   5: std::sys_common::backtrace::__rust_end_short_backtrace#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18#012   6: rust_begin_unwind#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5#012   7: core::panicking::panic_fmt#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14#012   8: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9#012   9: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/task.rs:47:49#012  10: {async_block#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:298:66#012  11: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<mountpoint_s3::checksums::ChecksummedBytes, mountpoint_s3::prefetch::PrefetchReadError<mountpoint_s3_client::s3_crt_client::S3RequestError>>> + core::marker::Send), alloc::alloc::Global>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9#012  12: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fs.rs:857:58#012  13: poll<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9#012  14: {closure#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:23#012  15: {closure#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:90:37#012  16: try_with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16#012  17: with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9#012  18: run_executor<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:86:27#012  19: block_on<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:5#012  20: read<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fuse.rs:145:15#012  21: dispatch_req<mountpoint_s3::fuse::S3FuseFilesystem<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>#012             at ./workplace/mountpoint-s3/vendor/fuser/src/request.rs:308:17#012  22: dispatch<mountpoint_s3::fuse::S3FuseFilesystem<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>#012             at ./workplace/mountpoint-s3/vend
00:42:20 tes-sv mount-s3[1370]: [WARN] fuser::reply: Reply not sent for operation 102674, replying with I/O error
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-14 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102675 ino=32 fh=42 offset=15765504 size=4096 name=6.mp4}: mountpoint_s3::logging: panic on Thread { id: ThreadId(20), name: Some("fuse-worker-15"), .. } at /workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9: cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102675 ino=32 fh=42 offset=15765504 size=4096 name=6.mp4}: mountpoint_s3::logging: backtrace:#012   0: tracing_panic_hook#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:65:21#012   1: {closure#0}#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/logging.rs:74:9#012   2: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9#012   3: std::panicking::rust_panic_with_hook#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13#012   4: std::panicking::begin_panic_handler::{{closure}}#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:649:13#012   5: std::sys_common::backtrace::__rust_end_short_backtrace#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18#012   6: rust_begin_unwind#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5#012   7: core::panicking::panic_fmt#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14#012   8: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9#012   9: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3RequestError>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch/task.rs:47:49#012  10: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:507:66#012  11: {async_fn#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:450:43#012  12: {async_block#0}<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>, mountpoint_s3_client::s3_crt_client::S3CrtClient>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/prefetch.rs:269:38#012  13: poll<alloc::boxed::Box<(dyn core::future::future::Future<Output=core::result::Result<mountpoint_s3::checksums::ChecksummedBytes, mountpoint_s3::prefetch::PrefetchReadError<mountpoint_s3_client::s3_crt_client::S3RequestError>>> + core::marker::Send), alloc::alloc::Global>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9#012  14: {async_fn#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fs.rs:857:58#012  15: poll<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9#012  16: {closure#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:23#012  17: {closure#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:90:37#012  18: try_with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:284:16#012  19: with<alloc::sync::Arc<futures_executor::local_pool::ThreadNotify, alloc::alloc::Global>, futures_executor::local_pool::run_executor::{closure_env#0}<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>, core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>>#012             at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/thread/local.rs:260:9#012  20: run_executor<core::result::Result<bytes::bytes::Bytes, mountpoint_s3::fs::error::Error>, futures_executor::local_pool::block_on::{closure_env#0}<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:86:27#012  21: block_on<tracing::instrument::Instrumented<mountpoint_s3::fs::{impl#10}::read::{async_fn_env#0}<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>>>#012             at ./root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:5#012  22: read<mountpoint_s3_client::s3_crt_client::S3CrtClient, mountpoint_s3::prefetch::Prefetcher<mountpoint_s3::prefetch::caching_stream::CachingPartStream<mountpoint_s3::data_cache::disk_data_cache::DiskDataCache, mountpoint_s3_crt::io::event_loop::EventLoopGroup>>>#012             at ./workplace/mountpoint-s3/mountpoint-s3/src/fuse.rs:145:15#012  23: dispatch_req<mountpoint_s3::fuse::S3FuseFilesystem<mountpoint_s3_client::s3_crt_client::S3CrtClient, mou
00:42:20 tes-sv mount-s3[1370]: [WARN] fuser::reply: Reply not sent for operation 102675, replying with I/O error
00:42:20 tes-sv mount-s3[1370]: [ERROR] mountpoint_s3::fuse::session: worker thread panicked thread_name=fuse-worker-15 panic_msg=cannot use a PartQueue after failure
00:42:20 tes-sv mount-s3[1370]: [WARN] fuser::mnt::fuse2: umount failed with Os { code: 16, kind: ResourceBusy, message: "Device or resource busy" }
00:43:27 tes-sv systemd: Stopping Mountpoint for Amazon S3 mount...
00:43:27 tes-sv fusermount: /usr/bin/fusermount: failed to unmount /mnt/applications/classtream/archives: Invalid argument

passaro commented 3 weeks ago

Hi @mura890, this is definitely an issue and we are looking into it:

00:42:20 tes-sv mount-s3[1370]: [ERROR] read{req=102661 ino=32 fh=42 offset=15757312 size=4096 name=6.mp4}: mountpoint_s3::logging: panic on Thread { id: ThreadId(8), name: Some("fuse-worker-3"), .. } at /workplace/mountpoint-s3/mountpoint-s3/src/prefetch/part_queue.rs:57:9: cannot use a PartQueue after failure

passaro commented 3 weeks ago

Regarding your test:

The verification flow is to investigate when the stored video A is replaced with video B while playing video A, and when the video A being played changes, the video being played changes

Even once we address the issue, the behavior to expect in this scenario depends on how your application handles the video file. As explained in the docs, when an open file is mutated remotely, Mountpoint will never mix data from different version. Instead, it will either return the original content (when it had already been downloaded/cached) or error out. In order to read the content of the mutated object, the application needs to re-open the file.

passaro commented 3 weeks ago

We have just merged a fix for the issue you reported (#933). If you are able to build from the latest main and run your workflow, you should not see this error any longer:

ls: cannot open directory /mnt/applications/classtream/archives: Transport endpoint is not connected

However, as mentioned above, we do expect that reads on a file mutated remotely will still fail and be reported in the logs with entries like this:

00:42:20 tes-sv mount-s3[1370]: [WARN] read{req=102660 ino=32 fh=42 offset=15757312 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely

mura890 commented 2 weeks ago

thank you very much! I am testing again with the merged module, and so far the following error has not occurred.

ls: cannot open directory /mnt/applications/classtream/archives: Transport endpoint is not connected

As expected the following WARN occurs as well.

Jul 10 01:24:29 tes-sv mount-s3[4007]: [WARN] get_object{id=52 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(68157440..195651171) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=41.218344ms request_id=MC9Q5TGX8E4QMNN6 error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:24:29 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] read{req=44145 ino=3 fh=25 offset=68108288 size=131072 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] get_object{id=62 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(68157440..69206016) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=40.720996ms request_id=6V52SEWSB35Z7PCQ error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] read{req=44146 ino=3 fh=25 offset=68239360 size=131072 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] get_object{id=63 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(68157440..69206016) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=35.803874ms request_id=6V53835NJRKHAZQ9 error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] read{req=44159 ino=3 fh=25 offset=68157440 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] get_object{id=64 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(68157440..70254592) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=35.173325ms request_id=6V530CV478TARASE error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] read{req=44160 ino=3 fh=25 offset=68157440 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] get_object{id=65 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(68157440..69206016) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=47.220248ms request_id=6V597FZVPYC8ZQFD error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] read{req=44161 ino=3 fh=25 offset=68210688 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] get_object{id=66 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(68157440..69206016) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=46.624235ms request_id=6V55Q9SANSXH5CCB error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:26:13 tes-sv mount-s3[4007]: [WARN] read{req=44162 ino=3 fh=25 offset=68206592 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
・・・
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] get_object{id=2487 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(72351744..73400320) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=39.546061ms request_id=QX448X6MZ81P9DSY error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] read{req=46623 ino=3 fh=25 offset=72663040 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] get_object{id=2488 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(72351744..73400320) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=36.902871ms request_id=QX4CDWKDB1WN6R6G error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] read{req=46624 ino=3 fh=25 offset=72646656 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] get_object{id=2489 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(72351744..73400320) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=33.574514ms request_id=QX4840MRZ1HK61AS error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] read{req=46625 ino=3 fh=25 offset=72663040 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] get_object{id=2490 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(72351744..73400320) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=27.698542ms request_id=QX4CC12T4T3XBMF5 error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] read{req=46626 ino=3 fh=25 offset=72663040 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] get_object{id=2491 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(72351744..73400320) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=37.290862ms request_id=QX47YW9F3SET97NN error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] read{req=46627 ino=3 fh=25 offset=72663040 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] get_object{id=2492 bucket=test-S3-bucket7 key=archives/30301013/6.mp4 range=Some(72351744..73400320) if_match=Some(ETag("\"707c907d18ed96346626511d92d9b8ef-24\""))}: mountpoint_s3_client::s3_crt_client:  duration=31.200832ms request_id=QX44PDHSQXANVKK8 error=ServiceError(PreconditionFailed)meta request failed
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] mountpoint_s3::prefetch::caching_stream: GetObject body part failed key=archives/30301013/6.mp4 error=ServiceError(PreconditionFailed)
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] read{req=46628 ino=3 fh=25 offset=72806400 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely

Is it possible to resolve this?

passaro commented 2 weeks ago

[...]
Jul 10 01:27:51 tes-sv mount-s3[4007]: [WARN] read{req=46628 ino=3 fh=25 offset=72806400 size=4096 name=6.mp4}: mountpoint_s3::fuse: read failed: object was mutated remotely

Is it possible to resolve this?

What would be your desired behavior in this scenario?

From what I can tell from your logs, the current behavior is expected and intended to ensure consistency. See the first paragraph in the Consistency and concurrency section for more details and potential suggestions.

mura890 commented 2 weeks ago

Checked consistency and concurrency. Since a large amount of logs were output, I thought that the problem was occurring continuously, but I understood that it was as expected. Thank you for your response.

awslabs / mountpoint-s3