search

yandex-cloud / k8s-csi-s3

GeeseFS-based CSI for mounting S3 buckets as PersistentVolumes
Other
494 stars 81 forks source link

s3fs and rclone support #16

Open nik123 opened 2 years ago

nik123 commented 2 years ago

In official documentation it is mentioned that that s3fs and rclone are supported in addition to geesefs. However it seems not to be the case. I'm not an expert in CSI and drivers implementation but it seems that CSI functionality is provided by image defined in cmd/s3driver/Dockerfile. This image only contains installation for geesefs.

Solution: either to add support for rclone, geesefs to CSI, or remove mentions of rclone and geesefs from the docs.

vitalif commented 2 years ago

Hi. Why did you need two issues for this?) I don't recommend using s3fs and rclone, s3fs is slow, rclone is even worse, sometimes it just hangs. So I'll probably remove them from README. Support is still there in the code by the way, so you can try to use it if you build your own docker image with s3fs and/or rclone, you just have to change mounter and options.

nik123 commented 2 years ago

Hi. Why did you need two issues for this?)

I had unsuccessfully tried to run k8s-csi-s3 with s3fs and rclone. So I needed examples to make it work. However when I looked into the code it seemed that s3fs/rclone are not supported at all. The lack of support seemed as related issue but not the same one. Therefore I've created 2 separate issues. Anyway, fill free to close the first one.

I don't recommend using s3fs and rclone, s3fs is slow, rclone is even worse, sometimes it just hangs.

I had the 'hanging' with geesefs. I think there are memory leaks in geesefs but I'm not sure. So I wanted to try out alternatives

vitalif commented 2 years ago

Ok, I changed Dockerfile to include both s3fs and rclone, they should also work in 0.31.3 now. What problems did you have with geesefs, by the way? Can you file an issue?

nik123 commented 2 years ago

What problems did you have with geesefs, by the way? Can you file an issue?

@vitalif I use geesefs to mount S3 buckets inside managed kubernetes cluster in Yandex Cloud. Sometimes when I do massive I/O operations my pods just hang until I reboot them. I presumed there may be problems with geesefs implementation and I tried other alternatives. Hence, this issue about s3fs and rclone support.

Anyway, I tried s3fs and rclone and I get the same problems: sometimes pods just hang during I/O operations. Probably I just use s3 in unintended way (basically as a usual file system).

vitalif commented 2 years ago

Is there a way to reproduce it? I can try to fix it if it's reproducible. Which massive operations exactly do you perform before hangs?

kyleli666 commented 1 year ago

Hi @vitalif , s3fs works well in 0.31.3, but cannot work in 0.35.5, it report fuseMount command error. My k8s is on EKS with bottle rocket os, and I cannot use geesefs for now, because somehow errors report that SElinux rejcted it. Could you please help? thanks.

pschichtel commented 1 year ago

I can confirm that both s3fs and rclone don't work, apparently the mount command fails. Sadly geesefs doesn't support chmod.

zunnurainbadar commented 1 year ago

@vitalif, I am also trying to change the last modified date of a folder inside the bucket but It is not working in geesefs, and in s3fs and rclone mount commands are not working.

szelga commented 9 months ago

@vitalif do you know, if there's an option for geesefs to ignore chmod/chown errors, if it's not supported?

vitalif commented 6 months ago

It ignores them, it only returns "operation not supported" on the root directory of mountpoint (i.e. mountpoint itself)