CloudFlare R2 Rclone

[ProfessorManhattan]

💡 Feature/Idea

The Rclone feature needs some love. Right now, services / mounts are set up in /mnt/Mount and ~/.local/mnt/s3 that link CloudFlare R2 storage buckets to the system via the ~/.local/bin/rclone-mount script.

For some reason, right now you can only create files on the mount but the files that are already in the S3 bucket are not showing up. Files stored in the S3 bucket should show up.

Second, the dot_local/bin/executable_rclone-mount includes a bunch of rclone executable flags. These flags should be moved to the ~/.config/rclone/rclone.conf configuration file or environment variables.

Third, the configuration uses --rc and --rc-user/pass right now but ideally we should use a more secure form of authentication such as certificates / PEM files --- please do research into this and automate the set up of a more secure method of authentication.

Also, I'm getting the following error on macOS when the service tries to mount via system service:

2023/07/31 02:07:55 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/07/31 02:08:05 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/07/31 02:08:05 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/07/31 02:08:15 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/07/31 02:08:25 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/07/31 02:08:25 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/07/31 02:08:35 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/07/31 02:08:36 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/07/31 02:08:46 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/07/31 02:08:56 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/07/31 02:08:56 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/07/31 02:09:06 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/07/31 02:09:16 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/07/31 02:09:16 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use

This is found in the /var/log/rclone/Public.log for the public mount.

:thumbsup: Can you contribute?

No response

[enggnr]

@ProfessorManhattan, here are my findings:

For some reason, right now you can only create files on the mount but the files that are already in the S3 bucket are not showing up.

Here are some discussions that pertain to this observation. Not all are mounting remote S3 or use MacOS, but I believe some of these pointers can be used to troubleshoot. These are all related to the cache settings, and in one case fuse settings (which may not apply to MacOS). I also wanted to know if the S3 has a large number of files and whether there are some large files. Are the files created locally successfully uploaded to S3?

I'm getting the following error on macOS when the service tries to mount via system service

This is happening since the --rc-addr flag is not provided and all the services are using the default port 5572, resulting in conflict. Could you please check if changing the port and restarting the services help fix this problem?

I'll continue looking into other items.