Cloudflare R2 support upgrade

chancefcc commented 4 months ago

Adding CF R2 Support

Current tiktag cli for hosting an asset is like,

tiktag image.png

By default, current version uses the credentials under the minio section of ~/.tiktag/config.yaml to media files onto the bucket of the specified object storage services, MinIO, as defined on this section.

After this upgrade, a user can run either

tiktag minio image.png

or

tiktag  r2 image.png

to host media files on the pre-defined object storage service.

Improve `~/.tiktag/config.yaml`

Rename these params to match the idioms used in their corresponding services/apps,

Old

minio:
  endpoint: "s3.myminio.com"
  id: "my_bucket_api_id"
  secret: "my_bucket_secret"
  useSSL: true
  bucketName: "my_bucket"

New

minio:
  endpoint: "s3.myminio.com"
  accessKey: "my_bucket_api_id"
  secretKey: "my_bucket_secret"
  useSSL: true
  bucketName: "my_bucket"

Naming conventions

All params naming shall follow the corresponding object storage's conventions, EXACTLY as they are used.

ZoomQuiet commented 4 months ago

根据找到的工程,快速修改已经部分可用, 比如, 这就是上传到空间的图片:

https://r2.101.so/zshot2024-03-1122.jpg

在俺的 m1macOS 中的使用过程, 具体参数我们私聊... 还有些问题, 在和 ant 讨论...

𝄢 neofetch
                    'c.          zoomq@ZQ160626rMBP
                 ,xNMM.          ------------------
               .OMMMMo           OS: macOS Big Sur 10.16 23D60 arm64
               OMMM0,            Host: MacBookPro18,4
     .;loddo:' loolloddol;.      Kernel: 23.3.0
   cKMMMMMMMMMMNWMMMMMMMMMM0:    Uptime: 30 days, 2 hours, 38 mins
 .KMMMMMMMMMMMMMMMMMMMMMMMWd.    Packages: 422 (brew)
 XMMMMMMMMMMMMMMMMMMMMMMMX.      Shell: bash 5.2.21
;MMMMMMMMMMMMMMMMMMMMMMMM:       Resolution: 2048x1280, 3200x1800
:MMMMMMMMMMMMMMMMMMMMMMMM:       DE: Aqua
.MMMMMMMMMMMMMMMMMMMMMMMMX.      WM: Spectacle
 kMMMMMMMMMMMMMMMMMMMMMMMMWd.    Terminal: iTerm2
 .XMMMMMMMMMMMMMMMMMMMMMMMMMMk   Terminal Font: VictorMonoNerdFontCompleteM-Regular 18 (normal) / SarasaMonoSCNerd-regular 18 (non-ascii)
  .XMMMMMMMMMMMMMMMMMMMMMMMMK.   CPU: Apple M1 Max
    kMMMMMMMMMMMMMMMMMMMMMMd     GPU: Apple M1 Max
     ;KMMMMMMMWXXWMMMMMMMk.      Memory: 11371MiB / 65536MiB
       .cooc,.    .,coo:.

𝄢 ./r2_m1mac configure
Profile [default]:
Account ID: ea...
Access Key ID: 9e6...
Secret Access Key: 374ee529c6e5a0c9...

༼~/Downloads/log ༽took 13s
𝄢 ./r2
r2_m1mac    r2_win.exe

༼~/Downloads/log ༽
λ ./r2_m1mac ls
2024-03-10 12:58:45 tiktag

༼~/Downloads/log ༽
𝄢 ./r2_m1mac ls tiktag
2024-03-10 22:53:50  198 KB  zshot 2024-03-04 09.25.09.jpg
2024-03-12 02:58:54   30 KB  zshot2024-03-1122-2.jpg
2024-03-12 02:52:39   30 KB  zshot2024-03-1122.jpg

༼~/Downloads/log ༽took 14s
𝄢 ./r2_m1mac cp r2_win.exe r2://tiktag/r2_win_v0.exe

ZoomQuiet commented 3 months ago

env


$ neofetch
            .-/+oossssoo+/-.               ubuntu@hia.chatek.co 
        `:+ssssssssssssssssss+:`           -------------------- 
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 22.04.3 LTS x86_64 
    .ossssssssssssssssssdMMMNysssso.       Host: OpenStack Nova 13.2.1-20230810114642_14f3 
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 5.15.0-97-generic 
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 4 days, 11 hours, 23 mins 
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 749 (dpkg) 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.1.16 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 1024x768 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Terminal: node 
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: Intel Xeon Gold 6161 (2) @ 2.200GHz 
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   GPU: 00:02.0 Cirrus Logic GD 5446 
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Memory: 970MiB / 3911MiB 
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+                             
   /ssssssssssshdmNNNNmyNMMMMhssssss/                              
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN='/opt/go/bin'
GOCACHE='/home/ubuntu/.cache/go-build'
GOENV='/home/ubuntu/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/opt/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/opt/go'
GOPRIVATE=''
GOPROXY='https://goproxy.cn'
GOROOT='/opt/go/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/go/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.1'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/opt/go/src/gitlab.com/tiktag/cli/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1836997668=/tmp/go-build -gno-record-gcc-switches'

现象

无法使用 erdos-one/r2 at v0.1.0-alpha 构建自己的 r2 上传器?

总是触发相同问题:

operation error S3: ListObjectsV2, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name.

过程

根据模块代码的理解, 快速构建一个 r2 子指令集, 无论使用自己的 viper 解析 .toml 秋粮, 还是使用原有内置 configure 指令进行配置, 都无法完成正确的基本 r2 空间查询,

SEE:

S3 API compatibility · Cloudflare R2 docs
基于 S3 SDK 来操作 Cloudflare 的 r2 对象存储空间;

尝试

原版正常

在相同主机, 相同 golang 配置中, 按照官方文档可以编译运行并没有出错:

ubuntu@hia:/opt/go/src/github.com/erdos-one/r2
$ go run . ls
2024-03-10 12:58:45 tiktag

$ go run . ls tiktag
2024-03-10 22:53:50  198 KB  zshot 2024-03-04 09.25.09.jpg

...

引用为代码

cmd/ls.go · main · Duouᗡ / tiktag / cli · GitLab

是根据理解从原版仓库复制迁移过来的指令,只进行了以下修订:

使用相对路径 "cli/pkg" 引用对应支持模块
修订了编译时错误: pkg/bucket.go:74:21: cannot use object.Size (variable of type *int64) as int64 value in argument to fileSizeFmt
将 fs := fileSizeFmt(object.Size) 修订为 fs := fileSizeFmt(*object.Size)

试运行时出错:


$ go run . ls
2024/03/11 14:46:38 operation error S3: ListBuckets, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name.
exit status 1

引用为模块

cmd/ll.go · main · Duouᗡ / tiktag / cli · GitLab 是参考官方: USAGE.md

将原版仓库引用为模块来尝试: 对应文档为: pkg package - github.com/erdos-one/r2/pkg - Go Packages

但是, 引用为:

r2 "github.com/erdos-one/r2/pkg"

运行时触发相同的问题:

$ go run . ll
# github.com/erdos-one/r2/pkg
/opt/go/pkg/mod/github.com/erdos-one/r2@v0.1.0-alpha/pkg/bucket.go:74:21: cannot use object.Size (variable of type *int64) as int64 value in argument to fileSizeFmt

所以, 切换为复制过来的目录引用:

r2 "cli/pkg"

然后触发相同问题:

$ go run . ll
2024/03/11 14:49:47 operation error S3: ListObjectsV2, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name.
exit status 1

思考

问题关键在:

为什么直接使用原版代码树时, 不触发 cannot use object.Size (variable of type *int64) 这种明显的数据类型问题?
以及, 是否这一问题, 引发了 endpoint 的网址拼接问题?

chancefcc commented 2 months ago

很细致的技术分析，我们暂且打住这个针对 R2 的新增和升级。接下来因为要升级 tiktag 品牌+功能，迁移到 tiktag.app 定义的品牌、功能和使用场景。5 月初可以出 60% 完成度的官网，user guide, developer guide，咱们用测试用例和文档驱动编程。

ZoomQuiet commented 2 months ago

got, r2 是个基本功能, 以及 cloudflare 相关免费资源的综合使用, 都准备先开发原型, 以便可以通过这个核心指令工具集来融合到具体业务功能中来.

chancefcc commented 2 months ago

我这样理解你上面的反馈：在支持 MinIO 的基础上新增对 R2 object storage 的支持有不可跨越的技术困难，如果超过预期要耗费的努力，就不值得继续了。

tiktagus / tiktag