Support config and limit

[wzshiming]

storage:
  driver: oss
  parameters: {}
  redirectLinks: ""
  linkExpires: 1h

mirrors:
- domain: "registry-1.docker.io"
  alias:
  - "docker.io"
  auth:
    username: xxx
    password: xxx

  limitRules:
  - matchImage: *
    request: 100r/m
    bandwidth: 10G/h
  - matchIP: 0.0.0.0/0
    bandwidth: 100m/m

limitRules:
- matchIP: 0.0.0.0/0
  bandwidth: 500m/m

[gebangfeng]

这个是配置文件吗，我应该如何使用呢

[wzshiming]

计划支持的限流的能力, 在纠结是否有这么细粒度的必要

[gebangfeng]

代理服务器拉取镜像过多会触发这个： Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 匿名用户6个小时有100个镜像拉取额度，登录后可以增加到200个，但貌似没法登录。

[wzshiming]

是的 你需要给 crproxy 配置一个 docker 账号

[wzshiming]

参考 -u 参数 user:pass@docker.io

[gebangfeng]

如何配置呢

发自我的iPhone

------------------ Original ------------------ From: Shiming Zhang @.> Date: Tue,Jun 11,2024 6:37 PM To: wzshiming/crproxy @.> Cc: 凡之 @.>, Comment @.> Subject: Re: [wzshiming/crproxy] Support config and limit (Issue #5)

是的 你需要给 crproxy 配置一个 docker 账号

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[gebangfeng]

感谢，了解了

发自我的iPhone

------------------ Original ------------------ From: Shiming Zhang @.> Date: Tue,Jun 11,2024 6:38 PM To: wzshiming/crproxy @.> Cc: 凡之 @.>, Comment @.> Subject: Re: [wzshiming/crproxy] Support config and limit (Issue #5)

参考 -u 参数 @.***

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[gebangfeng]

存储支持本地磁盘吗

发自我的iPhone

------------------ Original ------------------ From: myfanzhi @.> Date: Tue,Jun 11,2024 6:38 PM To: reply+ANL3BHD7TQZNEYYHIHTSTTGEOQFBFEVBNHHIXG64AE @.>, wzshiming/crproxy @.> Cc: 啊！我不会一个人 @.>, Comment @.***> Subject: Re: [wzshiming/crproxy] Support config and limit (Issue #5)

感谢，了解了

发自我的iPhone

------------------ Original ------------------ From: Shiming Zhang @.> Date: Tue,Jun 11,2024 6:38 PM To: wzshiming/crproxy @.> Cc: 凡之 @.>, Comment @.> Subject: Re: [wzshiming/crproxy] Support config and limit (Issue #5)

参考 -u 参数 @.***

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[wzshiming]

如果你要本地存储, 建议用 https://github.com/distribution/distribution

[wzshiming]

这个项目最初就是配合 https://github.com/distribution/distribution 一起, 作为一个多源聚代理合用的

[gebangfeng]

好的，我试试

发自我的iPhone

------------------ Original ------------------ From: Shiming Zhang @.> Date: Tue,Jun 11,2024 6:42 PM To: wzshiming/crproxy @.> Cc: 凡之 @.>, Comment @.> Subject: Re: [wzshiming/crproxy] Support config and limit (Issue #5)

这个项目最初就是配合 https://github.com/distribution/distribution 一起, 作为一个多源聚代理合用的

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[gebangfeng]

./crproxy_linux_amd64 -a :8888 --total-blobs-speed-limit "20MB" --blobs-speed-limit "1MB"

这两个限制参数有详细说明吗，假如我服务器带宽30M，这两个参数我应该配置多少。

另外

cat registry/config.yml 
version: 0.1
log:
  accesslog:
    #disabled: true
    disabled: false
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: "/var/lib/registry"
  maintenance:
    uploadpurging:
      enabled: false
    readonly:
      enabled: true
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]

health:
  storagedriver:
    enabled: false

validation:
  disabled: true

proxy:
  remoteurl: http://crproxy:8080

如果crproxy在registry后面貌似没法使用 ip的限制了，因为它获取不到客户端真实地址：

./crproxy_linux_amd64 --ips-speed-limit "10"

[gebangfeng]

./crproxy_linux_amd64 -a :8888 --total-blobs-speed-limit "20MB" --blobs-speed-limit "1MB"

这两个限制参数有详细说明吗，假如我服务器带宽30M，这两个参数我应该配置多少。

另外

cat registry/config.yml 
version: 0.1
log:
  accesslog:
    #disabled: true
    disabled: false
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: "/var/lib/registry"
  maintenance:
    uploadpurging:
      enabled: false
    readonly:
      enabled: true
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]

health:
  storagedriver:
    enabled: false

validation:
  disabled: true

proxy:
  remoteurl: http://crproxy:8080

如果crproxy在registry后面貌似没法使用 ip的限制了，因为它获取不到客户端真实地址：

./crproxy_linux_amd64 --ips-speed-limit "10"

[wzshiming]

是的, 限流不适合在 registry 后面使用.

[wzshiming]

我现在部署的形式在服务器上已经把 registry 这一层拿掉了

[gebangfeng]

这种可以，但不支持本地存储。

[wzshiming]

你是想在 服务器 上本地文件 存 blobs?

[wzshiming]

服务器磁盘够??

[gebangfeng]

如果registry 中已经有的镜像，就不会找crproxy了吧

[gebangfeng]

腾讯oss可以挂载磁盘上

[wzshiming]

那服务器带宽呢?

[wzshiming]

这种场景, 我建议你在前面的 nginx 配置限流

[wzshiming]

crproxy 不做任何限制

[gebangfeng]

oss和服务器之间走内网，上下都免费。只收存储费用。

[wzshiming]

oss和服务器之间走内网，上下都免费。只收存储费用。

这个我知道, 我是只服务的外网下行带宽, 这部分一般超过 5M 就很贵

[wzshiming]

5M基本上, 拉一两个大镜像就能导致你服务器带宽占满

[gebangfeng]

请问： 我现在的架构是 nginx->官方registry(数据保存磁盘)-->crproxy, 磁盘中的数据量越来越大了，如何分析 拉取量低的，并将它们删除呢。

[wzshiming]

写个脚本分析 nginx 日志统计使用量然后删除, 或者等 7 天会自己 GC

[gebangfeng]

gc 会把所有镜像删除吗

发自我的iPhone

------------------ Original ------------------ From: Shiming Zhang @.> Date: Mon,Jun 17,2024 6:18 PM To: wzshiming/crproxy @.> Cc: 凡之 @.>, Comment @.> Subject: Re: [wzshiming/crproxy] Support config and limit (Issue #5)

写个脚本分析 nginx 日志统计使用量然后删除, 或者等 7 天会自己 GC

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[wzshiming]

是的 七天 不用就删除

[wzshiming]

这种机制, 适合在内网集群做缓存, 不太适合提供公开服务