search

clyso / chorus

s3 multi provider data lifecycle management
Apache License 2.0
54 stars 6 forks source link

chorus_agent doesn't actually replicate new upstream objects? #40

Open PC-Admin opened 1 month ago

PC-Admin commented 1 month ago

So I have a multi-host setup where chorus_agent is on a different host then my chorus_worker containers.

Went from 50 -> 100 objects in test-bucket3, but the chorus_agent doesn't seem to notice this:

mcollins1@storage-13-09002:~/chorus$ ./tools/chorctl/chorctl --address=storage-13-09004:9670 repl
NAME                                      PROGRESS                 SIZE                  OBJECTS     EVENTS     PAUSED     LAG             AGE
test-user:test-bucket7:main->follower     [##########] 100.0 %     4.8 GiB/4.8 GiB       50/50       0/0        false      14.264635ms     4m
test-user:test-bucket4:main->follower     [##########] 100.0 %     9.7 GiB/9.7 GiB       100/100     0/0        false      45.017207ms     6m
test-user:test-bucket6:main->follower     [##########] 100.0 %     4.9 GiB/4.9 GiB       50/50       0/0        false      26.598635ms     22h44m
test-user:test-bucket5:main->follower     [##########] 100.0 %     4.9 GiB/4.9 GiB       50/50       0/0        false      72.514271ms     23h5m
test-user:test-bucket3:main->follower     [##########] 100.0 %     4.9 GiB/4.9 GiB       50/50       0/0        false      11.607007ms     23h5m
test-user:test-bucket2:main->follower     [##########] 100.0 %     2.0 GiB/2.0 GiB       20/20       0/0        false      11.666847ms     23h5m
test-user:test-bucket:main->follower      [##########] 100.0 %     17.5 GiB/17.5 GiB     105/105     0/0        false      79.487908ms     23h5m

It's logs don't show any errors. (Besides an initial connection error I believe it got past.)

mcollins1@storage-13-09002:~/chorus$ sudo docker service logs chorus_agent
chorus_agent.1.1udvp71t8jrf@storage-13-09002    | {"level":"info","time":"2024-08-01T03:28:44Z","message":"app config: reading default common config"}
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | {"level":"info","time":"2024-08-01T03:28:51Z","message":"app config: reading default common config"}
chorus_agent.1.1udvp71t8jrf@storage-13-09002    | {"level":"info","time":"2024-08-01T03:28:44Z","message":"app config: override with: agent_default_cfg"}
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | {"level":"info","time":"2024-08-01T03:28:51Z","message":"app config: override with: agent_default_cfg"}
chorus_agent.1.1udvp71t8jrf@storage-13-09002    | {"level":"info","time":"2024-08-01T03:28:44Z","message":"app config: override with: config/config.yaml"}
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | {"level":"info","time":"2024-08-01T03:28:51Z","message":"app config: override with: config/config.yaml"}
chorus_agent.1.1udvp71t8jrf@storage-13-09002    | {"level":"info","time":"2024-08-01T03:28:44Z","message":"app config: override with: config/override.yaml"}
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | {"level":"info","time":"2024-08-01T03:28:51Z","message":"app config: override with: config/override.yaml"}
chorus_agent.1.1udvp71t8jrf@storage-13-09002    | 2024-08-01T03:28:44Z INF ../build/service/agent/server.go:49 > app starting... app=agent app_id=cqlg1r57arvs73avg4sg commit="not set" version=development
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/service/agent/server.go:49 > app starting... app=agent app_id=cqlg1ssm0mac73d0lli0 commit="not set" version=development
chorus_agent.1.1udvp71t8jrf@storage-13-09002    | 2024-08-01T03:28:44Z INF ../build/service/agent/server.go:58 > redis app pool stats app=agent app_id=cqlg1r57arvs73avg4sg redis_pool={"Hits":0,"IdleConns":0,"Misses":0,"StaleConns":0,"Timeouts":0,"TotalConns":0} redis_pool_size=640
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/service/agent/server.go:58 > redis app pool stats app=agent app_id=cqlg1ssm0mac73d0lli0 redis_pool={"Hits":0,"IdleConns":0,"Misses":0,"StaleConns":0,"Timeouts":0,"TotalConns":0} redis_pool_size=640
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/service/agent/server.go:69 > app redis connected app=agent app_id=cqlg1ssm0mac73d0lli0
chorus_agent.1.1udvp71t8jrf@storage-13-09002    | {"level":"error","error":"LOADING Redis is loading the dataset in memory: unable to reach app redis","time":"2024-08-01T03:28:44Z","message":"critical error. Shutdown application"}
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/service/agent/server.go:73 > redis conf pool stats app=agent app_id=cqlg1ssm0mac73d0lli0 redis_pool={"Hits":0,"IdleConns":1,"Misses":1,"StaleConns":0,"Timeouts":0,"TotalConns":1} redis_pool_size=640
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/service/agent/server.go:82 > redis queue pool stats app=agent app_id=cqlg1ssm0mac73d0lli0 redis_pool_size=0
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/service/agent/server.go:104 > agent created app=agent app_id=cqlg1ssm0mac73d0lli0
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/service/agent/server.go:112 > metrics enabled app=agent app_id=cqlg1ssm0mac73d0lli0
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/pkg/util/serve.go:60 > server: start serving 3 workers app=agent app_id=cqlg1ssm0mac73d0lli0
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/pkg/util/serve.go:97 > server: start serving app=agent app_id=cqlg1ssm0mac73d0lli0
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/pkg/util/serve.go:67 > server: starting worker "agent_metrics" app=agent app_id=cqlg1ssm0mac73d0lli0
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/pkg/util/serve.go:67 > server: starting worker "agent_request_reply" app=agent app_id=cqlg1ssm0mac73d0lli0
chorus_agent.1.wqt9uu15a74r@storage-13-09002    | 2024-08-01T03:28:51Z INF ../build/pkg/util/serve.go:67 > server: starting worker "agent_http" app=agent app_id=cqlg1ssm0mac73d0lli0

Here's my worker_conf.yaml:

api:
  enabled: true # set true to enable management api
  grpcPort: 9670
  httpPort: 9671
  secure: false
log:
  json: false # false for dev console logger, true - json log for prod to export to Grafana&Loki.
  level: info
metrics:
  enabled: true
  port: 9090
trace:
  enabled: false
  endpoint: # url to Jaeger or other open trace provider
redis:
  address: "storage-13-09002:6379"
features:
  tagging: false # sync object/bucket tags
  acl: false # sync object/bucket ACLs
  lifecycle: false # sync bucket Lifecycle
  policy: false # sync bucket Policies

and my agent_conf.yaml:

port: 9673 # agent port to listen incoming notifications
url: "http://storage-13-09002:9673" # REQUIRED: url to be used by s3 storage to send notifications. The URL should be reachable for s3 storage.
fromStorage: "main" # REQUIRED: notifications source storage name from Chorus config

log:
  json: false # false for dev console logger, true - json log for prod to export to Grafana&Loki.
  level: info
metrics:
  enabled: true
  port: 9090
trace:
  enabled: false
  endpoint: # url to Jaeger or other open trace provider
redis:
  address: "redis:6379"
features:
  tagging: true # sync object/bucket tags
  acl: true # sync object/bucket ACLs
  lifecycle: false # sync bucket Lifecycle
  policy: false # sync bucket Policies

Am I holding this thing wrong? "url to be used by s3 storage to send notifications." I'm not sure how this 'url' field is supposed to be used tbh...

My apologies if this is a noob question.

arttor commented 1 month ago

url will be used by bucket notifications of your source s3 storage. This means that your source s3 storage should support bucket notification and this url should be reachable to your source s3 storage. Please check agent readme https://github.com/clyso/chorus/tree/main/service/agent