search

anyproto / any-sync-filenode

Implementation of file node from any-sync protocol
https://anytype.io
MIT License
34 stars 13 forks source link

Reduce s3 PUT/GET requests #118

Open hellodword opened 1 week ago

hellodword commented 1 week ago

Have you read a contributing guide?

Clear and concise description of the problem

I'm testing anytype-ts with self-hosted nodes, and have pasted this picture, which is 2734 bytes, into a note.

See what the mc admin trace tells:

minio-trace-1  | 2024-06-28T08:02:37.720 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeibiixdxackefhwkgb6nlbvvkdxw4mcysuhazvvslhdm4mx5ftflxm 172.26.0.10      1.361ms      ⇣  0s         ↑ 155 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.724 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeihylc6lq2ksrpqa7fidd4zbhsaqb7c2oezxcfpyzcrpbh3wmtqn5e 172.26.0.10      1.492ms      ⇣  0s         ↑ 365 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.730 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeienamcdkf45cbv57oeq2jn4md7q6h2zwjazihn4vc5jr74ygfr4ce 172.26.0.10      1.84ms       ⇣  0s         ↑ 210 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.735 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeia4t2ljvfotm2i7yhscp2i5bxm2gka3drlrn4tfsi7tupj5nmi2re 172.26.0.10      1.507ms      ⇣  0s         ↑ 283 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.740 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeigpqmwuepdoayscsudzbv6sg5bvnkuhxc43gwsbfqm4p74waau26e 172.26.0.10      1.421ms      ⇣  0s         ↑ 432 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.746 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeiabb6p4qcdloxtgkh3ixwwkv6n5w3prm4a24sb4j465j7jgieq2du 172.26.0.10      1.704ms      ⇣  0s         ↑ 210 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.751 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeidezvmvgozf6wtuuwrwfntbjw4mie3h347woteqeuosqr47tojntu 172.26.0.10      1.5ms        ⇣  0s         ↑ 441 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.755 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeicewxs7mqeqfr3xsuyj5rrxybjsr5h2evy2eo5rhfvyhyggh7rnae 172.26.0.10      1.405ms      ⇣  0s         ↑ 477 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.759 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeifql3nhojwypgu4ytwfpll7kle54yhzedeworqhhagce4hnnklawq 172.26.0.10      1.296ms      ⇣  0s         ↑ 210 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.764 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeiebfff2uuhml3iq4rmy2pnvnq5njrwmws35oswfu7rlmnjstoyf64 172.26.0.10      1.488ms      ⇣  0s         ↑ 441 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.770 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeiazdfu72kemfxjppzvmiaavc24jlf3hghygskw6t2dgslsvze2vvq 172.26.0.10      1.333ms      ⇣  0s         ↑ 477 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.775 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeigrtxgyahwcj5b664sji3soejtbh6fuulhmtkekaofpgm4ejmgkwq 172.26.0.10      1.282ms      ⇣  0s         ↑ 210 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.779 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeibqme42m7ofv3tgujxw6camwt44y2mbfglqxlrq53652ghrmw3bxe 172.26.0.10      1.768ms      ⇣  0s         ↑ 441 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.784 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeiefuw4kcah4qlrvhnmtepjz3753biqpmjeavsjwzzna5e73yt6xoq 172.26.0.10      1.401ms      ⇣  0s         ↑ 477 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.788 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeicgf55excipcegyd5qfwul3vvpfogaabki2vxb46othmsu5a4xrn4 172.26.0.10      1.56ms       ⇣  0s         ↑ 210 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.793 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeibrmhldqjifmkxqsfltev2wxr5pwvnm3y6gorvuu7ctppk7pcalli 172.26.0.10      1.356ms      ⇣  0s         ↑ 441 B ↓ 0 B
minio-trace-1  | 2024-06-28T08:02:37.797 [200 OK] s3.PutObject minio:9000/minio-bucket/bafybeifknna34y3ov3puy7e5essyalgyf2pbevjxxev6qr5s343gvfsf34 172.26.0.10      1.455ms      ⇣  0s         ↑ 477 B ↓ 0 B

I deleted the local directories and re-entered anytype-ts with the key, and opened that note:

minio-trace-1  | 2024-06-28T08:06:22.149 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeibiixdxackefhwkgb6nlbvvkdxw4mcysuhazvvslhdm4mx5ftflxm 172.26.0.10      560µs       ⇣  522.687µs  ↑ 93 B ↓ 50 B
minio-trace-1  | 2024-06-28T08:06:22.152 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeihylc6lq2ksrpqa7fidd4zbhsaqb7c2oezxcfpyzcrpbh3wmtqn5e 172.26.0.10      370µs       ⇣  349.228µs  ↑ 93 B ↓ 260 B
minio-trace-1  | 2024-06-28T08:06:22.156 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeienamcdkf45cbv57oeq2jn4md7q6h2zwjazihn4vc5jr74ygfr4ce 172.26.0.10      503µs       ⇣  467.057µs  ↑ 93 B ↓ 105 B
minio-trace-1  | 2024-06-28T08:06:22.159 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeigpqmwuepdoayscsudzbv6sg5bvnkuhxc43gwsbfqm4p74waau26e 172.26.0.10      368µs       ⇣  348.95µs  ↑ 93 B ↓ 327 B
minio-trace-1  | 2024-06-28T08:06:22.163 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeiabb6p4qcdloxtgkh3ixwwkv6n5w3prm4a24sb4j465j7jgieq2du 172.26.0.10      331µs       ⇣  314µs     ↑ 93 B ↓ 105 B
minio-trace-1  | 2024-06-28T08:06:22.166 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeicewxs7mqeqfr3xsuyj5rrxybjsr5h2evy2eo5rhfvyhyggh7rnae 172.26.0.10      336µs       ⇣  316.772µs  ↑ 93 B ↓ 372 B
minio-trace-1  | 2024-06-28T08:06:22.169 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeifql3nhojwypgu4ytwfpll7kle54yhzedeworqhhagce4hnnklawq 172.26.0.10      296µs       ⇣  280.156µs  ↑ 93 B ↓ 105 B
minio-trace-1  | 2024-06-28T08:06:22.171 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeiazdfu72kemfxjppzvmiaavc24jlf3hghygskw6t2dgslsvze2vvq 172.26.0.10      309µs       ⇣  292.842µs  ↑ 93 B ↓ 372 B
minio-trace-1  | 2024-06-28T08:06:22.174 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeigrtxgyahwcj5b664sji3soejtbh6fuulhmtkekaofpgm4ejmgkwq 172.26.0.10      365µs       ⇣  347.841µs  ↑ 93 B ↓ 105 B
minio-trace-1  | 2024-06-28T08:06:22.177 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeiefuw4kcah4qlrvhnmtepjz3753biqpmjeavsjwzzna5e73yt6xoq 172.26.0.10      342µs       ⇣  323.408µs  ↑ 93 B ↓ 372 B
minio-trace-1  | 2024-06-28T08:06:22.180 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeicgf55excipcegyd5qfwul3vvpfogaabki2vxb46othmsu5a4xrn4 172.26.0.10      347µs       ⇣  330.418µs  ↑ 93 B ↓ 105 B
minio-trace-1  | 2024-06-28T08:06:22.183 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeifknna34y3ov3puy7e5essyalgyf2pbevjxxev6qr5s343gvfsf34 172.26.0.10      311µs       ⇣  295.906µs  ↑ 93 B ↓ 372 B
minio-trace-1  | 2024-06-28T08:06:22.185 [200 OK] s3.GetObject minio:9000/minio-bucket/bafybeibrmhldqjifmkxqsfltev2wxr5pwvnm3y6gorvuu7ctppk7pcalli 172.26.0.10      325µs       ⇣  309.074µs  ↑ 93 B ↓ 336 B

I think it's kind of expensive for s3 usage.

Suggested solution

I'm not sure, but I guess something like BlockSize might help.

I believe restic and juicefs could provide more technical details about this.

Alternative

No response

Additional context

No response