search

ethersphere / bee

Bee is a Swarm client implemented in Go. It’s the basic building block for the Swarm network: a private; decentralized; and self-sustaining network for permissionless publishing and access to your (application) data.
https://www.ethswarm.org
BSD 3-Clause "New" or "Revised" License
1.46k stars 337 forks source link

Running out of disk space #1258

Closed alsakhaev closed 1 year ago

alsakhaev commented 3 years ago

Summary

We have a bunch of troubles running our bee node serving the swarm downloader (out hackathon project). 1) We a running a bee node under https://swarm.dapplets.org 2) the node takes all available space on HDD and obviously starts rejecting files we are uploading. Waiting for swarm hash either fails immediately or takes too long time. We have set a db-capacity: 2621440 chunks (aprox. 10gb) + 5GB freespace, but goes fully consumed.

Steps to reproduce

1) Created VPS server in Hetzner with following hardware (CX11, 1 VCPU, 2 GB RAM, 20 GB) with Ubuntu 20.04.2 LTS 2) Installed Bee via wget https://github.com/ethersphere/bee/releases/download/v0.5.0/bee_0.5.0_amd64.deb sudo dpkg -i bee_0.5.0_amd64.deb 3) Configured like in the config bellow 4) Installed nginx web-server and configured reverse proxy from https://swarm.dapplets.org to http://localhost:1633 with SSL of let's encrypt 5) Upload files to the node via POST https://swarm.dapplets.org/files/ 6) After a while disk space runs out

Expected behavior

I expect to see 5gb freespace :)

Actual behavior

1) Disk space runs out 2) in the log a lot of errors about it 3) cannot upload a file, node responses HTTP 500 internal server error

Config /etc/bee/bee.yaml

Uncommented lines from config file:

api-addr: 127.0.0.1:1633
clef-signer-endpoint: /var/lib/bee-clef/clef.ipc
config: /etc/bee/bee.yaml
data-dir: /var/lib/bee
db-capacity: 2621440
gateway-mode: true
password-file: /var/lib/bee/password
swap-enable: true
swap-endpoint: https://rpc.slock.it/goerli
acud commented 2 years ago

The limit on amount of chunks stored is not a hard limit for reasons which would be difficult to explain here. Let's start with evaluating the size of the sharky sub directory over time. Please update us here on your findings. The leveldb part (the localstore root) also does not get GCd so often and when it does, it is done on its own conditions (not a full gc). Let's start from sharky dir stats then we see how to progress.

jpritikin commented 2 years ago

Ah, well, maybe there is nothing wrong? Is this the output you need?

root@glow:/opt/bee/localstore# du -d 1 -h
17G ./sharky
18G .

I using btrfs for /opt. Perhaps the output of df -h /opt is not accurate? I can't see where 26GiB is used given that bee is only using 20GiB.

jpritikin commented 2 years ago

Looks like I was confused by df output. df shows 35G used, but disk usage by the sharky directory is unchanged vs my previous comment 4 days ago. Time to close this issue?

acud commented 2 years ago

Let's leave it open for now and continue tracking the issue

jpritikin commented 2 years ago

Should I upgrade to v1.6 or is there still value in running the custom build?

acud commented 2 years ago

yes you can, as we ported the fix into the release :+1:

jpritikin commented 2 years ago

Hm, something is still broken? I upgraded to 1.6.2 today and noticed that the disk was filling up again,

root@glow:/opt/bee/localstore# du -d 1 -h
36G ./sharky
37G .