bluesky-social / indigo

Go source code for Bluesky's atproto services.
https://atproto.com
Apache License 2.0
646 stars 97 forks source link

docs(bgs): add strong anti-recomendation for running ext4 + document dir_index #747

Open nepeat opened 1 week ago

nepeat commented 1 week ago

I've attempted to create a self hosted BGS and did not stand by the recommendation to use XFS instead of ext4. As a result of this, I likely have hit a hash collision due to the dir_index hashing feature of ext3 + ext4.

image

src: https://blog.merovius.de/posts/2013-10-20-ext4-mysterious-no-space-left-on/#do-you-have-dir_index-enabled

Suggestions

  1. Update the filesystem recommendation to recommend against ext4 explicitly for folks trying to self host a relay.
  2. Call out tune2fs -O "^dir_index" /dev/device as an option to disable this feature due to the millions of files that relay creates.
    # tune2fs -O "^dir_index"  /dev/device
    tune2fs 1.47.0 (5-Feb-2023)
    Disabling directory index on filesystem with checksums could take some time.
    Proceed anyway (or wait 5 seconds to proceed) ? (y,N) y
    #

    I'm not really a good document writer but the general ideas I hope to push in the relay README are these two.

bnewbold commented 1 week ago

Wow! did not know of that particular failure mode, thanks for reporting.

How is relay self-hosting going otherwise? We have pushed some small performance optimizations in the last week or so to help keep up with network traffic growth. The current relay codebase should continue to work (especially if the number of subscribers is smaller), but we are actively considering ways to refactor and scale the service, as well as any protocol tweaks that might make it easier/cheaper to run relays.