search

mikedilger / chorus

Chorus is a nostr relay
Other
40 stars 7 forks source link

Is scraping really slow? #20

Closed fiatjaf closed 2 months ago

fiatjaf commented 3 months ago

I'm running wss://chorus.fiatjaf.com/ and trying to use it to publish some articles for https://wikifreedia.xyz/, but that client does ["REQ", "_", {"kinds": [30818]}], which is what I really want, so I'm now inclined to allow scraping.

However the docs say somewhere that it is inefficient to allow scraping, so maybe I shouldn't?

fiatjaf commented 3 months ago

Hmm, I see there is no kind+created_at index, so this query specifically will be very bad.

mikedilger commented 3 months ago

In this case it is scanning every event. When I say 'inefficient' I mean in terms of pruning the events it needs to scan. It can probably scan every event in ridiculous-fast time because there are no memory copies going on and probably no paging either. Until your database gets quite full, allowing scraping is probably just fine.