[Issue] Problem loading big amount of domains

[pdelteil]

I'm trying to add a big program, around 5.5 million domains, the inscope and outscope are big also.

I don't know why it's failing, this is what I tried:

cat domains-big.txt | bbrf domain add - -s subfinder --show-new

Doesn't add any domain nor output any error.

If I do this sed -n '50001 ,100000p' domains-big.txt| bbrf domain add - -s subfinder --show-new

I get this error "[ERROR] local variable 'success' referenced before assignment"

The same error doing:

awk 'NR > 50000 && NR <= 100000' domains-big.txt | bbrf domain add - -s subfinder --show-new

Any idea why this might happen?

[honoki]

There was a bug in the --show-new flag that I have silently fixed yesterday in v1.1.3

However, that is probably unrelated to the large input failing. I'll run some local tests to see if there's a size limit on couchdb. I can imagine maybe it times out when the input is too large.

[pdelteil]

Thanks @honoki,

I added the debug:true in the config file and the command cat domains-big.txt | bbrf domain add - -s subfinder --show-new now outputs:

[ERROR] local variable 'success' referenced before assignment

[honoki]

@pdelteil - Have you updated to v1.1.3 already?

[pdelteil]

@pdelteil - Have you updated to v1.1.3 already?

After updating I get this result:

cat domains-big.txt | bbrf domain add - -s subfinder --show-new
[DEBUG] getting program scope
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): domain.tld:6984
DEBUG:urllib3.connectionpool:https://domain.tld:6984 "GET /bbrf/program HTTP/1.1" 200 794
[DEBUG] getting program blacklist

And nothing is added.

[z0mb13s3c]

Yup, 1.1.2 & 1.1.3 give me both the same error. At first I thought it's something to do with subfinder but then I tried chaos, enabled the debug and figured it's bbrf. I'm not sure if it's the client or the server though. Here's the full output, maybe it helps.

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): xxx.xxx.xxx.xxx:1234
DEBUG:urllib3.connectionpool:https://xxx.xxx.xxx.xxx:1234 "GET /bbrf/program HTTP/1.1" 200 862
[DEBUG] getting program scope
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): xxx.xxx.xxx.xxx:12345
DEBUG:urllib3.connectionpool:https://xxx.xxx.xxx.xxx:1234 "GET /bbrf/program HTTP/1.1" 200 862
[DEBUG] getting program blacklist
[DEBUG] adding document in bulk
DEBUG:urllib3.connectionpool:https://xxx.xxx.xxx.xxx:1234 "POST /bbrf/_bulk_docs HTTP/1.1" 400 91
[ERROR] string indices must be integers

[honoki]

@z0mb13s3c - can you post the BBRF command that results in this error? The 400 bad request looks like it might not like some characters in the POST body. Also feel free to share (or privately send me) the input that you're trying to feed to BBRF. That should help me troubleshoot.

[honoki]

@pdelteil - same question :) And can you verify the program in- and outscope are well defined? The absence of an error message makes me think it doesn't find suitable candidates to add.

[anasbousselham]

Hi, I have the same error with a huge sub file : [DEBUG] getting program blacklist Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'> BrokenPipeError: [Errno 32] Broken pipe [DEBUG] adding document in bulk DEBUG:urllib3.connectionpool:https://ip:443 "POST /bbrf/_bulk_docs HTTP/1.1" 400 91 [ERROR] string indices must be integers

[honoki]

Hi @anasbousselham - thanks for the input; could you specify the number of records you are trying to push?

[anasbousselham]

7288 records. https://stackoverflow.com/questions/26692284/how-to-prevent-brokenpipeerror-when-doing-a-flush-in-python/26738736

[honoki]

Ok, I was able to reproduce. Looking into it now.

[anasbousselham]

Merci 👍

[pdelteil]

@pdelteil - same question :) And can you verify the program in- and outscope are well defined? The absence of an error message makes me think it doesn't find suitable candidates to add.

The inscope and outscope are well defined. I'm trying to add 5 MM records.

[honoki]

I was unable to determine the root cause so far; I have pushed v1.1.4 with some more verbose output when debug mode is enabled. Could you please log the complete stack trace here when reproducing this after upgrading?

[honoki]

Ok, thanks to @z0mb13s3c we finally figured out the problem, which is that document identifiers with a leading _ are not supported in CouchDB. Will look into ways to remedy this.

[anasbousselham]

the limit before string indices must be integers is the 4050 lines in my case.