Hellfire not accepting canid server

nstudach commented 6 years ago

In the python script first canid is started with subprocess.Popen to keep it running. Then Hellfire is started with subprocess.call Both have environmet set to PATH.

        path = {'GOPATH': '/root/work/',
                'GOROOT':'/root/go/',
                'PATH':'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/go/bin:/root/work/bin'
                } 
        Popen(['setsid', 'canid', '-port', '8043'], env = path)
        call('hellfire --topsites --canid localhost:8043 > topsites.ndjson 2> helf.txt', shell=True, env=path)

The problem is that Hellfire is not seeing canid it seems. When running the script localy Hellfire works as sexpected.

The stderr: (helf.txt)

panic: Get http://localhost:8043/prefix.json?addr=54.154.71.64: EOF

goroutine 319 [running]:
pathspider.net/hellfire.GetAdditionalInfo(0xc00007b248, 0x4, 0x4, 0x7ffd4ecb3f45, 0xe, 0x0)
        /root/work/src/pathspider.net/hellfire/canid.go:16 +0x2a5
pathspider.net/hellfire.lookupWorker.func1(0xc00007e960, 0x12b, 0xc00017ae50, 0xc00007e900, 0xc00005e240, 0x6eb469, 0x4, 0x7ffd4ecb3f45, 0xe)
        /root/work/src/pathspider.net/hellfire/hellfire.go:163 +0x591
created by pathspider.net/hellfire.lookupWorker
        /root/work/src/pathspider.net/hellfire/hellfire.go:136 +0xbc

nstudach commented 6 years ago

@britram any ideas?

britram commented 6 years ago

would be difficult for me to guess what is happening here without standing up the full stack myself, which I don't have cycles to do.

Have not memorized the subprocess package, but I'm not sure Popen is what you want to do here: could canid be blocking because nobody is reading its output?

Try running a standalone instance of canid somewhere, and pointing hellfire at that, as opposed to integrating it into your toolchain, and seeing if the problem persists.

nstudach commented 6 years ago

Try running a standalone instance of canid somewhere, and pointing hellfire at that, as opposed to integrating it into your toolchain, and seeing if the problem persists.

This solves the problem. Well not quite, but the reason is clear know. While Popen keeps canid running, the subprocess.call of hellfire terminates canid.

I am sure there is a easy enough solution for this. Thanks

nstudach commented 6 years ago

Update: I assumed wrong. Cause is still unknown

I tried the following:

canid manually, hellfire via python ->works
canid via python, hellfire manually -> fail: canid works, port is listening. Starting hellfire manually kills canid
Starting both canid and hellfire via python but logged into the machine via ssh -> works
Starting both canid and hellfire via python but logged into the local machine -> fails (original problem)

starting manually = Connect via SSH, enter command.

nstudach / sugar

Hellfire not accepting canid server #16