search

CIRCL / AIL-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project
https://github.com/ail-project/ail-framework
GNU Affero General Public License v3.0
1.3k stars 282 forks source link

Pystemon feeder and import dir not working #379

Open syloktools opened 5 years ago

syloktools commented 5 years ago

I started pystemon and it is capturing data but the feeder is not bringing it in. So I tried to import from one of the directories manually and I get the following error:

(AILENV) root@app1:/opt/AIL-framework/bin# ./import_dir.py -d /opt/pystemon/alerts/pastebin.com_pro/2019/09/03/2JWnZ8Fw.gz
Traceback (most recent call last):
  File "./import_dir.py", line 67, in <module>
    socket.bind("tcp://*:{}".format(args.port))
  File "zmq/backend/cython/socket.pyx", line 550, in zmq.backend.cython.socket.Socket.bind
  File "zmq/backend/cython/checkrc.pxd", line 25, in zmq.backend.cython.checkrc._check_rc
zmq.error.ZMQError: Address already in use
Terrtia commented 5 years ago

It seem like a process is already using the feeder port. Can you check if a process is already using the 5556 port?

syloktools commented 5 years ago

I am reinstalling now. If the problem persists I will comment if it does not a will close this issue.

syloktools commented 5 years ago

Quick question before I proceed. Does ./bin/feeder/pystemon-feeder.py have to be ran in the AIL virtual environment and should be in ran in the background, &?

Patrick-Kelley commented 5 years ago

I generally run it in the AIL_ENV with screen on a linux host.

On Tue, Sep 3, 2019 at 11:46 AM Robert Nixon notifications@github.com wrote:

Quick question before I proceed. Does ./bin/feeder/pystemon-feeder.py have to be ran in the AIL virtual environment and should be in ran in the background, &?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/CIRCL/AIL-framework/issues/379?email_source=notifications&email_token=ABGPPY7X4HRB72GTUKFFWPTQH2BG5A5CNFSM4ITA3Q4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5YUT2A#issuecomment-527518184, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGPPY2C77XFUAPODD4QDWTQH2BG5ANCNFSM4ITA3Q4A .

--

Patrick Kelley, CISSP, C|EH, ITIL CTO patrick.kelley@criticalpathsecurity.com (o) 770-224-6482

The limit to which you have accepted being comfortable is the limit to which you have grown. Accept new challenges as an opportunity to enrich yourself and not as a point of potential failure.

syloktools commented 5 years ago

Thanks @logikphreak

syloktools commented 5 years ago

Manual paste works, going to test with directory soon.

syloktools commented 5 years ago

Directory import works. I have the feeder running but it is not auto ingesting. How can I troubleshoot this? It is copying all the directories from the Pystemon path to the PASTES directory in the AIL-Framework directory.

syloktools commented 5 years ago

@Terrtia Any thoughts? I am about to write my own script to push them through the API.

Terrtia commented 5 years ago

You can't use the pystemon and the import dir feeder at the same time. they are both using the same port.

Who is copying all the paste ? pystemon or AIL ?

I need to add a new option to the import dir script. This way we can submit paste via ZMQ or the API.

syloktools commented 5 years ago

That is correct. The pystemon feeder did not copy the files. That was my mistake. That was done when I used import dir script. When I launch the pystemon feeder it just sits there, if I kill it it always shows that it was in its sleep loop where it is looking for pastes.

My pystemon script lives in /opt/pystemon/

I have that in the config. But the feeder is not finding all the data in there.

I got my IP whitelisted by CIRCL yesterday and that data is coming in, so I know that part works.