No running queues Or no feed

[MickeyAl]

Hi all,

I have installed ail using the docker image by following the instructions. The docker image deployment went Successfully but the is no activity on the dashboard.

Any ideas?

thanks in advance.

[mokaddem]

Hi, What is your data source? Are you using pystemon to feed the plateform or are you using the import_dir.py file?

[MickeyAl]

Hi, I'm using pystemon and I edited the ail config file accordingly. Script seems to be running well, but no feeding (same for the pystemon-feeder script)

[mokaddem]

Hi, Sorry for the late response...

Could you share the output of pystemon-feeder.py as well as your configuration file? A misconfiguration could prevent one of the script to pop pastes.

[Asunhack]

Hi, I've the same problem, all working good and I reboot my server and this "error" appear.. You want a screen ?

[mokaddem]

Sure!

[Asunhack]

[mokaddem]

Do you have the latest version of AIL? We added a backoff time (see #169) From what I can see, pystemon is not pushing pastes to redis, so pystemon-feeder.py has nothing to pop. Can you check that pystemon is really fetching pastes? (by checking output logs of pystemon.py or by checking inside the redis server)

[Asunhack]

He really fetching paste, I've recolted paste last week and friday I reboot all and the error appear.. However I've change nothing in the pystemon-feeder.py script... One output of pystemon

So I guess it works good ..

[Asunhack]

I don't know if that can be related but I've this output sometimes from Flask_server.py

[Asunhack]

I did the modification of #169 but nothing change...

[mokaddem]

This error of Flask_server.py is a known 'bug' and is not related to your issue. So, pystemon seems to be downloading. Can you check that the option redis->queue is sets to yes in pystemon.yaml? If it isn't, can you set it and try again?

[Asunhack]

Ok. Yes the option redis->queue is set to yes..

[Asunhack]

I check yhe output of :

(1) Queue_AIL 
(2) Logging_AIL
(3) LevelDB_AIL
(4) Redis_AIL

They were normal

But the outuput of Script_AIL was only : spleeping (exept forMixer.py)

I don't know if that can be related ?

[mokaddem]

The sleeping output is normal, it just means that there is nothing to process; which is normal as AIL is not receiving pastes. Anyway, it seems that the problem is between pystemon and AIL since pystemon-feeder.py is not receiving data. Can you manually check that pystemon is writing pastes on disk and that it is pushing paths into redis (by doing a monitor or checking inside the key).

[Asunhack]

I think it pystemon didn't writing pastes because inlogs/Script_info_-2018-02-13.log I have only that :

How can I check that it is pushinh paths into redis ?

[mokaddem]

The chain is as follow: pystemon -> pystemon-feeder -> mixer -> global -> ...

Does pystemon writes pastes to disk?

To check in redis (with pystemon running):

• connect to the redis server with the redis client: redis-cli -p 6379 -n 10
• execute the monitor command: monitor
• check for the LPUSH instructions

[Asunhack]

In pystemon.yaml I set save-all : yes into archive: And now i'ts working !

Thanks you very much for your help and your time !

[mokaddem]

Good! @MickeyAl does this solution solves your issue?

[cybertschunk]

I've unfortunately the same problem. I'm also using pystemon to feed the framework, but pystemon-feeder isn't giving any output. On the website it says "No running queues or no feed". I'm using an up to date version and followed the setup instructions

[mokaddem]

Can you check your pystemon configuration? Does save-all : yes and queue: yes apply? What is the output of pystemon? Is it downloading pastes?

[Asunhack]

And don't you miss to start pytemon-feeder.py ? It's located in /bin/feeder

[elbae]

Hi, I'm facing the same problem. The scripts pystemon.py and pystemon-feeder.py are running. Redis->queue is enabled in pystemon.yaml, with db 10 and save-all is marked yes. redis-cli -p 6379 -n 10 shows proper activities. Data is save under pystemon dir (ie /opt/pystemon/archive/slexy.org/2019/07/16/s2ZHVerp6i.gz). Do I miss something?

[elbae]

Ok, I was missing this:

# 101 pastes processed feed
# 102 raw pastes feed
topic = '102'

instead of topic = '101'

[DeepCodeSec]

In my case, pystemon was failing because the path defined by archive > dir-all in pystemon.yaml did not exist. After creating the directory archive, pastes were downloaded and fed into AIL.

[albatroaz94]

Hi, I have the same "No running queues" problem. I have followed the instructions and pystemon is downloading the pastes but the problem is as soon as i run pystemon-feeder.py it gives the following error.

root@miyuru-VirtualBox:/home/miyuru/AIL-framework/bin/feeder# ./pystemon-feeder.py Traceback (most recent call last): File "./pystemon-feeder.py", line 19, in import zmq ModuleNotFoundError: No module named 'zmq'

i have applied "save-all : yes" and "queue: yes" and "redis-cli -p 6379 -n 10" is prompting activities too.

Any help would be greatly appriciated. Thank you.

[Asunhack]

Hi, It's the first time that you run AIL ? Do you have the latest version of AIL ? Did you used pystemon.yaml ?

Can you check that redis -> queue is sets to yes in pystemon.yaml ?

And did you check if pystemon if really fetching pastes ..?

[albatroaz94]

Hi, sorry for the late reply. Yes this is my first time and thank you for pointing out that i was using an older version of AIL and i reinstalled the latest version. by the way, the above mentioned error has solved and pystemon-feeder.py showing some activities.

(AILENV) root@miyuru-VirtualBox:~/AIL-framework/bin/feeder# ./pystemon-feeder.py archive/pastebin.com/2019/11/21/3A5mx7nk.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/70PAeHKT.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/G08XGkJy.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/NQk2sNjX.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/xqEy29Xi.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/3gh1Aps5.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/5hQ0kCw0.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/QrppxkCg.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/z2kcjkVa.gz IOError: Increasing sleep time archive/pastebin.com/2019/11/21/9RWhRadU.gz IOError: Increasing sleep time

Is this normal? redis -> queue is set to yes in pystemon.yaml and i think pystemon is fetching pastes. pastes are saved under pystemon/archive/ directory. The "No running queues Or no feed" remains the same. Any solution?

[mokaddem]

Hey @albatroaz94, Could you check that your pystemon path is set correctly in /configs/core.cfg under the [Directories] section pystemonpath variable. Basically, the pystemonpath should be set so that

pystemonpath + 'archive/pastebin.com/2019/11/21/z2kcjkVa.gz'

is the absolute path.

[rc042]

Hi, I don't know if it's better to open a new thread... I try here and apologies if I digging up this topic.

I have the same issue and the web interface "No running queue or no feed".

I use pystemon from https://github.com/cvandeplas/pystemon because I have some issue with the one provided by CIRCL.

• pystemon is started and file are downloaded in gz
• pystemon-feeder.py is launched and archives seems to be processed [1]
• Redis seems to be working fine, I have the LPUSH [2]
• I checked the different screens but I didn't see something relevant [3]

I probably forget something but I don't know where to search.

Thanks for your feedbacks :)

[1]

archive/gist.github.com/2020/05/07/polRk_772cbed408a6d387ae99ea0ef90d5305.gz
archive/ideone.com/2020/05/07/FGKKy0.gz
archive/ideone.com/2020/05/07/GlVgVc.gz
archive/ideone.com/2020/05/07/Ej9A5o.gz

[2]

1588856982.151460 [0 10.211.55.7:49111] "SELECT" "10"
1588856982.151696 [10 10.211.55.7:49111] "LPUSH" "pastes" "archive/gist.github.com/2020/05/07/polRk_675f464b32540b3e99dbbcfc018f590d.gz"
1588856982.153305 [10 127.0.0.1:37198] "LPOP" "pastes"

[3]

Redis_AIL
ARDB_AIL
Logging_AIL
Queue_AIL
Script_AIL
Flask_AIL

[Gr3gbug]

Hi Guys, I'm in the same situation of @reedcrif . I try to do all the suggested steps, and it seems that all works fine, but i can't see anything on the AIL framework. Some suggestions about this issue? Below you can see all configurations file and some screenshot, in the following order: AIL framework, core.cfg file, pystemon yaml file, pystemon_feeder.py output, pystemon output, redis output. AIL SCREEN core.cfg file of ail Pystemon Yaml file Pystemon feeder output Pystemon Output Redis output

[Gr3gbug]

Hi, I finally solved the issue. The problem was related to the default pystemonpath variable inside the core.cfg file. Because in the latest update of pystemon, both the folder of "archives" and "alerts" are under <your-path>/pystemon and not as before, under <your-path>/pystemon/pystemon. So, at least for me, this change in the core.cfg file, solved the issue.