Feeder_Pystemon not importing

[Phil-ThePower-Pearce]

Hi, Is there a way of troubleshooting the import from feeder_pystemon?

I created a regex tracker, created a pastebin paste, pystemon has downloaded it, to the archive "/pystemon/archive/pastebin.com_pro/2020/01/29" directory & the AIL-Framework/PASTES dir. The feeder script is running (LAUNCH -f)

Ive checked the downloaded paste against my regex and it should hit. Ive also used the monitor on redis (loads of LPOP)

1580306576.298088 [10 127.0.0.1:51172] "LPOP" "pastes" 1580306576.308506 [10 127.0.0.1:51172] "LPOP" "pastes" 1580306576.318949 [10 127.0.0.1:51172] "LPOP" "pastes" 1580306576.329354 [10 127.0.0.1:51172] "LPOP" "pastes" 1580306576.339737 [10 127.0.0.1:51172] "LPOP" "pastes" 1580306576.350122 [10 127.0.0.1:51172] "LPOP" "pastes" 1580306576.360517 [10 127.0.0.1:51172] "LPOP" "pastes" 1580306576.370925 [10 127.0.0.1:51172] "LPOP" "pastes" 1580306576.376606 [0 127.0.0.1:53952] "SELECT" "10" 1580306576.376799 [10 127.0.0.1:53952] "LPUSH" "pastes" "archive/pastebin.com_pro/2020/01/29/xM0R2fk7"

Ive looked at the keys in the redis, I can see the db10 and a key of pastes but cant see any other data under that key being imported and nothing is "seen" on the tracker, and nothing appears when searching by tag. The queues seem to be doing something as they go green for a while every few mins

https://pasteboard.co/ISlQ4sc.png

Its seems quite empty compared to that of the screenshots of a working AIL environment.

Any ideas?

[Terrtia]

Hi @Phil-ThePower-Pearce ! Some issue with the tagging system have been fixed with 727bc5596691da467d9585e205a2a7d183c64506

Please let me know if this fix your issue.

https://pasteboard.co/ISlQ4sc.png

Its seems quite empty compared to that of the screenshots of a working AIL environment.

Any ideas?

Can you please check the list of providers in the statistic section ?

[Phil-ThePower-Pearce]

Ive killed the processes, and re-ran the LAUNCH script, ran pystemon, then LAUNCH -f. Starts for a while. Then goes to "no running queues"

In the monitor script, I can see a lot of "cleared invalid pid in module-type_xxxx" in the logs. Nothing is displayed in the gui regarding these logs.

The provider in the statistics seems to have populated. Nothing else seems to

Ive rebooted, and redone the above again. the global queue seems to be increasing, but never decreasing. the global queue goes to stuck, then all the queues in the gui disappear, also in the monitor I can see the queues arent running "stuck or idle, restarting disabled"

[Phil-ThePower-Pearce]

Update. Ive reset_ail with the script, created default user, removed all previous pastes from directories and started again.

Global still increases, but nothing ever happens to it http://tinyurl.com/smoprtj then http://tinyurl.com/trnxe98

Ive also tried manually importing the dir python3 ./import_dir.py -c 102 -n pastebin_pro -d /home/ubuntu/Apps/pystemon/archive/pastebin.com_pro/

All the queues look like they are doing something, but nothing is actually displayed in gui or logs

[Terrtia]

It seem like the Global module crashed. Can you`please check the output of this module in the Script_AIL screen ?

Here is a list of shortcuts to manage/navigate in the screen:

• Ctrl-a + d detach screen
• Ctrl-a + c Create new window
• Ctrl-a + n next window screen
• Ctrl-a + p previous window screen
• Ctrl-a + " get a list of all terminal/ select a terminal by name

[Phil-ThePower-Pearce]

screen is blank logging is blank flask is showing the website get commands queue is showing a static list of queues

Duplicates has no publisher.
DomClassifier has no publisher.
Indexer has no publisher.
Attributes has no publisher.
DumpValidOnion has no publisher.
WebStats has no publisher.
ModuleStats has no publisher.
MISP_The_hive_feeder has no publisher.
Release has no publisher.
SentimentAnalysis has no publisher.
submit_paste has no suscriber
Duplicates output queue finished.
Indexer output queue finished.
Attributes output queue finished.
DomClassifier output queue finished.
DumpValidOnion output queue finished.
WebStats output queue finished.
ModuleStats output queue finished.
MISP_The_hive_feeder output queue finished.
SentimentAnalysis output queue finished.
Release output queue finished.
submit_paste input queue finished.

ARDB screen - lots of Period statisitcs dumps redis screen - "ready to accept connections message"

[Terrtia]

You have multiple terminals/windows in the Script_AIL screen.

You can use this list of commands to navigate in this screen:

• Ctrl-a + n next window screen
• Ctrl-a + p previous window screen
• Ctrl-a + " get a list of all terminal/select a terminal by name

Can you please check all terminals outputs (Mixer, Global, Tag ...) ?

[Phil-ThePower-Pearce]

Mixer Empty Queues: Waiting... Empty Queues: Waiting... Empty Queues: Waiting... {'unnamed_feeder': 0} Mixer; ; ; ;mixer_all All_feeders Processed 0 paste(s) in 30sec Mixer; ; ; ;mixer_unnamed_feeder unnamed_feeder Processed 0 paste(s) in 30sec Mixer; ; ; ;mixer_unnamed_feeder unnamed_feeder Duplicated 0 paste(s) in 30sec Empty Queues: Waiting... Empty Queues: Waiting... Empty Queues: Waiting...

Global Global; ; ; ;glob Processed 0 paste(s) in 31.047673225402832 s Global; ; ; ;glob Processed 0 paste(s) in 31.030948638916016 s Global; ; ; ;glob Processed 46 paste(s) in 31.09716296195984 s Global; ; ; ;glob Processed 0 paste(s) in 31.03010892868042 s Global; ; ; ;glob Processed 3 paste(s) in 31.0340576171875 s Global; ; ; ;glob Processed 18 paste(s) in 31.059768438339233 s Global; ; ; ;glob Processed 14 paste(s) in 31.04516100883484 s Global; ; ; ;glob Processed 0 paste(s) in 31.031121730804443 s Global; ; ; ;glob Processed 14 paste(s) in 31.04599380493164 s Global; ; ; ;glob Processed 0 paste(s) in 31.030436754226685 s Global; ; ; ;glob Processed 12 paste(s) in 31.04767370223999 s Global; ; ; ;glob Processed 9 paste(s) in 31.04705286026001 s Global; ; ; ;glob Processed 0 paste(s) in 31.03182578086853 s Global; ; ; ;glob Processed 8 paste(s) in 31.0413658618927 s Global; ; ; ;glob Processed 5 paste(s) in 31.037606477737427 s Global; ; ; ;glob Processed 8 paste(s) in 31.039519786834717 s Global; ; ; ;glob Processed 0 paste(s) in 31.032305002212524 s Duplicates - Sleeping DomClassifier - Blank Categ - Blank CreditCards - Blank BankAccount - Blank Onion - Crawler Disabled Mail - Sleeping Apikey - blank web - sleeping credential - blank termtrackermod - blank regextracker - blank indexer Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/BLiqhVPY Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/SKX290W9 Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/8MjgdzFz Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/vXkPtbtJ Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/YWmsW6Gf Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/pkMWYiku Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/EgvHWQ1P Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/VujAY19F Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/gTPuD2N1 Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/1RtqNeYy Indexing - 1580809702 : archive/pastebin.com_pro/2020/02/04/GkjXSbiB keys -blank pgdump - blank decoder - blank cryptocurrency - blank tools - blank phone - blank release - sleeping cve - blank webstats - sleeping modulestats - sleeping sqlinjectiondetection - blank libinjection - blank msphivefeeder - keys not present tags - blank sentimentanalysis Dropped: application/x-empty Dropped: application/x-empty Dropped: application/x-empty Dropped: application/x-empty Dropped: application/x-empty Dropped: application/x-empty dbcleaner - Term Stats Cleaned updatebackground - blank submitpaste - blank ipaddress - blank

Hope this is of some value

This might be unrelated but another issue: After a long time, well after the global queue went red in the global terminal File already exist /home/ubuntu/Apps/AIL-framework/PASTES/archive/pastebin.com_pro/2020/02/04/m7GGJNGp Traceback (most recent call last): File "./Global.py", line 119, in <module> curr_file_content = f.read() File "/usr/lib/python3.6/gzip.py", line 276, in read return self._buffer.read(size) File "/usr/lib/python3.6/gzip.py", line 463, in read if not self._read_gzip_header(): File "/usr/lib/python3.6/gzip.py", line 411, in _read_gzip_header raise OSError('Not a gzipped file (%r)' % magic) OSError: Not a gzipped file (b'ex')

[Terrtia]

Do you have any tag in the UI ?

464 - It seem like you have an issue with your importer. All files need to be gziped.

(all invalid files are logged)

[Phil-ThePower-Pearce]

So all pastes need to be compressed in the pystemon.yaml?

[Terrtia]

You need to set the compress field in pystemon.yaml compress: yes # Store the pasties compressed