search

CIRCL / AIL-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project
https://github.com/ail-project/ail-framework
GNU Affero General Public License v3.0
1.3k stars 285 forks source link

Displayed information #25

Closed SYNchroACK closed 10 years ago

SYNchroACK commented 10 years ago

Two Questions:

1) I'm only just seeing the follow information on webinterface which differs from your screenshot on documentation page. Is it ok? image

2) What means each icon in message column? http://s7.postimg.org/vobprz64r/image2.png

Starow commented 10 years ago

1) No, normally you should see a "Queues tables" and 2 others graphs (Dygraph), in the "Queue monitor" part. 2) The eye mean "detected" and the thumb up mean "verified" In other words you can have 45 detected but just 20 verified which mean that their domains exists and are responding.

SYNchroACK commented 10 years ago

@Starow thank you for your reply.

1) Hum... the logs table information come from your stream or its a local collector that is installed on VM? Because the problem could be that Im not receiving the information from stream. I tried check the traffic with tcpdump and i didnt see any traffic from and to your server.

Every 3.0s: netstat -ptn | grep "149."                                                                                       Wed Aug 27 15:51:52 2014

tcp        0      0 127.0.0.1:5003          127.0.0.1:38971         ESTABLISHED 1495/python2
tcp        0      0 127.0.0.1:5003          127.0.0.1:38967         ESTABLISHED 1495/python2
tcp        0      0 127.0.0.1:5003          127.0.0.1:38992         ESTABLISHED 1495/python2
tcp        0      0 127.0.0.1:5003          127.0.0.1:38985         ESTABLISHED 1495/python2
tcp        0      0 127.0.0.1:5003          127.0.0.1:38966         ESTABLISHED 1495/python2
tcp        0      0 127.0.0.1:5003          127.0.0.1:38972         ESTABLISHED 1495/python2
tcp        0      0 127.0.0.1:5003          127.0.0.1:38978         ESTABLISHED 1495/python2
tcp        0      0 127.0.0.1:5003          127.0.0.1:38982         ESTABLISHED 1495/python2
tcp        0      0 10.0.2.15:51735         1xx.xx.3x.x3:5556       ESTABLISHED 1351/python2
tcp6       0      0 ::1:48577               ::1:6380                ESTABLISHED 1495/python2
tcp6       0      0 ::1:58980               ::1:6381                ESTABLISHED 1495/python2
tcp6       0      0 ::1:53596               ::1:6381                ESTABLISHED 1495/python2

2) Ok, cool! But is a automatic process, right? Any information is not being analysed by a human, right? Sorry for the questions, but just curious and Im seeing an opportunity to create a descentralized way to analyse pastes and share the results within European CERTs. (may be it sounds crazy :P )

Starow commented 10 years ago

The logs table info came from the stream you want, so i think it's our now. And yes, it's an auto process (actually 2 differents) and it can contain sometime some false positive detection but normally there is not so much of them ;)

Vikran commented 10 years ago

Hi, I have similar issue after installation i am getting following screen (notice no query table and logs) ail_capture

Logs suggest : [2014-08-30 09:56] INFO: Queuing: Suscribed to channel filelist [2014-08-30 09:56] INFO: Queuing: Suscribed to channel filelist [2014-08-30 09:56] INFO: Queuing: Suscribed to channel words [2014-08-30 09:56] INFO: Queuing: Suscribed to channel Shortlines [2014-08-30 09:56] INFO: Queuing: Suscribed to channel creditcard_categ [2014-08-30 09:56] INFO: Queuing: Suscribed to channel onion_categ [2014-08-30 09:56] INFO: Queuing: Suscribed to channel mails_categ [2014-08-30 09:56] INFO: Queuing: Suscribed to channel web_categ [2014-08-30 09:56] INFO: Queuing: Suscribed to channel words [2014-08-30 09:56] INFO: Queuing: Suscribed to channel filelist

Script log : [2014-08-31 01:59] INFO: Script: Script subscribed to channel onion_categ Also, to clarify i can add any other term to onion_categ e.g. twitter etc and it should look in stream right ? I am looking for possibiliy with better trending as this can be great help for organizaiton moniotoring for pastes, and social media.

Regards

adulau commented 10 years ago

Hi Vikran,

Thanks for the feedback.

Could you check if the 3 redis instances are properly running?

Could you check the reattach in the various screen sessions running (screen -ls and then screen -r -d -S Script)? To be sure that the scripts are not stopped due to a missing configuration or alike.

Cheers.

Vikran commented 10 years ago

Hi ,

I tried reinstalling from scratch and sill same issue

There are screens on: 4558.Script (09/07/2014 03:58:04 PM) (Detached) 4443.Queue (09/07/2014 03:58:02 PM) (Detached) 4421.Logging (09/07/2014 03:58:02 PM) (Detached) 4397.LevelDB (09/07/2014 03:58:02 PM) (Detached) 4369.Redis (09/07/2014 03:58:01 PM) (Attached)

Tried attaching to different ones and still same.

Regards

p-ho commented 10 years ago

I experience the very same issue like @Vikran . Queue list, monitor and log are empty.

All my screens are on, too.

When I attach to a screen the path to the config file is displayed. I don't know if this indicates an error or not.

Starow commented 10 years ago

Normally all the scripts are running inside screens like :

4558.Script ( aproximatelly 10 scripts ) 4443.Queue ( 10 also ) 4421.Logging ( 2 or 3 ) 4397.LevelDB ( 2 ) 4369.Redis ( 3 ) Did you check all of them ? with the ctrl+a +" to display the list ( then you can select which one you want to attach ) You need to attach to each of them to see the crash output if there is some. If scripts are running correctly nothing is displayed when you go and attach them, or some debug infos...

If you see some error crash please copy-paste them here :) Thank you !

p-ho commented 10 years ago

@Starow I've already done that and I can attach to all the screens.

Here are my dumps:

$screen -ls
There are screens on:
    2752.Script (09/09/2014 10:16:09 AM)    (Detached)
    2689.Queue  (09/09/2014 10:16:08 AM)    (Detached)
    2668.Logging    (09/09/2014 10:16:08 AM)    (Detached)
    2644.LevelDB    (09/09/2014 10:16:07 AM)    (Detached)
    2614.Redis  (09/09/2014 10:16:07 AM)    (Detached)

When I attach to the screens I get the following outputs:

Script and Queue have the same output:

[...]/AIL-framework/bin/packages/config.cfg

Logging has no output.

Output of LevelDB:

HOSTADDR:(127.0.0.1)
DB_PATH:([...]/AIL-framework/LEVEL_DB_DATA/2014/)
PORT:(2014)
Server running successfully

Output of Redis:

[2643] 09 Sep 10:16:08.099 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
[2643] 09 Sep 10:16:08.100 # Redis can't set maximum open files to 10032 because of OS error: Operation not permitted.
[2643] 09 Sep 10:16:08.100 # Current maximum open files is 1024. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 2.8.14 (db6e874a/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in stand alone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6381
 |    `-._   `._    /     _.-'    |     PID: 2643
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'

[...]

[2643] 09 Sep 10:16:08.101 # Server started, Redis version 2.8.14
[2643] 09 Sep 10:16:08.101 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[2643] 09 Sep 10:16:08.101 * The server is now ready to accept connections on port 6381

Is any of these outputs indicating any error?

Thank you.

Starow commented 10 years ago

Okay, now I get it :) Everything is working, no error I presume, it's just that you don't have the "feed" of pastes to analyse.

If you are a CERT contact CIRCL for more info http://www.circl.lu/contact/ If not, we will add a description of the input feed format to your own feed from pystemon or alike. See pystemon : https://github.com/CIRCL/pystemon

p-ho commented 10 years ago

@Starow thanks I have to run pystemon and AIL simultanously. Did I get it right?

Starow commented 10 years ago

Yes, pystemon is gathering pastes which you'll be able to analyse with AIL. Then you can create your own feed with pastes that you collected and push it to AIL.

p-ho commented 10 years ago

@Starow Now I've tried to run them both. The following questions arised to me:

That's my approach for the redis part:

redis:
  queue: yes
  server: "localhost"
  port: 6381
  database: 0
Starow commented 10 years ago

@p-ho I don't know about the pystemon config and the yaml..

@adulau will push a "connector" soon inside the repo to interface pystemon with AIL. :)

Normally the queues don't need an extra configuration since when you'll have data to process the queues will shows up.

p-ho commented 10 years ago

@Starow Maybe you can tell me anyway what are the correct settings for the DB (port, database, etc..) to which data should be queued. Thank you.

Vikran commented 10 years ago

Thanks this is informative i have tried default as well as changed port YAML and its not showing anything. Tried AIL before launching pystemon and other way round its not picking up the queue. Will Appreciate any further direction which might help Thanks

adulau commented 10 years ago

Hi Vikran,

To ease the use of AIL, I added a feeder script (from pystemon to AIL) in #30. The default configuration matches the default setup of the Redis queue setup in Pystemon. I hope this helps.

Cheers

p-ho commented 10 years ago

I just would like to confirm briefly that with the help of #30 everything's working fine now.

adulau commented 10 years ago

Great, thank you for the feedback.