Luiska12 commented 4 years ago

Hello!

I'm trying to install AIL on my environment, but when I finish to launch the script ./LAUNCH.sh -l it appears that nothing is working on my https://localhost:7000/

I tried to install the tool repeatedly but nothing works.

I'm using Ubuntu 18.04 with 8 RAM and 50 Gb HDD

Thanks!

BeRambo commented 4 years ago

I'm having the same problem, have you resolved this issue yet?

researchlab17 commented 4 years ago

^ Same have never been able to get this Framework to "work" on 3 different servers over the period of 3 months now.

Sorrow81 commented 4 years ago

Same here en 20.04 LTS

BeRambo commented 4 years ago

My problem was that I was calling http instead of httpS. Hope that works for you too

Sorrow81 commented 4 years ago

No same with https and http

I suppose web server not starting

BeRambo commented 4 years ago

Have you tried loading the page locally using curl? I think you should be able to with curl -k https://127.0.0.1:7000

Sorrow81 commented 4 years ago

Hmm connection refused

sorrow@sorroz-vm:~/Documents/ail-framework$ . ./AILENV/bin/activate
(AILENV) sorrow@sorroz-vm:~/Documents/ail-framework$ cd bin/
(AILENV) sorrow@sorroz-vm:~/Documents/ail-framework/bin$ ./LAUNCH.sh -l
    * Checking configuration
Config File: Nothing to update
Config File: Nothing to update
    * Configuration up-to-date
******************************************************************
*                        Updating AIL ...                        *
******************************************************************
Modified Files:
installing_deps.sh

Please, commit your changes or stash them before you can update AIL

Aborting ...
******************************************************************
*                    AIL Not Updated                             *
******************************************************************

    * Launching Redis servers
    * Launching ARDB servers
    * Launching logging process
    * Launching all the queues
    * Checking configuration
Config File: Nothing to update
Config File: Nothing to update
    * Configuration up-to-date
    * Launching scripts
    * Launching Flask server
(AILENV) sorrow@sorroz-vm:~/Documents/ail-framework/bin$ curl -k https://127.0.0.1:7000
curl: (7) Failed to connect to 127.0.0.1 port 7000: Connexion refusée
(AILENV) sorrow@sorroz-vm:~/Documents/ail-framework/bin$ curl -k http://127.0.0.1:7000
curl: (7) Failed to connect to 127.0.0.1 port 7000: Connexion refusée
(AILENV) sorrow@sorroz-vm:~/Documents/ail-framework/bin$

BeRambo commented 4 years ago

Hmm that's weird, are you running the latest version? And have you tried manually installing all components using pip3 install -r requirements.txt?

And can you check what screen -r Flask_AIL returns?

Terrtia commented 4 years ago

Hi @Sorrow81 !

Can you please check the output of the Flask server with screen -r Flask_AIL.

annetteshajan commented 4 years ago

I'm getting a similar error: On screen -r Flask_AIL it returns: blueprints Flask_server.py static create_default_user.py modules templates create_new_web_module.py server.crt update_thirdparty.sh Flask_base_template.py server.key Misp not connected The HIVE not connected VT submission is disabled Traceback (most recent call last): File "/usr/lib/python3.6/configparser.py", line 1138, in _unify_values sectiondict = self._sections[section] KeyError: 'Splash_Manager'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./Flask_server.py", line 41, in from blueprints.crawler_splash import crawler_splash File "/home/shajan/AIL-framework/var/www/blueprints/crawler_splash.py", line 28, in import crawlers File "/home/shajan/AIL-framework//bin/lib/crawlers.py", line 41, in splash_manager_url = config_loader.get_config_str('Splash_Manager', 'splash_url') File "/home/shajan/AIL-framework//bin/lib/ConfigLoader.py", line 45, in get_config_str return self.cfg.get(section, key_name) File "/usr/lib/python3.6/configparser.py", line 781, in get d = self._unify_values(section, vars) File "/usr/lib/python3.6/configparser.py", line 1141, in _unify_values raise NoSectionError(section) configparser.NoSectionError: No section: 'Splash_Manager'

viszsec commented 3 years ago

anyone can write something workable installing this AIL ? Like installation tutorial and get the AIL to work with ubuntu 64bit? If we follow exactly from official CIRCL AIL, it doesnt work at all.

gallypette commented 3 years ago

git clone https://github.com/ail-project/ail-framework.git
cd ail-framework
./installing_deps.sh

cd ~/ail-framework/
cd bin/
./LAUNCH.sh -l

I followed these setps this morning on a fresh Ubuntu 20.04 without issues.

viszsec commented 3 years ago

git clone https://github.com/ail-project/ail-framework.git
cd ail-framework
./installing_deps.sh

cd ~/ail-framework/
cd bin/
./LAUNCH.sh -l
I followed these setps this morning on a fresh Ubuntu 20.04 without issues.

well, the think is..did you get the web interface running?

viszsec commented 3 years ago

FYI, i followed exactly as what you did previously and nothing works..not been able to access to the web interface.

gallypette commented 3 years ago

Sure I got the web interface running. What error do you get when going to https://127.0.0.1:7000 if any? Can you go into the Flask screen screen -r Flask_AIL and check whether there are some error there? If Flask does not spit out errors but you cannot access the web interface, my guess is that on of the redis server does not manage to start. You can go check that with screen -r Redis_AIL? (there are 3 windows in this screen, make sure to check the one named 6379 by using the screen shortcut Ctrl+a 1)

viszsec commented 3 years ago

Sure I got the web interface running. What error do you get when going to https://127.0.0.1:7000 if any? Can you go into the Flask screen screen -r Flask_AIL and check whether there are some error there? If Flask does not spit out errors but you cannot access the web interface, my guess is that on of the redis server does not manage to start. You can go check that with screen -r Redis_AIL? (there are 3 windows in this screen, make sure to check the one named 6379 by using the screen shortcut Ctrl+a 1)

if you do the curl command, with an error port 7000: connection refused.

if screen -r FLASK_AIL, it shows as there is no screen to be resumed matching FLASK_AIL

gallypette commented 3 years ago

screen -ls ?

viszsec commented 3 years ago

screen -ls ?

    10472.Flask_AIL (12/23/2020 10:30:41 AM)        (Detached)
    10265.Script_AIL        (12/23/2020 10:30:37 AM)        (Detached)
    10063.Queue_AIL (12/23/2020 10:30:34 AM)        (Detached)
    10045.Logging_AIL       (12/23/2020 10:30:34 AM)        (Detached)
    10023.ARDB_AIL  (12/23/2020 10:30:34 AM)        (Detached)
    10003.Redis_AIL (12/23/2020 10:30:34 AM)        (Detached)

viszsec commented 3 years ago

If i attached flask_ail redis and ardb, stilll not able to access them. do you have any idea?

viszsec commented 3 years ago

now without screen been attached..with curl command..

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">

Redirecting...

You should be redirected automatically to target URL: /login?next=%2F. If not click the link.(AILENV pretty odd, with curl you can access them, but not with browser.

gallypette commented 3 years ago

The curl bit is odd. But we still need to test my assumption :) What happens if you attach Redis_AIL by: screen -r Redis_AIL and then type Ctrl+a 1 Do you see any error there?

viszsec commented 3 years ago

The curl bit is odd. But we still need to test my assumption :) What happens if you attach Redis_AIL by: screen -r Redis_AIL and then type Ctrl+a 1 Do you see any error there?

it shown like this:

25524:M 24 Dec 2020 05:03:40.729 # Server initialized 25524:M 24 Dec 2020 05:03:40.729 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 25524:M 24 Dec 2020 05:03:40.729 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 25524:M 24 Dec 2020 05:03:40.730 * Ready to accept connections

viszsec commented 3 years ago

now via browser, if you access it, there is an error:

Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

viszsec commented 3 years ago

after fixing all the error for Redis_AIL, and screen it,,there is no error but now different issues came out.

./LAUNCH -m

tried to launch the rest..

but got this..

Script not started, waiting 5 more secondes Could not connect to Redis at 127.0.0.1:6379: Connection refused 6379 not ready Could not connect to Redis at 127.0.0.1:6380: Connection refused 6380 not ready Could not connect to Redis at 127.0.0.1:6381: Connection refused 6381 not ready Script not started Script not started, waiting 5 more secondes Could not connect to Redis at 127.0.0.1:6379: Connection refused 6379 not ready Could not connect to Redis at 127.0.0.1:6380: Connection refused 6380 not ready Could not connect to Redis at 127.0.0.1:6381: Connection refused 6381 not ready Script not started Script not started, waiting 5 more secondes Could not connect to Redis at 127.0.0.1:6379: Connection refused 6379 not ready Could not connect to Redis at 127.0.0.1:6380: Connection refused 6380 not ready Could not connect to Redis at 127.0.0.1:6381: Connection refused 6381 not ready Script not started Script not started, waiting 5 more secondes Could not connect to Redis at 127.0.0.1:6379: Connection refused 6379 not ready Could not connect to Redis at 127.0.0.1:6380: Connection refused 6380 not ready Could not connect to Redis at 127.0.0.1:6381: Connection refused 6381 not ready Script not started Script not started, waiting 5 more secondes Could not connect to Redis at 127.0.0.1:6379: Connection refused 6379 not ready Could not connect to Redis at 127.0.0.1:6380: Connection refused 6380 not ready Could not connect to Redis at 127.0.0.1:6381: Connection refused 6381 not ready Script not started Script not started, waiting 5 more secondes

viszsec commented 3 years ago

when we try to launch the screen for Flask_AIL..

this came out..

ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/client.py", line 667, in execute_command connection.send_command(args) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 610, in send_command self.send_packed_command(self.pack_command(args)) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 585, in send_packed_command self.connect() File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 489, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to localhost:6382. Connection refused.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 484, in connect sock = self._connect() File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 541, in _connect raise err File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 529, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./Flask_server.py", line 246, in Tag.enable_taxonomy(taxo) File "/home/ail/ail-framework/bin/packages/Tag.py", line 98, in enable_taxonomy r_serv_tags.sadd('active_taxonomies', taxonomie) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/client.py", line 1600, in sadd return self.execute_command('SADD', name, values) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/client.py", line 673, in execute_command connection.send_command(args) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 610, in send_command self.send_packed_command(self.pack_command(*args)) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 585, in send_packed_command self.connect() File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 489, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to localhost:6382. Connection refused.

gallypette commented 3 years ago

All of these errors are connections refused because of redis servers failing to start. The most likely to me is that you launched the script twice. You can use LAUNCH.sh -k to kill all screens and go back to a clean slate. Then relaunch the whole bunch of screen with LAUNCH.sh -l

All Redis server (6379 6380 6381) must be running. My guess from the beginning is that you already have a redis server running on 6379 because you may have installed redis via apt-get for smth else (in such case sudo systemctl stop redis will solve it).

it shown like this:

25524:M 24 Dec 2020 05:03:40.729 # Server initialized 25524:M 24 Dec 2020 05:03:40.729 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 25524:M 24 Dec 2020 05:03:40.729 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 25524:M 24 Dec 2020 05:03:40.730 * Ready to accept connections

This looks all right (but I am not sure if the redis running on 6379 as the beginning of this redis output is truncated): If it does not look like in the following https://asciinema.org/a/YF642HDlTfWHfWL7iUk8qKo9Y then there is an issue.

now via browser, if you access it, there is an error:

Internal Server Error

The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Like you did before check out the Flask_AIL screen for errors.

viszsec commented 3 years ago

for Redis...successfully launched with no errors shown..but still not able to access via web interface. for Flask, successfully and attached..results for Flask:

blueprints Flask_server.py static create_default_user.py modules templates create_new_web_module.py server.crt update_thirdparty.sh Flask_base_template.py server.key Misp not connected The HIVE not connected VT submission is disabled

Serving Flask app "Flask_server" (lazy loading)
Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
Debug mode: off
Running on https://127.0.0.1:7000/ (Press CTRL+C to quit) 127.0.0.1 - - [24/Dec/2020 10:13:56] "GET / HTTP/1.1" 302 -

viszsec commented 3 years ago

All of these errors are connections refused because of redis servers failing to start. The most likely to me is that you launched the script twice. You can use LAUNCH.sh -k to kill all screens and go back to a clean slate. Then relaunch the whole bunch of screen with LAUNCH.sh -l

All Redis server (6379 6380 6381) must be running. My guess from the beginning is that you already have a redis server running on 6379 because you may have installed redis via apt-get for smth else (in such case sudo systemctl stop redis will solve it).

it shown like this: 25524:M 24 Dec 2020 05:03:40.729 # Server initialized 25524:M 24 Dec 2020 05:03:40.729 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 25524:M 24 Dec 2020 05:03:40.729 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 25524:M 24 Dec 2020 05:03:40.730 * Ready to accept connections

This looks all right (but I am not sure if the redis running on 6379 as the beginning of this redis output is truncated): If it does not look like in the following https://asciinema.org/a/YF642HDlTfWHfWL7iUk8qKo9Y then there is an issue.

now via browser, if you access it, there is an error: Internal Server Error The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

Like you did before check out the Flask_AIL screen for errors.

When I got it accessed via interface..got this error:

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 484, in connect sock = self._connect() File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 541, in _connect raise err File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 529, in _connect sock.connect(socket_address) ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise raise value File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/home/ail/ail-framework/var/www/blueprints/root.py", line 45, in login login_failed_ip = r_cache.get('failed_login_ip:{}'.format(current_ip)) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/client.py", line 976, in get return self.execute_command('GET', name) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/client.py", line 673, in execute_command connection.send_command(args) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 610, in send_command self.send_packed_command(self.pack_command(args)) File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 585, in send_packed_command self.connect() File "/home/ail/ail-framework/AILENV/lib/python3.8/site-packages/redis/connection.py", line 489, in connect raise ConnectionError(self._error_message(e)) redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused. 127.0.0.1 - - [24/Dec/2020 10:21:35] "GET /login?next=%2F HTTP/1.1" 500 - 127.0.0.1 - - [24/Dec/2020 10:21:35] "GET /favicon.ico HTTP/1.1" 302 -

adulau commented 3 years ago

Could you run a netstat -anpt | grep LISTEN and show us the output? I have the impression that redis and ardb are not listening or binding on the wrong interface or protocol (v4/v6)? Thanks

viszsec commented 3 years ago

Could you run a netstat -anpt | grep LISTEN and show us the output? I have the impression that redis and ardb are not listening or binding on the wrong interface or protocol (v4/v6)? Thanks

Here are the results:

(AILENV) ail@ail:~/ail-framework$ netstat -anpt | grep LISTEN (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 127.0.0.1:6382 0.0.0.0: LISTEN 13602/ardb-server
tcp 0 0 127.0.0.53:53 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN -
tcp 0 0 127.0.0.1:631 0.0.0.0: LISTEN -
tcp 0 0 127.0.0.1:9050 0.0.0.0: LISTEN -
tcp6 0 0 :::22 ::: LISTEN -
tcp6 0 0 ::1:3350 ::: LISTEN -
tcp6 0 0 ::1:631 ::: LISTEN -
tcp6 0 0 :::3389 :::* LISTEN -

with ROOT results:

tcp 0 0 127.0.0.1:6382 0.0.0.0: LISTEN 13602/ardb-server
tcp 0 0 127.0.0.53:53 0.0.0.0: LISTEN 722/systemd-resolve tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN 939/sshd: /usr/sbin tcp 0 0 127.0.0.1:631 0.0.0.0: LISTEN 746/cupsd
tcp 0 0 127.0.0.1:9050 0.0.0.0: LISTEN 890/tor
tcp6 0 0 :::22 ::: LISTEN 939/sshd: /usr/sbin tcp6 0 0 ::1:3350 ::: LISTEN 860/xrdp-sesman
tcp6 0 0 ::1:631 ::: LISTEN 746/cupsd
tcp6 0 0 :::3389 :::* LISTEN 893/xrdp

adulau commented 3 years ago

`redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.``

As you can see in your netstat output, your redis server is not started.

viszsec commented 3 years ago

`redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.``

As you can see in your netstat output, your redis server is not started.

Even if get it restarted..still not able to access the web interface with 302 http error code.

Now all these 3 has been restarted and attached.

    20117.Flask_AIL (12/24/2020 01:05:05 PM)        (Attached)
    20101.ARDB_AIL  (12/24/2020 01:05:05 PM)        (Attached)
    20074.Redis_AIL (12/24/2020 01:05:04 PM)        (Attached)

3 Sockets in /run/screen/S-ail.

tcp 0 0 127.0.0.53:53 0.0.0.0: LISTEN -
tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN -
tcp 0 0 127.0.0.1:631 0.0.0.0: LISTEN -
tcp 0 0 127.0.0.1:9050 0.0.0.0: LISTEN -
tcp6 0 0 :::22 ::: LISTEN -
tcp6 0 0 ::1:3350 ::: LISTEN -
tcp6 0 0 ::1:631 ::: LISTEN -
tcp6 0 0 :::3389 ::: LISTEN -

Then, what else missing from this part?

viszsec commented 3 years ago

finally, i got it work. Only launch redis, ardb and flask.

but

the web interface can only be accessed locally. Not able to access from remote os. Other vm instances works well but not this one.

Should allow port via ufw?

gallypette commented 3 years ago

good.

You need to bind the Flask to an interface that is reachable from outside or 0.0.0.0. By default it's 0.0.0.0 so it shoud be ok, check config/core.cfg.

And yes, if you have a firewall rules blocking that traffic then you have to change these rules accordingly.

lobojc commented 2 years ago

Hello AIL Framework team, I have the same problem running AIL Framework, the Website doesn't start.

I installed pybgpranking manually because it failed and I updated jsonschema to 4.9.1

ERROR: pymisp 2.4.160.1 has requirement jsonschema<5.0.0,>=4.9.1, but you'll have jsonschema 3.2.0 which is incompatible.

After check the screens I detected:

Flask_AIL ERROR:

Flask_base_template.py  create_default_user.py    server.key
Flask_server.py         create_new_web_module.py  static
__pycache__             modules                   templates
blueprints              server.crt                update_thirdparty.sh
Traceback (most recent call last):
  File "./Flask_server.py", line 24, in <module>
    from pytaxonomies import Taxonomies
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/__init__.py", line 1, in <module>
    from .api import Taxonomies, Taxonomy, Predicate, Entry
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/api.py", line 13, in <module>
    import requests
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/__init__.py", line 147, in <module>
    from . import packages, utils
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/utils.py'

Script_AIL ERROR:

Flask_base_template.py  create_default_user.py    server.key
Flask_server.py         create_new_web_module.py  static
__pycache__             modules                   templates
blueprints              server.crt                update_thirdparty.sh
Traceback (most recent call last):
  File "./Flask_server.py", line 24, in <module>
    from pytaxonomies import Taxonomies
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/__init__.py", line 1, in <module>
    from .api import Taxonomies, Taxonomy, Predicate, Entry
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/api.py", line 13, in <module>
    import requests
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/__init__.py", line 147, in <module>
    from . import packages, utils
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/sugar/poll.py'

Redis_AIL info:

52383:M 10 Aug 2022 22:27:17.193 # Server initialized
52383:M 10 Aug 2022 22:27:17.193 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
52383:M 10 Aug 2022 22:27:17.193 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
52383:M 10 Aug 2022 22:27:17.193 * Ready to accept connections

Other screen without error messages !

PS: AIL Starting with ./LAUNCH.sh -l output without ERRORS:

    * Launching Redis servers
    * Launching ARDB servers
    * Launching logging process
    * Launching all the queues
    * Checking configuration

Config File: Nothing to update Config File: Nothing to update

Configuration up-to-date
Launching core scripts ...
Launching scripts
Launching Flask server

Thank you

lobojc commented 2 years ago

BTW the errors change every time when I run LAUNCH -k and LAUNCH -l

SOME STOP + START ->

Flask_base_template.py  create_new_web_module.py  static
Flask_server.py         modules                   templates
blueprints              server.crt                update_thirdparty.sh
create_default_user.py  server.key
Traceback (most recent call last):
  File "./Flask_server.py", line 16, in <module>
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/flask/__init__.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/flask/json/__init__.py'

Another STOP + START ->

Flask_base_template.py  create_new_web_module.py  static
Flask_server.py         modules                   templates
blueprints              server.crt                update_thirdparty.sh
create_default_user.py  server.key
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
  File "/usr/lib/python3.8/site.py", line 597, in <module>
    main()
  File "/usr/lib/python3.8/site.py", line 580, in main
    known_paths = venv(known_paths)
  File "/usr/lib/python3.8/site.py", line 498, in venv
    with open(virtual_conf, encoding='utf-8') as f:
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/pyvenv.cfg'

lobojc commented 2 years ago

And to finish ./LAUNCH -t command output:

ubuntu@ailframework:~/ailframework/bin$ ./LAUNCH.sh -t
EEE
======================================================================
ERROR: Failure: PermissionError ([Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/data/misp-galaxy/clusters/tea-matrix.json')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/ailframework/tests/testApi.py", line 17, in <module>
    from Flask_server import app
  File "/home/ubuntu/ailframework/var/www/Flask_server.py", line 160, in <module>
    importlib.import_module(name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/ubuntu/ailframework/var/www/modules/Tags/Flask_Tags.py", line 39, in <module>
    clusters = Clusters(skip_duplicates=True)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/api.py", line 294, in __init__
    with open(cluster_file, 'r') as f:
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/data/misp-galaxy/clusters/tea-matrix.json'
-------------------- >> begin captured logging << --------------------
pymisp: DEBUG: pymisp loaded properly
--------------------- >> end captured logging << ---------------------

======================================================================
ERROR: Failure: ImportError (libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/ailframework/tests/testHelper.py", line 9, in <module>
    from Helper import Process
  File "/home/ubuntu/ailframework/bin/Helper.py", line 17, in <module>
    import zmq
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
    from zmq import backend
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 31, in <module>
    raise original_error from None
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 26, in <module>
    _ns = select_backend(first)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/select.py", line 31, in select_backend
    mod = import_module(name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
    from . import (
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory

======================================================================
ERROR: Failure: ImportError (libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
    raise self.exc_val.with_traceback(self.tb)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
    module = self.importer.importFromPath(
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/usr/lib/python3.8/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.8/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 702, in _load
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 848, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ubuntu/ailframework/tests/test_modules.py", line 15, in <module>
    from modules.ApiKey import ApiKey
  File "/home/ubuntu/ailframework/bin/modules/ApiKey.py", line 21, in <module>
    from modules.abstract_module import AbstractModule
  File "/home/ubuntu/ailframework/bin/modules/abstract_module.py", line 17, in <module>
    from Helper import Process
  File "/home/ubuntu/ailframework/bin/Helper.py", line 17, in <module>
    import zmq
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
    from zmq import backend
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 31, in <module>
    raise original_error from None
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 26, in <module>
    _ns = select_backend(first)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/select.py", line 31, in select_backend
    mod = import_module(name)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
    from . import (
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory

Name                                   Stmts   Miss  Cover
----------------------------------------------------------
bin/BankAccount.py                        72     72     0%
bin/Crawler.py                           281    281     0%
bin/Cryptocurrencies.py                   78     78     0%
bin/Cve.py                                26     26     0%
bin/DumpValidOnion.py                     23     23     0%
bin/Duplicates.py                        116    116     0%
bin/Helper.py                            158    153     3%
bin/IPAddress.py                          50     50     0%
bin/Languages.py                          21     21     0%
bin/MISP_The_Hive_feeder.py              156    156     0%
bin/Mail.py                              135    135     0%
bin/Mixer.py                             114    114     0%
bin/ModuleStats.py                        80     80     0%
bin/ModulesInformationV2.py              549    549     0%
bin/NotificationHelper.py                 52     52     0%
bin/PgpDump.py                           164    164     0%
bin/QueueIn.py                            13     13     0%
bin/QueueOut.py                           14     14     0%
bin/Release.py                            46     46     0%
bin/SourceCode.py                         39     39     0%
bin/Tools.py                              62     62     0%
bin/Update-conf.py                        59     59     0%
bin/Update.py                            329    329     0%
bin/__init__.py                            0      0   100%
bin/ailleakObject.py                      90     90     0%
bin/core/ail_2_ail.py                    852    697    18%
bin/core/screen.py                       122    105    14%
bin/empty_queue.py                        18     18     0%
bin/export/AILObjects.py                  39     22    44%
bin/export/Export.py                      49     26    47%
bin/export/MispExport.py                 331    276    17%
bin/export/MispImport.py                 172    140    19%
bin/import/importer.py                    61     34    44%
bin/indexer_lookup.py                     60     60     0%
bin/launch_queues.py                      49     49     0%
bin/lib/ConfigLoader.py                   41     11    73%
bin/lib/Correlate_object.py              371    327    12%
bin/lib/Decoded.py                       260    210    19%
bin/lib/Domain.py                        573    463    19%
bin/lib/Investigations.py                294    219    26%
bin/lib/Language.py                       27     18    33%
bin/lib/Screenshot.py                    161    120    25%
bin/lib/Tracker.py                       944    799    15%
bin/lib/Username.py                       13      1    92%
bin/lib/btc_ail.py                        40     31    22%
bin/lib/crawlers.py                     1009    833    17%
bin/lib/d4.py                             58     39    33%
bin/lib/domain_basic.py                   16      5    69%
bin/lib/exceptions.py                      8      2    75%
bin/lib/item_basic.py                    161    117    27%
bin/lib/objects/CryptoCurrencies.py       34     19    44%
bin/lib/objects/Decodeds.py               40     20    50%
bin/lib/objects/Domains.py                33     17    48%
bin/lib/objects/Items.py                 426    319    25%
bin/lib/objects/Pgps.py                   30     15    50%
bin/lib/objects/Screenshots.py            24      8    67%
bin/lib/objects/Usernames.py              31     16    48%
bin/lib/objects/abstract_object.py        57     31    46%
bin/lib/objects/ail_objects.py            54     27    50%
bin/lib/queues_modules.py                 53     34    36%
bin/modules/ApiKey.py                     50     44    12%
bin/modules/Categ.py                      49     49     0%
bin/modules/Credential.py                111    111     0%
bin/modules/CreditCards.py                40     40     0%
bin/modules/Decoder.py                    99     99     0%
bin/modules/DomClassifier.py              51     51     0%
bin/modules/Global.py                    122    122     0%
bin/modules/Hosts.py                      31     31     0%
bin/modules/Indexer.py                    89     89     0%
bin/modules/Keys.py                      116    116     0%
bin/modules/LibInjection.py               57     57     0%
bin/modules/Onion.py                      61     61     0%
bin/modules/Phone.py                      40     40     0%
bin/modules/SQLInjectionDetection.py      48     48     0%
bin/modules/SentimentAnalysis.py          98     98     0%
bin/modules/Tags.py                       26     26     0%
bin/modules/Telegram.py                   53     53     0%
bin/modules/Urls.py                       40     40     0%
bin/modules/Zerobins.py                   30     30     0%
bin/modules/__init__.py                    0      0   100%
bin/modules/abstract_module.py            52     46    12%
bin/modules/submit_paste.py              189    189     0%
bin/packages/Correlation.py              266    213    20%
bin/packages/Cryptocurrency.py            68     49    28%
bin/packages/Date.py                     127    101    20%
bin/packages/Hash.py                      30     19    37%
bin/packages/HiddenServices.py           231    190    18%
bin/packages/Import_helper.py             56     43    23%
bin/packages/Item.py                     401    298    26%
bin/packages/Paste.py                    195    146    25%
bin/packages/Pgp.py                       17      5    71%
bin/packages/Tag.py                      483    396    18%
bin/packages/Term.py                     392    325    17%
bin/packages/User.py                      40     24    40%
bin/packages/__init__.py                   0      0   100%
bin/packages/git_status.py               110     95    14%
bin/packages/lib_refine.py                94     94     0%
bin/packages/lib_words.py                 51     51     0%
bin/template.py                           18     18     0%
bin/tor_fetcher.py                        46     46     0%
bin/update-background.py                  44     44     0%
----------------------------------------------------------
TOTAL                                  13359  11447    14%
----------------------------------------------------------------------
Ran 3 tests in 0.002s

FAILED (errors=3)

Thank you

CIRCL / AIL-framework

AIL Framework not starting #442

Redirecting...