Error loading bot queues information:

kodebrain commented 1 year ago

Installation completed using native package - Ubuntu 20.04 Followed - Installation

 Error loading bot queues information: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
webmaster@localhost to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.41 (Ubuntu) Server at 3<> Port 80</address>
</body></html>
Internal Server Error
Get an error error, Internal Server Error when trying to obtain config file properly

The default paths created during the installation are as below
:/etc/intelmq$ ls -la
total 68
drwxrwxr-x   3 intelmq intelmq  4096 Jan  4 10:47 .
drwxr-xr-x 101 root    root     4096 Jan  4 10:47 ..
-rw-rw-r--   1 intelmq intelmq   506 Sep  2  2021 api-apache.conf
-rw-rw-r--   1 intelmq intelmq   241 Sep  2  2021 api-config.json
-rw-rw-r--   1 intelmq intelmq   264 Sep  2  2021 api-sudoers.conf
-rw-rw-r--   1 intelmq intelmq 21097 Sep 10  2021 harmonization.conf
drwxrwxr-x   2 intelmq intelmq  4096 Jan  4 10:47 manager
-rw-rw-r--   1 intelmq intelmq   385 Sep  2  2021 manager-apache.conf
-rw-rw-r--   1 intelmq intelmq  6324 Jan  4 10:46 runtime.yaml
-rw-rw-r--   1 intelmq intelmq  6313 Jan  4 10:46 runtime.yaml.bak

:/var/log/intelmq$ ls -al
total 12
drwxr-xr-x  2 intelmq intelmq 4096 Jan  4 10:46 .
drwxrwxr-x 13 root    syslog  4096 Jan  4 10:47 ..
-rw-r--r--  1 intelmq intelmq 1045 Jan  4 11:04 intelmqctl.log

/var/lib/intelmq$ ls -al
total 24
drwxr-xr-x  3 intelmq intelmq 4096 Jan  4 11:11 .
drwxr-xr-x 42 root    root    4096 Jan  4 10:47 ..
-rw-------  1 intelmq intelmq  915 Jan  4 11:11 .viminfo
drwxrwxr-x  4 intelmq intelmq 4096 Jan  4 10:46 bots
-rw-rw-r--  1 intelmq intelmq 7125 Sep 10  2021 state.json

/usr/share/dbconfig-common/data/intelmq-api/install$ ls -la
total 12
drwxr-xr-x 2 root root 4096 Jan  4 10:47 .
drwxr-xr-x 3 root root 4096 Jan  4 10:47 ..
-rw-r--r-- 1 root root  429 Sep  2  2021 sqlite3

cat /etc/intelmq/api-config.json 
{
        "intelmq_ctl_cmd": ["sudo", "-u", "intelmq", "intelmqctl"],
        "allowed_path": "/opt/intelmq/var/lib/bots/",
        "session_store": "/var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi",
        "session_duration": 86400,
        "allow_origins": ["*"]
}

~$ intelmqctl check
Reading configuration files.
Checking runtime and pipeline configuration.
Checking harmonization configuration.
Checking for bots.
No issues found.

this is the name of your application's database.

dbc_dbname='/var/lib/dbconfig-common/sqlite3/intelmq-api' The above path doesn't exist on my installation

"/var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi"
:/etc/intelmq$ jq .session_store ${INTELMQ_API_CONFIG:-/etc/intelmq/api-config.json} | tr -d '"' | xargs ls -l
ls: cannot access '/var/lib/dbconfig-common/sqlite3/intelmq-api/intelmqapi': No such file or directory```

changed the path of session_store to `/usr/share/dbconfig-common/data/intelmq-api/install`

$ intelmq-api-adduser --user <> --password <>
Loading config from /etc/intelmq/api-config.json
Traceback (most recent call last):
  File "/usr/bin/intelmq-api-adduser", line 27, in <module>
    session_store = intelmq_api.session.SessionStore(str(api_config.session_store), api_config.session_duration)
  File "/usr/lib/python3/dist-packages/intelmq_api/session.py", line 86, in __init__
    self.init_sqlite_db()
  File "/usr/lib/python3/dist-packages/intelmq_api/session.py", line 100, in init_sqlite_db
    with self.connect() as con:
  File "/usr/lib/python3/dist-packages/intelmq_api/session.py", line 91, in connect
    return sqlite3.connect(self.dbname, check_same_thread=False,
sqlite3.OperationalError: unable to open database file

How to create the api-session.sqlite file?

sebix commented 1 year ago

The user needs read and write access to the database and the directory containing it. Is this the case?

kodebrain commented 1 year ago

I had to create the db manually using sudo api-session.sqlite based on the config in /etc/intelmq/api-config.json /usr/share/dbconfig-common/data/intelmq-api/install/sqlite3 write permission set for user and group for api-session.sqlite

After logging back into the portal, getting more errors 9:07 AM 341× Error loading bot queues information: [31mNot logging to file: [Errno 13] Permission denied: '/var/log/intelmq/intelmqctl.log'[0m sudo -u www-data intelmqctl --type json list queues-and-status

8:59 AM Saved positions are not valid or not complete. The configuration has possibly been modified outside of the IntelMQ-Manager.

-rw-r--r-- 1 intelmq intelmq 1245 Feb 1 08:58 intelmqctl.log

all errors disappeared after setting the permission to 774 -rwxrwxr-- 1 intelmq intelmq 1245 Feb 1 08:58 intelmqctl.log

To import custom bots, is the below the correct location to install it?

/usr/lib/python3/dist-packages/intelmq/bots$ ls -al total 28 drwxr-xr-x 7 root root 4096 Jan 4 10:46 . drwxr-xr-x 8 root root 4096 Jan 4 10:46 .. -rw-r--r-- 1 root root 0 Sep 10 2021 init.py drwxr-xr-x 2 root root 4096 Jan 4 10:46 pycache drwxr-xr-x 24 root root 4096 Jan 4 10:46 collectors drwxr-xr-x 40 root root 4096 Jan 4 10:46 experts drwxr-xr-x 22 root root 4096 Jan 4 10:46 outputs drwxr-xr-x 52 root root 4096 Jan 4 10:46 parsers

sebix commented 1 year ago

To import custom bots, is the below the correct location to install it?

Yes, but if you want a development installation, I recommend reading the dev guide: https://intelmq.readthedocs.io/en/maintenance/dev/guide.html#installation Using a local repository makes development easier

certtools / intelmq-manager