Open hasamba opened 4 months ago
On the same machine where this error happens, can you upload the Plaso file using the web UI or dftimewolf? Does it result in the same errors?
Are there any related errors on the server side in the timesketch/logs/wsgi_errors.log or worker.log files?
wsgi_errors.log
forgot to mention that we tried upload the same plaso file through the UI and it worked.
could not find wsgi_errors.log
worker.log last 20 lines:
[2024-07-14 08:57:51,429] celery.apps.worker/WARNING Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.
[2024-07-14 08:57:53,361] celery.worker.consumer.connection/INFO Connected to redis://redis:6379//
[2024-07-14 08:57:53,384] celery.worker.consumer.mingle/INFO mingle: searching for neighbors
[2024-07-14 08:57:54,411] celery.worker.consumer.mingle/INFO mingle: all alone
[2024-07-14 08:57:54,445] celery.apps.worker/INFO celery@b16ba86fb63f ready.
[2024-07-14 12:08:10,220] py.warnings/WARNING /usr/local/lib/python3.10/dist-packages/celery/app/utils.py:204: CDeprecationWarning:
The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
version 6.0.0. Use the result_backend instead
deprecated.warn(description=f'The {setting!r} setting',
[2024-07-14 12:08:10,221] celery.apps.worker/WARNING Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.
[2024-07-14 12:08:11,161] celery.worker.consumer.connection/INFO Connected to redis://redis:6379//
[2024-07-14 12:08:11,165] celery.worker.consumer.mingle/INFO mingle: searching for neighbors
[2024-07-14 12:08:12,181] celery.worker.consumer.mingle/INFO mingle: all alone
[2024-07-14 12:08:12,197] celery.apps.worker/INFO celery@b16ba86fb63f ready.
[2024-07-15 07:33:08,227] celery.worker.strategy/INFO Task timesketch.lib.tasks.run_plaso[ce70e9e801c947f085e90f77b8b80e9d] received
[2024-07-15 07:33:08,483] timesketch.tasks/INFO Index timeline [artifacts] to index [d1762398f7804957a38d8349c91346e8] (source: plaso)
[2024-07-15 07:33:10,580] celery.app.trace/INFO Task timesketch.lib.tasks.run_plaso[ce70e9e801c947f085e90f77b8b80e9d] succeeded in 2.3513273459975608s: 'd1762398f7804957a38d8349c91346e8'
If you have a docker deployment, the wsgi_error.log
should be in the same folder as the worker.log
(e.g. /opt/timesketch/logs/wsgi_error.log
).
Worker log looks good.
response.json
or response.text
. return io.open(fd, mode, buffering, encoding, *args, **kwargs)
[2024-07-14 12:08:04 +0000] [8] [INFO] Booting worker with pid: 8
[2024-07-14 12:08:04 +0000] [9] [INFO] Booting worker with pid: 9
[2024-07-14 12:08:05 +0000] [10] [INFO] Booting worker with pid: 10
[2024-07-14 12:08:05 +0000] [11] [INFO] Booting worker with pid: 11
[2024-07-14 12:08:06 +0000] [7] [INFO] Starting gunicorn 19.10.0
[2024-07-14 12:08:06 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
[2024-07-14 12:08:06 +0000] [7] [INFO] Using worker: sync
/usr/lib/python3.10/os.py:1030: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used
return io.open(fd, mode, buffering, encoding, *args, **kwargs)
[2024-07-14 12:08:06 +0000] [8] [INFO] Booting worker with pid: 8
[2024-07-14 12:08:06 +0000] [9] [INFO] Booting worker with pid: 9
[2024-07-14 12:08:06 +0000] [10] [INFO] Booting worker with pid: 10
[2024-07-14 12:08:06 +0000] [11] [INFO] Booting worker with pid: 11
[2024-07-15 07:17:33,750] flask_wtf.csrf/INFO The CSRF token is missing.
[2024-07-15 07:19:17,789] flask_wtf.csrf/INFO The CSRF token is missing.
[2024-07-15 07:19:49,444] flask_wtf.csrf/INFO The CSRF token is missing.
[2024-07-15 07:20:58,221] flask_wtf.csrf/INFO The CSRF token is missing.
[2024-07-15 07:21:26,652] flask_wtf.csrf/INFO The CSRF token is missing.
[2024-07-15 07:33:02,308] timesketch.analyzers.hashlookup/ERROR Hashlookup conf not found
<!DOCTYPE html><html lang=en><head><meta name=csrf-token content="ImQwMzM1ODEwNzgyZDc4Mjk1NmQ4MDA2NzA4ODA0YWFiYWIwZDk5ZjEi.ZpYkzA.txxqdZ5gv_5mD1uekpjcQkO--yA"><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><link rel=icon href=/legacy/dist/favicon.ico><title>Timesketch</title><style>html,
body {
display: flex;
height: 100%;
width: 100%;
align-items: center;
justify-content: center;
}
.card.is-wide {
width: 520px;
height: 350px;
padding-top:30px;
}
.card.has-text-centered,
.card-content {
justify-content: center;
align-items: center;
}</style><link href=/legacy/dist/css/chunk-common.673b3577.css rel=preload as=style><link href=/legacy/dist/css/chunk-vendors.96a655cf.css rel=preload as=style><link href=/legacy/dist/js/chunk-common.d12c964e.js rel=preload as=script><link href=/legacy/dist/js/chunk-vendors.91a8ae5d.js rel=preload as=script><link href=/legacy/dist/js/login.10bd00b4.js rel=preload as=script><link href=/legacy/dist/css/chunk-vendors.96a655cf.css rel=stylesheet><link href=/legacy/dist/css/chunk-common.673b3577.css rel=stylesheet></head><body class=login-page><div class=columns><div class=column><div class="card is-wide has-text-centered"><div class=card-content><div class=content><div><img src=/legacy/dist/timesketch-color.png style=width:40px;><div style="font-size: 1.5em;">time<b>sketch</b></div><div style="font-size: 0.8em;">Digital Forensic Timeline Analysis</div></div><br><form method=post style="width:50%;margin-left: auto; margin-right: auto;"><div class=field><div class=control><input type=text class=input name=username placeholder=Username style=text-align:center;></div></div><div class=field><div class=control><input type=password class=input name=password placeholder=Password style=text-align:center;></div></div><div class=field><div class="control has-text-centered"><button type=submit class="button is-rounded is-wide is-info" style=width:100%;>Sign in</button></div></div><input id="csrf_token" name="csrf_token" type="hidden" value="ImQwMzM1ODEwNzgyZDc4Mjk1NmQ4MDA2NzA4ODA0YWFiYWIwZDk5ZjEi.ZpYkzA.txxqdZ5gv_5mD1uekpjcQkO--yA"></form></div></div></div></div></div><script src=/legacy/dist/js/chunk-vendors.91a8ae5d.js></script><script src=/legacy/dist/js/chunk-common.d12c964e.js></script><script src=/legacy/dist/js/login.10bd00b4.js></script></body></html>
The API response you have sheared is actually the code of the login page. It points to an issue with the authentication.
Can you please remove the ~/.timesketchrc
and ~/.timesketch.token
files on your system and try again? (You will be asked to re-authenticate to Timesketch)
Alternatively you can also try to pass the authentication in the cli: timesketch_importer.py --host <URL> -u <USERNAME> -p <PASSWORD> <PATH>
we always send the full syntax with host and user/pwd because we are planning to run it inside a script. anyway i deleted the files, same results.
can it be related to the fact that we are trying to run it in a virtual environment? no python is installed on the host, only portable python inside the venv
something very strange, it didnt work for something like a week, and we tried alot of times each day. now i tried with on the same machine with the same plaso file and it working. nothing was changed (except the date/time) or installed
timesketch_importer --host http://localhost:5000 -u import -p import /tmp/artifacts.plaso
[2024-07-22 10:53:00,127] timesketch_importer.importer_frontend/INFO Saving Credentials.
[2024-07-22 10:53:00,246] timesketch_importer.importer_frontend/INFO Creating a client.
[2024-07-22 10:53:00,616] timesketch_importer.importer_frontend/INFO Client created.
[2024-07-22 10:53:00,616] timesketch_importer.importer_frontend/INFO Saving TS config.
timesketch_importer [response]:{"meta":{},"objects":[{"active_timelines":[],"all_permissions":"{\"user/import\": [\"read\", \"write\", \"delete\"], \"is_public\": false}","created_at":"2024-07-22T10:53:00.621622","description":"New Sketch From Importer CLI","graphs":[],"id":2,"label_string":"","my_permissions":"[\"read\", \"write\", \"delete\"]","name":"New Sketch From Importer CLI","status":[{"created_at":"2024-07-22T10:53:00.621622","id":2,"status":"new","updated_at":"2024-07-22T10:53:00.621622"}],"timelines":[],"updated_at":"2024-07-22T10:53:00.621622","user":{"active":true,"admin":false,"email":null,"groups":[],"id":2,"name":"import","profile":{"picture_filename":null,"picture_url":null},"username":"import"}}]}
[2024-07-22 10:53:00,750] timesketch_importer.importer_frontend/INFO New sketch created: [2] New Sketch From Importer CLI
What is the timeline name [artifacts]:
[2024-07-22 10:53:10,827] timesketch_importer.importer_frontend/INFO Uploading file.
[2024-07-22 10:53:10,836] timesketch_importer.importer_frontend/INFO About to upload file.
[2024-07-22 10:53:27,227] timesketch_importer.importer_frontend/INFO File upload completed.
i think i found the bug, seems like there is a problem with my 'admin' user and also the user in the command line (-u) is not taking into consideration if the timesketch.token/rc is already available :
timesketch_importer --host http://localhost:5000 -u admin -p admin /tmp/artifacts.plaso
[2024-07-24 06:42:00,436] timesketch_api.config_assistance/WARNING No timesketch section in the config
[2024-07-24 06:42:00,436] timesketch_importer.importer_frontend/INFO Saving Credentials.
[2024-07-24 06:42:00,553] timesketch_importer.importer_frontend/INFO Creating a client.
[2024-07-24 06:42:00,885] timesketch_importer.importer_frontend/INFO Client created.
[2024-07-24 06:42:00,886] timesketch_importer.importer_frontend/INFO Saving TS config.
timesketch_importer [response]:<!DOCTYPE html><html lang=en><head><meta name=csrf-token content="ImExYTg5MzBmMWUzM2QwYTAyMWYyNWU2MjM5ZjU3NGI4N2IyNzg1NDUi.ZqCiOA.YCkbe1Cd_5aFVbehfhN7nCbXTlU"><meta charset=utf-8><meta http-equiv=X-UA-Compatible content="IE=edge"><meta name=viewport content="width=device-width,initial-scale=1"><link rel=icon href=/legacy/dist/favicon.ico><title>Timesketch</title><style>html,
body {
display: flex;
height: 100%;
width: 100%;
align-items: center;
justify-content: center;
}
.card.is-wide {
width: 520px;
height: 350px;
padding-top:30px;
}
.card.has-text-centered,
.card-content {
justify-content: center;
align-items: center;
}</style><link href=/legacy/dist/css/chunk-common.673b3577.css rel=preload as=style><link href=/legacy/dist/css/chunk-vendors.96a655cf.css rel=preload as=style><link href=/legacy/dist/js/chunk-common.d12c964e.js rel=preload as=script><link href=/legacy/dist/js/chunk-vendors.91a8ae5d.js rel=preload as=script><link href=/legacy/dist/js/login.10bd00b4.js rel=preload as=script><link href=/legacy/dist/css/chunk-vendors.96a655cf.css rel=stylesheet><link href=/legacy/dist/css/chunk-common.673b3577.css rel=stylesheet></head><body class=login-page><div class=columns><div class=column><div class="card is-wide has-text-centered"><div class=card-content><div class=content><div><img src=/legacy/dist/timesketch-color.png style=width:40px;><div style="font-size: 1.5em;">time<b>sketch</b></div><div style="font-size: 0.8em;">Digital Forensic Timeline Analysis</div></div><br><form method=post style="width:50%;margin-left: auto; margin-right: auto;"><div class=field><div class=control><input type=text class=input name=username placeholder=Username style=text-align:center;></div></div><div class=field><div class=control><input type=password class=input name=password placeholder=Password style=text-align:center;></div></div><div class=field><div class="control has-text-centered"><button type=submit class="button is-rounded is-wide is-info" style=width:100%;>Sign in</button></div></div><input id="csrf_token" name="csrf_token" type="hidden" value="ImExYTg5MzBmMWUzM2QwYTAyMWYyNWU2MjM5ZjU3NGI4N2IyNzg1NDUi.ZqCiOA.YCkbe1Cd_5aFVbehfhN7nCbXTlU"></form></div></div></div></div></div><script src=/legacy/dist/js/chunk-vendors.91a8ae5d.js></script><script src=/legacy/dist/js/chunk-common.d12c964e.js></script><script src=/legacy/dist/js/login.10bd00b4.js></script></body></html>
[2024-07-24 06:42:00,895] timesketch_api.client/WARNING Unable to json decode the Timesketch API response!
Traceback (most recent call last):
File "/home/Bacteria5570/mssp/risx-mssp-python-script/mssp_env/lib/python3.10/site-packages/requests/models.py", line 974, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/Bacteria5570/mssp/risx-mssp-python-script/mssp_env/lib/python3.10/site-packages/timesketch_api_client/error.py", line 97, in get_response_json return response.json() File "/home/Bacteria5570/mssp/risx-mssp-python-script/mssp_env/lib/python3.10/site-packages/requests/models.py", line 978, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/Bacteria5570/mssp/risx-mssp-python-script/mssp_env/bin/timesketch_importer", line 8, in
3. running with different user (import) failed
timesketch_importer --host http://localhost:5000 -u import -p import /tmp/artifacts.plaso [2024-07-24 06:44:26,632] timesketch_importer.importer_frontend/INFO Using cached credentials. [2024-07-24 06:44:26,691] timesketch_importer.importer_frontend/INFO Creating a client. [2024-07-24 06:44:27,024] timesketch_importer.importer_frontend/INFO Client created. [2024-07-24 06:44:27,024] timesketch_importer.importer_frontend/INFO Saving TS config. timesketch_importer [response]:<!DOCTYPE html>
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/home/Bacteria5570/mssp/risx-mssp-python-script/mssp_env/lib/python3.10/site-packages/timesketch_api_client/error.py", line 97, in get_response_json return response.json() File "/home/Bacteria5570/mssp/risx-mssp-python-script/mssp_env/lib/python3.10/site-packages/requests/models.py", line 978, in json raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/Bacteria5570/mssp/risx-mssp-python-script/mssp_env/bin/timesketch_importer", line 8, in
4. delete both .timesketch files
5. running again with import user success
timesketch_importer --host http://localhost:5000 -u import -p import /tmp/artifacts.plaso [2024-07-24 06:45:32,316] timesketch_api.config_assistance/WARNING No timesketch section in the config [2024-07-24 06:45:32,317] timesketch_importer.importer_frontend/INFO Saving Credentials. [2024-07-24 06:45:32,435] timesketch_importer.importer_frontend/INFO Creating a client. [2024-07-24 06:45:32,777] timesketch_importer.importer_frontend/INFO Client created. [2024-07-24 06:45:32,777] timesketch_importer.importer_frontend/INFO Saving TS config. timesketch_importer [response]:{"meta":{},"objects":[{"active_timelines":[],"all_permissions":"{\"user/import\": [\"read\", \"write\", \"delete\"], \"is_public\": false}","created_at":"2024-07-24T06:45:32.780496","description":"New Sketch From Importer CLI","graphs":[],"id":4,"label_string":"","my_permissions":"[\"read\", \"write\", \"delete\"]","name":"New Sketch From Importer CLI","status":[{"created_at":"2024-07-24T06:45:32.780496","id":4,"status":"new","updated_at":"2024-07-24T06:45:32.780496"}],"timelines":[],"updated_at":"2024-07-24T06:45:32.780496","user":{"active":true,"admin":false,"email":null,"groups":[],"id":2,"name":"import","profile":{"picture_filename":null,"picture_url":null},"username":"import"}}]}
[2024-07-24 06:45:32,898] timesketch_importer.importer_frontend/INFO New sketch created: [4] New Sketch From Importer CLI
6. running again with user admin successed but it seems it runs as import
timesketch_importer --host http://localhost:5000 -u admin -p admin /tmp/artifacts.plaso [2024-07-24 06:46:13,784] timesketch_importer.importer_frontend/INFO Using cached credentials. [2024-07-24 06:46:13,843] timesketch_importer.importer_frontend/INFO Creating a client. [2024-07-24 06:46:14,183] timesketch_importer.importer_frontend/INFO Client created. [2024-07-24 06:46:14,183] timesketch_importer.importer_frontend/INFO Saving TS config. timesketch_importer [response]:{"meta":{},"objects":[{"active_timelines":[],"all_permissions":"{\"user/import\": [\"read\", \"write\", \"delete\"], \"is_public\": false}","created_at":"2024-07-24T06:46:14.186401","description":"New Sketch From Importer CLI","graphs":[],"id":5,"label_string":"","my_permissions":"[\"read\", \"write\", \"delete\"]","name":"New Sketch From Importer CLI","status":[{"created_at":"2024-07-24T06:46:14.186401","id":5,"status":"new","updated_at":"2024-07-24T06:46:14.186401"}],"timelines":[],"updated_at":"2024-07-24T06:46:14.186401","user":{"active":true,"admin":false,"email":null,"groups":[],"id":2,"name":"import","profile":{"picture_filename":null,"picture_url":null},"username":"import"}}]}
[2024-07-24 06:46:14,281] timesketch_importer.importer_frontend/INFO New sketch created: [5] New Sketch From Importer CLI
Thanks for going the extra mile to debug this @hasamba . We will have to take a look at how the script handles authentication and how to permanently fix this. I guess, if the login is provided via parameters, the stored creds should be ignored.
while importing a plaso file, getting the following error:
Timesketch version: 20240508.1 API Client Version: 20240215 Importer Client Version: 20230721
Importing the same plaso file on another machine (both ubuntu server 22.04+python VENV 3.10.12) with same timesketch+timesketch_importer versions works!
what can be the problem? thanks