timesketch importer can not index plaso file

[hasamba]

Describe the bug trying to add plaso timeline, the file is uploaded but cannot index [2023-09-21 11:03:02,418] timesketch_importer.importer_frontend/INFO Using cached credentials. <timesketch_importer/main> [2023-09-21 11:03:02,446] timesketch_importer.importer_frontend/INFO Creating a client. <timesketch_importer/main> [2023-09-21 11:03:02,738] timesketch_importer.importer_frontend/INFO Client created. <timesketch_importer/main> [2023-09-21 11:03:02,738] timesketch_importer.importer_frontend/INFO Saving TS config. <timesketch_importer/main> [2023-09-21 11:03:02,877] timesketch_importer.importer_frontend/INFO New sketch created: [19] lab <timesketch_importer/main> [2023-09-21 11:03:02,877] timesketch_importer.importer_frontend/INFO Uploading file. <timesketch_importer/main> [2023-09-21 11:03:02,880] timesketch_importer.importer_frontend/INFO About to upload file. <timesketch_importer/upload_file> [2023-09-21 11:03:21,480] timesketch_importer.importer_frontend/INFO File upload completed. <timesketch_importer/upload_file> Checking file upload status: .[FAIL] Unable to index timeline win10minimal, reason: fail

To Reproduce create plaso file with plaso version 20230717 for some reason pinfo report the created file as format version 20230327 running timesketch_importer -u **** -p *** --host http://localhost:81 --timeline_name demo --sketch_name lab --verbose /home/*/outputs/vuln.plaso gives the above error, same when importing the same file through web.

• tried about 3 different plaso files with same errors
• demo sigma_events.csv file works fine

[boingomw]

If this was a new install, check to see if the file plaso_formatters.yaml is in the right folder. the new deploy script puts it in the wrong place, and that causes a blank "fail" error.

curl -s $GITHUB_BASE_URL/data/plaso_formatters.yaml > timesketch/etc/plaso_formatters.yaml

Should be curl -s $GITHUB_BASE_URL/data/plaso_formatters.yaml > timesketch/etc/timesketch/plaso_formatters.yaml

[hasamba]

If this was a new install, check to see if the file plaso_formatters.yaml is in the right folder. the new deploy script puts it in the wrong place, and that causes a blank "fail" error.

curl -s $GITHUB_BASE_URL/data/plaso_formatters.yaml > timesketch/etc/plaso_formatters.yaml

Should be curl -s $GITHUB_BASE_URL/data/plaso_formatters.yaml > timesketch/etc/timesketch/plaso_formatters.yaml

indeed it is a fresh install, the file was at the wrong place as you said but after moving/redownload the file to the correct folder still getting the same error. now i noticed that when i upload the demo file sigma_events.csv through the gui, it keep on spinning and not getting processed

[jaegeral]

Plase check https://timesketch.org/guides/admin/troubleshooting/#issues-importing-plaso-file and provide all the infos, esp. around versions used to create the plaso files.

[hasamba]

Plase check https://timesketch.org/guides/admin/troubleshooting/#issues-importing-plaso-file and provide all the infos, esp. around versions used to create the plaso files.

I followed that troubleshooting before posting... as I originally mentioned : create plaso file with plaso version 20230717 for some reason pinfo report the created file as format version 20230327 timesketch plaso version is 20230717 i tried creating the plaso file through timesketch and also through it's own docker... same results

[jaegeral]

if pinfo says it was created with 20230327 chances are the file was created with that version. Also please provide the error message from the celery job and provide the output of: pinfo.py $FILENAME

[hasamba]

pinfo:

worker.log

[2023-09-26 08:49:45,332] py.warnings/WARNING /usr/local/lib/python3.10/dist-packages/celery/app/utils.py:204: CDeprecationWarning:
    The 'CELERY_RESULT_BACKEND' setting is deprecated and scheduled for removal in
    version 6.0.0. Use the result_backend instead

  deprecated.warn(description=f'The {setting!r} setting',

[2023-09-26 08:49:45,333] celery.apps.worker/WARNING Please run `celery upgrade settings path/to/settings.py` to avoid these warnings and to allow a smoother upgrade to Celery 6.0.
[2023-09-26 08:49:46,425] celery.worker.consumer.connection/INFO Connected to redis://redis:6379//
[2023-09-26 08:49:46,428] celery.worker.consumer.mingle/INFO mingle: searching for neighbors
[2023-09-26 08:49:47,436] celery.worker.consumer.mingle/INFO mingle: all alone
[2023-09-26 08:49:47,446] celery.apps.worker/INFO celery@8f5993b766ce ready.
[2023-09-26 08:52:11,370] celery.worker.strategy/INFO Task timesketch.lib.tasks.run_plaso[ebc72a77baa64544a87b186d99d36ae3] received
[2023-09-26 08:52:11,524] timesketch.tasks/INFO Index timeline [test] to index [7b0556a9f6f740d0b11f01bd8e9f0efd] (source: plaso)
[2023-09-26 08:52:12,792] celery.app.trace/ERROR Task timesketch.lib.tasks.run_plaso[ebc72a77baa64544a87b186d99d36ae3] raised unexpected: CalledProcessError(1, ['pinfo.py', '--output-format', 'json', '--sections', 'events', '/tmp/1dc8108edefa4f20818847a1d3a78083'])
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/timesketch/app.py", line 228, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/timesketch/lib/tasks.py", line 676, in run_plaso
    command = subprocess.run(cmd, capture_output=True, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pinfo.py', '--output-format', 'json', '--sections', 'events', '/tmp/1dc8108edefa4f20818847a1d3a78083']' returned non-zero exit status 1.

[jaegeral]

so as pinfo indicated, the file was created with a very old plaso version, highly recommend to create a new one.

For the Traceback, was there anything after that in the logs by any chance? Also which version of timesketch are you running?

[hasamba]

the psort version is the latest 20230717

the file that this (20230717) psort creates, shown as 20230327 with pinfo I tried both psort version from timesketch docker and independent log2timeline docker

the log i attached is the full log (i deleted the file before the latest run)

timesketch latest version

[jaegeral]

Can you run pinfo.py --output-format json --sections /tmp/1dc8108edefa4f20818847a1d3a78083 in the timesketch docker container please?

[hasamba]

pinfo.py --output-format json --sections /tmp/1dc8108edefa4f20818847a1d3a78083 gives missing argument

this is output when adding --sections sources:

[berggren]

pinfo.py --output-format json --sections /tmp/1dc8108edefa4f20818847a1d3a78083 gives missing argument

this is output when adding --sections sources:

it should be pinfo.py --output-format json --sections events /tmp/1dc8108edefa4f20818847a1d3a78083

[hasamba]

pinfo.zip

[berggren]

Thanks, I cannot look at that file right now. Did the tool successfully run, or did it crash?

[hasamba]

ran successfully

[jaegeral]

Plase check https://timesketch.org/guides/admin/troubleshooting/#issues-importing-plaso-file and provide all the infos, esp. around versions used to create the plaso files.

please explain further what you mean with +1. If you run into a similar issue, please follow: https://timesketch.org/guides/admin/troubleshooting/#issues-importing-plaso-file and provide all the infos, esp. around versions used to create the plaso files.

Otherwise a +1 does not add any value.

Thank you