Too many values to unpack in timesketch_importer.py

[ddnomad]

Describe the bug timesketch_importer.py crashes with ValueError: too many values to unpack (expected 2) if file to upload does not end with json or csv. The same will happen if upload_file function fails and returns an error string.

To Reproduce Steps to reproduce the behavior: 1, Run timesketch instance locally on port 8080 (just in case)

2. Create a file in a current directory: echo '{}' > myfile.json
3. Run timesketch_importer --host http://127.0.0.1:8080 -u admin -p 'my_sweet_password' --timeline-name my_timeline myfile.json

Expected behavior Script terminates with a clear error message stating the problem.

Screenshots Traceback instead:

Traceback (most recent call last):
  File "/usr/local/bin/timesketch_importer", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/tools/timesketch_importer.py", line 432, in main
    my_sketch=my_sketch, config_dict=config_dict, file_path=options.path)
ValueError: too many values to unpack (expected 2)

Desktop (please complete the following information):

• OS: Arch Linux
• Browser: Mosaic
• Version: Importer Client Version: 20210305

Additional context Offending line in the source: https://github.com/google/timesketch/blob/master/importer_client/python/tools/timesketch_importer.py#L450

[berggren]

Thanks for the report - I have assigned @kiddinn to take a look at this one.

[splunk-user1]

I'm facing the same issue ... is there a workaround ? Thanks

Tried sample jsonl on https://timesketch.org/learn/create-timeline-from-json-csv/, no go either.

Also if it helps, same json file upload using gui throws Internal Server Error

PS: CSV upload using either CLI or GUI works as expected.

[mpilking]

I'm facing the exact same error using jsonl via timesketch_importer. And as @splunk-user1 reported, I also get Internal Server Error when using the web UI to upload. However, he reported that the CSV works, but that does not work for me. Well, it sort of works, but it drops more than 10% of the events in the resulting index.

(Searching the web for "too many values to unpack" shows a lot of hits for this issue for Python.)

[jaegeral]

As we have not seen recent reports, I am going to move this to Q3.

[jpvlsmv]

This is still happening (March 2024, version timesketch-import-client 20230721). I had psteal write its dynamic format to output.txt, and got the same ValueError: too many values to unpack (expected 2).

When I renamed the file to output.csv, it imported successfully.

[jpvlsmv]

Ah, I see the error (message) clearly now. timesketch_importer.py line 625 expects two values to be returned from the upload_file function. The type attributes of upload_file show it returning only one string.

In this case, since file_path does not end in plaso, csv, or json, we end up in the body of line 84 returning a single formatted string error message.

In contrast the successful return value (L150) returns both a string (timelline) and the task_id, as upload_file expects it to.

The (correct) fix would be to indicate that the upload_file returns two values as upload_function expects, and when an error occurs to propagate that error information another way:

class CustomException(Exception):
  def __init__(self, message):
    self._message=message
    super().__init__(message)

... (in upload_file)

    if not my_sketch or not hasattr(my_sketch, "id"):
        raise CustomException("Sketch ID needs to be set")

... in main

    try:
        timeline, task_id = upload_file(
            my_sketch=my_sketch, config_dict=config_dict, file_path=options.path
        )
    except CustomException as e:
        print("Could not upload -")
        print(e.message)
        exit(1)