Add a Timesketch output module that uses the Timesketch API

[kiddinn]

Description of feature request:

To create a new output module in plaso that can be used to import data directly into Timesketch. This output module would be named something like timesketch_api or just timesketch and would be available if the user has the timesketch_api_client and timesketch-import-client installed.

The output module would differ from the current TS module in the sense that it would not use Elastic directly, but rather depend on the importer client to send data over to TS. That would mean that you can add data to Timesketch from any machine that can reach the Timesketch endpoint over HTTPS, since it utilises the REST API instead of direct database access.

[kiddinn]

I won't have time to work on this right now, so I'm creating the feature request if anyone has time

[mpilking]

This feature would be very useful for large plaso databases. In particular, it would be super useful if we could upload a subset of events to Timesketch based on a psort filter for time range, parser types, etc. I have worked with some very large plaso files and this would provide a big optimization in those situations.

[joachimmetz]

@mpilking I don't understand this comment.

@kiddinn I recall you indicated that the elastic_ts approach was more efficient for bulk inserts. Why would going through the API (an additional layer of code) be "very useful for large plaso databases" ?

In particular, it would be super useful if we could upload a subset of events to Timesketch based on a psort filter for time range, parser types, etc. I have worked with some very large plaso files and this would provide a big optimization in those situations.

Can't you do this with elastic_ts ? Isn't it even more effective not to process things you don't want to analyze in the first place?

[kiddinn]

elastic_ts is more efficient for large bulk inserts, however that is only possible from the backend, so it only works if you've got access to the ES database. At that point psort filters don't work, we could add support for additional filtering or parameters into the API, but having the ability to run psort and directly import into TS from any machine that's able to communicate to the rest API is also very useful to have.

In terms of large plaso files, it provides the option to apply psort filters among other things to limit what actually gets sent over. It also allows other psort parameters to be processed, which are not available inelastic_ts which only uses a fixed way of running psort.

I think both options should be made available.

[joachimmetz]

to the rest API is also very useful to have.

agree, but that is a different issue than indicated by @mpilking

which are not available inelastic_ts which only uses a fixed way of running psort.

Doesn't Timesketch importer just invoke psort with elastic_ts directly? with some TS specific logic around it? Isn't the limitation here that Timesketch importer does not support passing event filters?

[kiddinn]

Yes but that's another issue that should be filed under timesketch, that is the ability to accept filters (wouldn't be an option in the current UI but can be added to the importer)

[joachimmetz]

@kiddinn thx for confirming, that what I thought as well.

@mpilking can you file an FR with Timesketch, sounds like your comment does not apply to the issue Kristinn outlined. I also strongly recommend explaining your use case / workflow, since as I have indicated before, these are not transparent to us.

[mpilking]

Ok, I will write up a thorough feature request for Timesketch.

[joachimmetz]

Blocked on Timesketch