search

elastic / elastic-integration-corpus-generator-tool

Command line tool used for generating events corpus dynamically given a specific integration
Other
21 stars 12 forks source link

Range from to for date type #117

Closed aspacca closed 9 months ago

aspacca commented 9 months ago

We want to be able to generate date fields between a range. We introduce range.from and range.to config settings that will be considered only for date fields. It is possible to specify both from and to or only one of them. If both are specified, the dates will be generate between from and to If only one is specified, the dates will be generated in the interval given by either from or to and time.Now() In any case the dates generated will keep progressive order, regardless the interval is negative of positive.

gizas commented 9 months ago

@aspacca thanks for this. Really helpful

I have made this draft branch to test here

Some minor comments:

aspacca commented 9 months ago
  • The from/to dates must have this format from: "2023-05-15T00:00:00.999999999-00:01" otherwise we have parse error Error: parsing time "2023-05-15" as "2006-01-02T15:04:05.999999999-07:00": cannot parse "" as "T" accessing 'fields.15.range.from'. I dont mind, but please change the documentation example It can be misleading for someone that starts creating a template

👍

  • In order to make the date compatible with what we have in ES , I had to format it again like here: Mind that in your format is 999999999-00:01 and in previous templates we had 999999Z07+00:00. So we revert - with +. Do you think shall we also change this?

parsing format and formatting format should be necessarily the same and might differ a lot: see an example from nginx.

I would not change

  • Last one (minor) See a sample of results:
2023-05-15T00:00:00.999999-00+00:00"
"2023-05-15T01:12:00.974999-00+00:00"
"2023-05-15T02:24:00.949999-00+00:00"
"2023-05-15T03:36:00.924999-00+00:00"
"2023-05-15T04:48:00.899999-00+00:00"
"2023-05-15T06:00:00.874999-00+00:00"
"2023-05-15T07:12:00.849999-00+00:00"

They are sorted (really cool !!!), only thing is that last part is always -00+00:00. Maybe we dont need such accuracy but worth to mention it

I guess it comes from https://github.com/elastic/elastic-integration-corpus-generator-tool/blob/timedateperiod/assets/templates/kubernetes.pod/gotext.tpl#L3, does it?

The format is somehow wrong: https://go.dev/play/p/8HZtVYOe6t0

gizas commented 9 months ago

Thanks Andrew.

So I think you can skip the micro seconds for now.

I would only suggest the documentation update for the correct timestamp and also to update this doc with a reminder/mention to an example for the format.

aspacca commented 9 months ago

@gizas docs fixed!

gizas commented 9 months ago

Andrew still the eg. fails for parsing Error: parsing time "2023-11-23T11:29:48" as "2006-01-02T15:04:05.999999999-07:00": cannot parse "" as "-07:00" accessing 'fields.15.range.from'

Please can you update the example https://github.com/elastic/elastic-integration-corpus-generator-tool/pull/117/files#diff-3e822219875fd3448087df4174dd709f82eb3b229fc3a64cddcea005e4eb2f2dR32-R33

aspacca commented 9 months ago

Andrew still the eg. fails for parsing

I should have tested it! 😭

aspacca commented 9 months ago

@gizas https://go.dev/play/p/V5COIdqSILz