redpanda-data / connect

Fancy stream processing made operationally mundane
https://docs.redpanda.com/redpanda-connect/about/
8.09k stars 817 forks source link

Format timestamp into a duration string #2185

Open mpartipilo opened 11 months ago

mpartipilo commented 11 months ago

I'm looking for a method to parse timestamps into a duration string. In short, basically the opposite of parse_duration.

root.delay_for = this.delay_for_ns.ts_duration()

# In: {"delay_for_ns":50000}
# Out:  {"delay_for":"50us"}
ZiyanK commented 9 months ago

Hello. For this method, I was thinking of taking in the output parameters as a variable in this format: root.delay_for = this.delay_for_ns.ts_duration(format: "us")

This way we can provide the output in whichever format is preferred by the user. What do you think? @mihaitodor

mihaitodor commented 9 months ago

That would differ from the builtin Golang Duration.String() method so I'm not sure what people in general would expect to be able to provide as input to that format parameter, which could open a big can of worms...

Note that for simple cases such as us, it's probably easier to just do the arithmetic in bloblang.

mpartipilo commented 9 months ago

I never thought that formatting would be an issue.

I was thinking of just reflecting what Go does without many moving parts, just for pretty printing.

Any more complex scenario could be handled like @mihaitodor suggests.

ZiyanK commented 9 months ago

But how would you know which format the user would want the output in? Unless we are just going to do it for us format? @mihaitodor

mihaitodor commented 9 months ago

That's why I suggested to do what Go does:

String returns a string representing the duration in the form "3d1h3m". Leading zero units are omitted. As a special case, durations less than one second format use a smaller unit (milli-, micro-, or nanoseconds) to ensure that the leading digit is non-zero. Duration more than a day or more than a week lose granularity and are truncated to resp. days-hours-minutes and weeks-days-hours. The zero duration formats as 0s.

Not sure if that's what most users expect / need, so it might be worth thinking of other options. Maybe it's ISO-8601 is another option, since we already import https://pkg.go.dev/github.com/rickb777/date/period. I'm trying to avoid having to come up with some new standard.

ZiyanK commented 9 months ago

I guess the Golang's Duration.String() makes most sense then. You're right. Can I work on this implementation?

mihaitodor commented 9 months ago

@ZiyanK sure! Please do.