Add support for custom timestamp storage

[georgiosd]

I have a very efficient Timestamp class that I use for my time series project and do a whole load of timezone conversions around that.

Is there any way to make your library aware of custom InfluxTimestamp types?

Also it looks like the dates are deserialized back in Utc instead of Unknown. Is there any way to configure that too?

[MikaelGRA]

In regards to timezones, I was recently looking a bit into that sort of functionality as you can see here:

https://github.com/MikaelGRA/InfluxDB.Client/blob/master/src/Vibrant.InfluxDB.Client/InfluxQueryOptions.cs#L57

The idea would be if a timezone is specified in the query (fx. tz('America/New York')), it would allow simply keeping the timestamps without any conversion and using the DateTimeKind.Unspecified.

When I parse timestamps I use the DateTime.Parse method with the following DateTimeStyles (for UTC):

private static readonly DateTimeStyles OnlyUtcStyles = DateTimeStyles.AdjustToUniversal | DateTimeStyles.AssumeUniversal;

The problem that I faced (as far as I can remember), was that if you do not "AdjustToUniversal" then it will do the opposite and "AdjustToLocal (this enum value does not actually exist)" rather than simply NOT making a conversion and using the DateTimeKind.Unspecified.

Perhaps such an approach might still be possible if the CurrentCulture was changed during the time of the conversion, although I am not sure.

As for custom DateTimes, maybe this could be solved using some sort of a ITimestampParser in the InfluxQueryOptions. I will probably look into some potential solutions to this, but I have a feeling this could cause some breaking changes.

[MikaelGRA]

So, I managed to implement support for both custom timestamp field types and preserving the offsets when deserializing the time column.

By default the following timestamp column types are supported: DateTime, DateTime?, DateTimeOffset, DateTimeOffset?.

When using DateTimeOffset and DateTimeOffset? the offset is preserved in the returned instance.

A DateTime can never be used to retrieve local datetimes, as I am sure you are aware, converting from a local datetime + timezone to UTC can be a very troubling effort due to daylight savings.

Now let's get into the meat of the matter. I've also implemented support for custom timestamp fields. I've added a TimestampParserRegistry to the InfluxClient with the following interface:

   /// <summary>
   /// TimestampParserRegistry allowing for looking up a parser for a specific timestamp type.
   /// </summary>
   public interface ITimestampParserRegistry
   {
      /// <summary>
      /// Finds the timestamp parser for the specified timestamp type.
      /// </summary>
      /// <typeparam name="TTimestamp"></typeparam>
      /// <returns></returns>
      ITimestampParser<TTimestamp> FindTimestampParser<TTimestamp>();

      /// <summary>
      /// Adds or replaces the timestamp parser for the specified timestamp type.
      /// </summary>
      /// <typeparam name="TTimestamp"></typeparam>
      /// <typeparam name="TTimestampParser"></typeparam>
      /// <param name="timestampParser"></param>
      void AddOrReplace<TTimestamp, TTimestampParser>( TTimestampParser timestampParser ) where TTimestampParser : ITimestampParser<TTimestamp>;

      /// <summary>
      /// Removes the timestamp parser for the specified timestamp type.
      /// </summary>
      /// <typeparam name="TTimestamp"></typeparam>
      void Remove<TTimestamp>();

      /// <summary>
      /// Checks if a timestamp parser is registered for the specified timestamp type.
      /// </summary>
      /// <typeparam name="TTimestamp"></typeparam>
      /// <returns></returns>
      bool Contains<TTimestamp>();
   }

You can use this to register implementation of ITimestampParser, which looks like this:

   /// <summary>
   /// ITimestampParser is responsible for parsing the 'time' column
   /// of data returned, allowing use of custom DateTime types.
   /// </summary>
   /// <typeparam name="TTimestamp"></typeparam>
   public interface ITimestampParser<TTimestamp>
   {
      /// <summary>
      /// Parses a epoch time (UTC) or ISO8601-timestamp (potentially with offset) to a date and time.
      /// This is used when reading data from influxdb.
      /// </summary>
      /// <param name="precision">TimestampPrecision provided by the current InfluxQueryOptions.</param>
      /// <param name="epochTimeLongOrIsoTimestampString">The raw value returned by the query.</param>
      /// <returns>The parsed timestamp.</returns>
      TTimestamp ToTimestamp( TimestampPrecision? precision, object epochTimeLongOrIsoTimestampString );

      /// <summary>
      /// Converts the timestamp to epoch time (UTC). This is used when writing data to influxdb.
      /// </summary>
      /// <param name="precision">TimestampPrecision provided by the current InfluxWriteOptions.</param>
      /// <param name="timestamp">The timestamp to convert.</param>
      /// <returns>The UTC epoch time.</returns>
      long ToEpoch( TimestampPrecision precision, TTimestamp timestamp );
   }

Once this is registered you can use the type on a property with the InfluxTimestampAttribute.

If you want to use it with the interface IInfluxRow, I have split it up into two:

   /// <summary>
   /// IInfluxRow is an interface that allows using more dynamic types as rows in InfluxDB.
   /// 
   /// When implementing this interface, InfluxAttributes are ignored and the interface 
   /// is used instead.
   /// 
   /// This interface uses DateTime? as the timestamp type.
   /// </summary>
   public interface IInfluxRow : IInfluxRow<DateTime?>
   {
   }

   /// <summary>
   /// IInfluxRow is an interface that allows using more dynamic types as rows in InfluxDB.
   /// 
   /// When implementing this interface, InfluxAttributes are ignored and the interface 
   /// is used instead.
   /// 
   /// The generic parameter is the type that is used for the timestamp.
   /// </summary>
   /// <typeparam name="TTimestamp"></typeparam>
   public interface IInfluxRow<TTimestamp>
   {
      /// <summary>
      /// Sets the timestamp.
      /// </summary>
      /// <param name="value"></param>
      void SetTimestamp( TTimestamp value );

      /// <summary>
      /// Gets the timestamp.
      /// </summary>
      /// <returns></returns>
      TTimestamp GetTimestamp();

      /// <summary>
      /// Sets a field.
      /// </summary>
      /// <param name="name"></param>
      /// <param name="value"></param>
      void SetField( string name, object value );

      /// <summary>
      /// Gets a field.
      /// </summary>
      /// <param name="name"></param>
      /// <returns></returns>
      object GetField( string name );

      /// <summary>
      /// Sets a tag.
      /// </summary>
      /// <param name="name"></param>
      /// <param name="value"></param>
      void SetTag( string name, string value );

      /// <summary>
      /// Gets a tag.
      /// </summary>
      /// <param name="name"></param>
      /// <returns></returns>
      string GetTag( string name );

      /// <summary>
      /// Gets all tags contained in the IInfluxRow.
      /// </summary>
      /// <returns></returns>
      IEnumerable<KeyValuePair<string, string>> GetAllTags();

      /// <summary>
      /// Gets all fields cotnained in the IInfluxRow.
      /// </summary>
      /// <returns></returns>
      IEnumerable<KeyValuePair<string, object>> GetAllFields();
   }

So you can now just implement the generic version of the interface if you want to have a different timestamp type.

Finally, here's an two implementation I made for DateTime and DateTimeOffset:

   /// <summary>
   /// Implementation of ITimestampParser that always parses to UTC DateTimes.
   /// </summary>
   public class UtcDateTimeParser : ITimestampParser<DateTime>
   {
      private static readonly DateTimeStyles OnlyUtcStyles = DateTimeStyles.AdjustToUniversal | DateTimeStyles.AssumeUniversal;

      /// <inheritdoc />
      public long ToEpoch( TimestampPrecision precision, DateTime timestamp )
      {
         return timestamp.ToPrecision( precision );
      }

      /// <inheritdoc />
      public DateTime ToTimestamp( TimestampPrecision? precision, object epochTimeLongOrIsoTimestampString )
      {
         if( !precision.HasValue )
         {
            // if no precision is specified, the time column is returned as a ISO8601-timestamp.
            return DateTime.Parse( (string)epochTimeLongOrIsoTimestampString, CultureInfo.InvariantCulture, OnlyUtcStyles );
         }
         else
         {
            // if a precision is specified, the time column is returned as a long epoch time (accuracy based on precision)
            return DateTimeExtensions.FromEpochTime( (long)epochTimeLongOrIsoTimestampString, precision.Value );
         }
      }
   }

   /// <summary>
   /// Implementation of ITimestampParser that maintains the offset from UTC.
   /// </summary>
   public class LocalDateTimeOffsetParser : ITimestampParser<DateTimeOffset>
   {
      /// <inheritdoc />
      public long ToEpoch( TimestampPrecision precision, DateTimeOffset timestamp )
      {
         return timestamp.ToPrecision( precision );
      }

      /// <inheritdoc />
      public DateTimeOffset ToTimestamp( TimestampPrecision? precision, object epochTimeLongOrIsoTimestampString )
      {
         if( !precision.HasValue )
         {
            // if no precision is specified, the time column is returned as a ISO8601-timestamp.
            return DateTimeOffset.Parse( (string)epochTimeLongOrIsoTimestampString, CultureInfo.InvariantCulture );
         }
         else
         {
            // the offset cannot be preserved with an epoch time, will throw to alert user
            throw new InfluxException( Errors.MissingOffsetInEpochTime );
         }
      }
   }

This should be fairly simple to work with. I will upload this once I have refined it a bit.

[georgiosd]

That's great, thanks!

How does InfluxDb actually expect the timestamp? Would a unix time (ms) fly with the database?

[MikaelGRA]

InfluxDB expects the timestamp as unix time (UTC). The accuracy of it depends on a parameter passed to influxdb during the write operation. This is represented by the TimestampPrecision passed into the "ToEpoch" method.

In my library this comes from the InfluxWriteOptions parameter.

[MikaelGRA]

I have release version 3.4.0, which includes this feature.

[georgiosd]

If you want, you could include this class: https://github.com/discretelogics/TeaFiles.Net-time-series-storage-in-flat-files/blob/master/TeaFiles/Base/Time.cs

It's very efficient!

[MikaelGRA]

I could consider creating an add-on library to the main nuget package:

Vibrant.InfluxDB.Client.TeaFiles

In fact I have already considered doing this same thing with NodaTime.

However, for now I am going to skip that. But it should be easy to implement an ITimestampParser for it. The only problem I can really see, is how you would parse a string (ISO8601 format) into the timestamp, without first creating a standard .NET DateTime object (or maybe you dont consider that a problem?), since I cannot find a 'Parse' method that takes a string in that library.

I would be happy to include it, if you provide an implementation though.

[georgiosd]

You would have to go by a DateTime object to parse.

But as you know it's writing that needs to be more efficient - which it would :)

Sure, I can definitely contribute that. Give me some time.