Create OIH compliant trigger 'Get new and updated files polling'

[hannah-bulmer]

OIH standard

SFTP should have a trigger similar to read, which returns files polling based on date.

Config fields:

• file path to search (supports globbing)
• Start Time (string, optional): Indicates the beginning time to start polling from (defaults to the begining of time)
• End Time (string, optional): If provided, don’t fetch records modified after this time (defaults to never)

Note that our api to connect to sftp does not support fetching data in pages

[jhorbulyk]

My Thoughts:

• It might make sense to separate folder to watch (which could be a dynamic dropdown) from filename glob. A recursive flag (e.g. look for new files in subdirectories) might also make sense.

• SFTP supports the following timestamp info about files:

  • createtime/createtime-nseconds
  • mtime/mtime-nseconds (Last modified time)
  • atime/atime-nseconds (Access Time)
  • ctime/ctime-nseconds (File attributes changed time)

  It probably makes sense to expose a dropdown for created vs modified time (at least).

• The body of the messages produced by this trigger should have the Stewart URL of the file in the body along with all available meta-data about the file in the body. There is no need to populate the attachments part of the message.
• When reading files, the requested access level for the file should be SSH_FXF_BLOCK_READ so that we can be sure that the file is not being written while it is being consumed.