Create OIH compliant action 'Lookup Files (many)'

[hannah-bulmer]

OIH Standard

SFTP should have an action Lookup Files (many) that allows for the lookup of multiple files based on criteria including

• pathname (including globbing) (will be required)
• file size
• file type
• last modified

Config fields:

• Behavior (dropdown: Fetch all, Emit Individually)

Instead of supplying # search terms in the config fields, since there as a static number of attributes by which to search files in SFTP, it may make more sense to display them all in the metadata. An object type dropdown field is not necessary as the only object type is files.

Note our api to connect to sftp does not offer fetchPage type results.

Metadata:

• max result size: optional positive integer that defaults to 1000 (only if fetch all mode)
• search terms
  • file path name (globbing supported)
  • file size and search condition (e.g. <, >, etc)
  • file last modified
  • others as detailed by jacob below will be left for future enhancements

The information emitted from this action will be file metadata, rather than explicitly having to possibly read a large number of files into Steward. The files themselves can then be accessed through lookup files (at most one).

[jhorbulyk]

My notes:

• When doing searches in file systems, I can imagine two use cases: Searching to learn file metadata information (but not the files themselves) and cases to learn the file data as well. Perhaps as an intial pass, we just return the file metadata without taking the time to read the files? (And leave individual file retrieval to Lookup File (at most one).)
• In addition to path, SFTP lists the following file attributes:
  • type (e.g. symlink, etc): Makes sense for search but is less common. Let's skip it on the first pass.
  • size: We should allow range searches (e.g. <, =<, >=, >) on this numeric field.
  • allocation-size: IMO does not make sense for searches.
  • owner and group: Could make sense in searches but lets skip on the intial pass.
  • permissions: Could make sense in searches but lets skip on the intial pass.
  • atime/atime-nseconds (Access Time) and ctime/ctime-nseconds (File attributes changed time): IMO does not make sense for searches.
  • createtime/createtime-nseconds and mtime/mtime-nseconds (Last modified time): We should allow range searches (e.g. <, =<, >=, >) on this order-able field.
  • acl, attrib-bits, attrib-bits-valid, text-hint, mime-type, link-count, untranslated-name, extended-count, extensions: IMO does not make sense for searches.
• If file type is refering to extenstions, then it makes sense to handle this as part of filename/file path.