File helper improvements

[BioPhoton]

The file helpers do not really scale at the moment.

The current API of the crawlFileSystem function looks like this:

export async function crawlFileSystem<T = string>(options: {
  directory: string;
  pattern?: string | RegExp; // file name pattern
  fileTransform?: (filePath: string) => Promise<T> | T;
}): Promise<T[]> {

}

API Improvements:

• add file typing to accept string or GlobPattern for path pattern matching
• add content typing to accept string or RegExp for content pattern matching
• allow multiple values

Logic improvements:

• use the fastest libs for filesystem and pattern matching see perf benchmarks in utils

The potential API of the crawlFileSystem function looks like this:

export async function crawlFileSystem<T = string>(options: {
  file: (string | GlobPattern)[]; // file name pattern
  filterByContent?: (string | RegExp | (content: string) => boolean)[]; // file content pattern
}): Promise<T[]> {

}

[vmasek]

Lets threat this ticket as a research and refine separate issues for glob and file access.

Spike: 2md

[BioPhoton]

export async function crawlFileSystemAndLoad<T = string>(options: {
  file: ['**/tmp/**', '(search|test).component.ts', (path: string) => boolean ],
  // is Angular inline styles present
  content: [string | RegExp, (content: string) => boolean]
}) => Promise<T[]>

const filePaths =  await crawlFileSystem<T = string>({
  file: [(path: string) => boolean ],
  content: [(content: string) => boolean]
}): Promise<T[]> {

}

filePaths.map(loadFile)