FSReader: add support for files and lists of files

[immerrr]

Right now FSReader only allows dir+pattern way of looking up inputs and sometimes to parallelise things we want to fetch the exact files matching that and distribute them evenly across subprocesses running actual exporters code.

I was thinking about the following way to accept input (i'm writing in yaml, but please imagine the same in json):

input: /path/to/file   # specify one file

input:  # specify a directory, like before
  dir: /path/to/dir  
  pattern: foo.*bar

input:  # specify a list of files
  - /path/to/file1
  - /path/to/file2

input:
  - dir: /path/to/dir1
    pattern: foo.*bar
  - dir: /path/to/dir2
    pattern: qux.*quux

So the idea is simple:

• input element contains either a single input-unit or a list of them
• input-unit is either a file or a dictionary with dir and optional pattern keys

What do you think?

[eliasdorneles]

This sounds good to me. I have a minor concern about backwards compatibility, but since exporters is not yet open sourced (will be in a couple of weeks) and you've been the early users of FSReader, if you're happy with changing the API, we're happy too. :)

[bbotella]

Same here. Let's do it!

[eliasdorneles]

Closed by https://github.com/scrapinghub/exporters/pull/268