Use YAML for config instead of INI

[rshipp]

Solves the auth duplication issues and allowing multiple auths issue. Cleaner and still human readable/writable.

Depends on #44.

Final Design

general:
    sleep: 1500
    daemon: true

credentials:
  - name: twitter-myuser
    token: EXAMPLE
    token_key: EXAMPLE
    con_secret_key: EXAMPLE
    con_secret: EXAMPLE

sources:
  - name: twitter-open-directory
    credentials: twitter-myuser
    module: twitter
    saved_state: 
    q: '"open directory" #malware'

  - name: twitter-inquest-c2-list
    credentials: twitter-myuser
    module: twitter
    saved_state: 
    owner_screen_name: InQuest
    slug: c2-feed

operators:
  - name: mycsv
    module: csv
    filename: output.csv
    allowed_sources: [mysource, myothersource]
    filter: '([^\.]google.com$|google.com[^/])'
    artifact_types: [URL, Domain]

Old brainstorming info

Currently, config looks like this: ``` [source:twitter-inquest-c2-list] module = twitter saved_state = token = EXAMPLE token_key = EXAMPLE con_secret_key = EXAMPLE con_secret = EXAMPLE owner_screen_name = InQuest slug = c2-feed [source:twitter-open-directory] module = twitter saved_state = token = EXAMPLE token_key = EXAMPLE con_secret_key = EXAMPLE con_secret = EXAMPLE q = "open directory" #malware ``` When adding multiple plugins with the same module, you have to specify credentials each time, which is a hassle. I'd like to get rid of that requirement, and let people set up something that looks more like this: ```yaml twitter-myuser: module: twitter token: EXAMPLE token_key: EXAMPLE con_secret_key: EXAMPLE con_secret: EXAMPLE sources: twitter-open-directory: saved_state: q: '"open directory" #malware' twitter-inquest-c2-list: saved_state: owner_screen_name: InQuest slug: c2-feed ``` In this case, you only have to define the credentials once per account, and you can still use multiple Twitter accounts if desired, by creating a second base object, e.g.: ```yaml twitter-myotheruser: module: twitter token: EXAMPLE ... ``` I'm not sure how clean the implementation would be for this, so we should think about and map out a complete design for this to see if there's a better way to do it. For example, it might be better to have separate base sections for `credentials`, `sources`, and `operators`, and just include a reference to a certain named `credential` in each `source` or `operator`, something like this: ```yaml credentials: twitter-myuser: token: EXAMPLE token_key: EXAMPLE con_secret_key: EXAMPLE con_secret: EXAMPLE sources: twitter-open-directory: credentials: twitter-myuser module: twitter saved_state: q: '"open directory" #malware' twitter-inquest-c2-list: credentials: twitter-myuser module: twitter saved_state: owner_screen_name: InQuest slug: c2-feed ``` Or to abstract it even further, and allow including any section in any other section: ```yaml twitter-myuser: module: twitter token: EXAMPLE token_key: EXAMPLE con_secret_key: EXAMPLE con_secret: EXAMPLE source-twitter-open-directory: include: twitter-myuser saved_state: q: '"open directory" #malware' source-twitter-inquest-c2-list: include: twitter-myuser saved_state: owner_screen_name: InQuest slug: c2-feed ``` This is the most flexible design, but I'm not sure whether I prefer having a defined `source` section, vs having the plugin type be based off the name (`source-*`, `operator-*`), vs having a key that defines the type (`type: source`). We need some way to differentiate sources and operators, since there can be duplication between them (e.g. `SQS` is both a source and an operator). ### Moved from comment below. Looking at some example YAML configs in other tools (Kubernetes, Ansible, etc), it seems having a list of items, each with a `name` key, is a common pattern. This makes me lean towards the second proposal above, with a slight modification: ```yaml sources: - name: twitter-open-directory credentials: twitter-myuser module: twitter saved_state: q: '"open directory" #malware' - name: twitter-inquest-c2-list credentials: twitter-myuser module: twitter saved_state: owner_screen_name: InQuest slug: c2-feed ``` I'm still not sure how to best handle credentials. The third proposal's "include" solution is more flexible, but I can't think of a case where you'd want to reuse anything that wasn't credentials. Having creds in their own named "credentials" section seems clearer for the end user, but doesn't accurately represent how the config parsing would be implemented, as you could feasibly define any parameters in the "credentials" section and reuse them for some other purpose.

[rshipp]

Closed by #45.