crest scraper

[blitzmann]

some thoughts on how to put together a CREST scraper.

If there is a dict or list, that means we need to separate into another table/json file. for example, items have a dogma dict:

dogma: {
    'attributes': [list of attributes]
    'effects': [list of effects]
}

We would delve in, knowing that the attribute list would be transcribed to a type_attribute table/file (currently dgmTypeAttribs.json). Same with effects.

These lists also include href's: (shown is one member of a list)

attribute: {
    id_str: "6",
    href: "https://public-crest.eveonline.com/dogma/attributes/6/",
     id: 6,
     name: "capacitorNeed"
},
value: 240

This member can be inserted directly into the type_attribute table, and then the attribute.href can be accessed to provide info on the actual attribute (for inclusion into dgmattribs table).

Not sure if this is something that can be abstracted for the whole CREST interface, or if we'll have to develop some sort of "Miner" with instructions for every resource. If the later, we could have things like and Item miner or Dogma miner (possibly a miner for every 'dict' and list, basically every level of an item or other resource). For the something like the Dogma miner, maybe we could pass the dogma dict of the items and it'll sort it out. These would have to keep track of which attribute IDs and effect IDs we've already accessed so that we don't access them again