Better deck scraper - Githubissues

Everlag / preordain-backend

backend for preorda.in

Other

1 stars 0 forks source link

Better deck scraper #9

Closed Everlag closed 8 years ago

Everlag commented 8 years ago

The current scraper generates a static cache file which we cannot update. This means we cannot provide any historical data on an archetype which limits our utility.

A better deck scraper would acquire decks from mtgtop8, massage the data, and put everything into a series of sane tables in postgres.

Additionally, a video scraper from mtgcoverage looks possible and would provide a lot of value.

Everlag commented 8 years ago

Some notes

Structure the scraper ala priceWriter with two asymmetric sources that handle the database access separately. Make sure that any panics are recovered inside the scrapers to avoid full crashes on markup changes.
Deck names need to be normalized between mtgtop8 and mtgcoverage.
mtgtop8 provides extremely broad categories, ie UrzaTron for U Tron. Granular naming by looking for specific cards, ie snapcaster or sad bot in U tron, could do it.

Everlag commented 8 years ago

82664ae1ac0db8bb001a2ab778b12d763415f52f handles modern decks from mtgtop8 with normalization to our names on egress and blurring on ingress.

Opening #10 to handle mtgcoverage.