Closed bakert closed 6 years ago
https://img.scryfall.com/cards/large/en/mm2/75.jpg?1509843238
They do have it. So the part that bluebones does for us now is 'just' the compositing of 2-6 images.
Website could switch to scryfall tomorrow.
You can get image from names rather easily. https://api.scryfall.com/cards/named?exact=Mother+of+Runes&format=image
It's a shame they have those white corners.
The PNG versions don't. https://api.scryfall.com/cards/named?exact=Mother+of+Runes&format=image&version=png
Some rudimentary testing suggests this is pretty much a strict upgrade:
[deepthought pd] wget 'https://api.scryfall.com/cards/named?exact=Mother+of+Runes\&format=image\&version=png'
--2017-11-30 11:06:57-- https://api.scryfall.com/cards/named?exact=Mother+of+Runes%5C&format=image%5C&version=png
Resolviendo api.scryfall.com... 23.23.118.21, 204.236.236.192, 184.73.203.207, ...
Conectando con api.scryfall.com[23.23.118.21]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: no especificado [application/json]
Grabando a: "named?exact=Mother+of+Runes%5C&format=image%5C&version=png.2"
named?exact=Mother+of+Runes%5C&format= [ <=> ] 3.60K --.-KB/s en 0.002s
2017-11-30 11:06:57 (1.58 MB/s) - "named?exact=Mother+of+Runes%5C&format=image%5C&version=png.2" guardado [3683]
[deepthought pd] wget 'http://mtg.bluebones.net/proxies/index2.php?c=Mother+of+Runes'
--2017-11-30 11:07:19-- http://mtg.bluebones.net/proxies/index2.php?c=Mother+of+Runes
Resolviendo mtg.bluebones.net... 178.79.178.155
Conectando con mtg.bluebones.net[178.79.178.155]:80... conectado.
Petici'on HTTP enviada, esperando respuesta... 404 Not Found
2017-11-30 11:07:20 ERROR 404: Not Found.
[deepthought pd] wget 'http://magic.bluebones.net/proxies/index2.php?c=Mother+of+Runes'
--2017-11-30 11:07:25-- http://magic.bluebones.net/proxies/index2.php?c=Mother+of+Runes
Resolviendo magic.bluebones.net... 178.79.178.155
Conectando con magic.bluebones.net[178.79.178.155]:80... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: no especificado [image/jpeg]
Grabando a: "index2.php?c=Mother+of+Runes"
index2.php?c=Mother+of+Runes [ <=> ] 49.39K 141KB/s en 0.4s
The one thing magic.bluebones.net still does for us is compositing, of course, which, of course, can still be moved into our code. I think breaking that dependency is probably worth the effort.
Nope, I'm drunk. These images are large and high quality and take a long time to download even on a fast connection if bash doesn't fritz with the URL:
[deepthought pd] time curl -o image.png -L -G --data-urlencode "exact=Mother of Runes" --data-urlencode "format=image" --data-urlencode "version=png" https://api.scryfall.com/cards/named
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 125 0 125 0 0 0 0 --:--:-- 1:11:35 --:--:-- 263
100 1709k 100 1709k 0 0 569k 0 0:00:03 0:00:03 --:--:-- 729k
real 0m3.188s
user 0m0.039s
sys 0m0.031s
Even the small JPEG image does a redirect and takes 2s real time.
[deepthought pd] time curl -o image.jpg -L -G --data-urlencode "exact=Mother of Runes" --data-urlencode "format=image" --data-urlencode "version=small" https://api.scryfall.com/cards/named
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 127 0 127 0 0 0 0 --:--:-- 1:11:35 --:--:-- 141
100 17178 100 17178 0 0 3 0 1:35:26 1:11:36 0:23:50 33290
real 0m1.939s
user 0m0.026s
sys 0m0.008s
One thing that occurs to me is that we could literally have every image local for a few GB of disk space. That doesn't help web much but it helps the bot.
I'm back to wanting to put magiccards.info id into magic db and then use it to refer directly to the mci image right now, though.
The images for the tooltips are pretty fast, too. We could consider using those (which can be referenced by name only).
Tooltips very similar speed to bluebones and lower quality:
[deepthought pd] wget https://deckbox.org/mtg/Elite%20Inquisitor/tooltip
--2017-11-30 11:22:09-- https://deckbox.org/mtg/Elite%20Inquisitor/tooltip
Resolviendo deckbox.org... 5.9.7.215
Conectando con deckbox.org[5.9.7.215]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 30719 (30K) [text/html]
Grabando a: "tooltip"
tooltip 100%[===========================================================================>] 30.00K 71.8KB/s en 0.4s
2017-11-30 11:22:10 (71.8 KB/s) - "tooltip" guardado [30719/30719]
[deepthought pd] open tooltip
[deepthought pd] wget https://deckbox.org/mtg/Elite%20Inquisitor/tooltip
--2017-11-30 11:22:23-- https://deckbox.org/mtg/Elite%20Inquisitor/tooltip
Resolviendo deckbox.org... 5.9.7.215
Conectando con deckbox.org[5.9.7.215]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 30719 (30K) [text/html]
Grabando a: "tooltip.1"
tooltip.1 100%[===========================================================================>] 30.00K 74.5KB/s en 0.4s
2017-11-30 11:22:25 (74.5 KB/s) - "tooltip.1" guardado [30719/30719]
[deepthought pd] wget https://deckbox.org/mtg/Far%20//%20Away/tooltip
--2017-11-30 11:22:40-- https://deckbox.org/mtg/Far%20//%20Away/tooltip
Resolviendo deckbox.org... 5.9.7.215
Conectando con deckbox.org[5.9.7.215]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 31742 (31K) [text/html]
Grabando a: "tooltip.2"
tooltip.2 100%[===========================================================================>] 31.00K 79.2KB/s en 0.4s
2017-11-30 11:22:41 (79.2 KB/s) - "tooltip.2" guardado [31742/31742]
So I'm now really focused on direct reference to MCI.
Getting the small JPEG via direct URL (no redirect) on scryfall is pretty fast:
[deepthought pd] wget https://img.scryfall.com/cards/small/en/ema/22.jpg?1509843805
--2017-11-30 11:24:36-- https://img.scryfall.com/cards/small/en/ema/22.jpg?1509843805
Resolviendo img.scryfall.com... 2400:cb00:2048:1::681c:1fb4, 2400:cb00:2048:1::681c:1eb4, 104.28.31.180, ...
Conectando con img.scryfall.com[2400:cb00:2048:1::681c:1fb4]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 17178 (17K) [image/jpeg]
Grabando a: "22.jpg?1509843805"
22.jpg?1509843805 100%[===========================================================================>] 16.78K --.-KB/s en 0.04s
2017-11-30 11:24:37 (388 KB/s) - "22.jpg?1509843805" guardado [17178/17178]
[deepthought pd] wget https://img.scryfall.com/cards/small/en/ema/22.jpg?1509843805
--2017-11-30 11:24:41-- https://img.scryfall.com/cards/small/en/ema/22.jpg?1509843805
Resolviendo img.scryfall.com... 2400:cb00:2048:1::681c:1fb4, 2400:cb00:2048:1::681c:1eb4, 104.28.31.180, ...
Conectando con img.scryfall.com[2400:cb00:2048:1::681c:1fb4]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 17178 (17K) [image/jpeg]
Grabando a: "22.jpg?1509843805.1"
22.jpg?1509843805.1 100%[===========================================================================>] 16.78K --.-KB/s en 0.1s
2017-11-30 11:24:42 (148 KB/s) - "22.jpg?1509843805.1" guardado [17178/17178]
[deepthought pd] wget https://img.scryfall.com/cards/small/en/ema/22.jpg?1509843805
--2017-11-30 11:24:47-- https://img.scryfall.com/cards/small/en/ema/22.jpg?1509843805
Resolviendo img.scryfall.com... 2400:cb00:2048:1::681c:1fb4, 2400:cb00:2048:1::681c:1eb4, 104.28.31.180, ...
Conectando con img.scryfall.com[2400:cb00:2048:1::681c:1fb4]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 17178 (17K) [image/jpeg]
Grabando a: "22.jpg?1509843805.2"
22.jpg?1509843805.2 100%[===========================================================================>] 16.78K --.-KB/s en 0.1s
2017-11-30 11:24:47 (154 KB/s) - "22.jpg?1509843805.2" guardado [17178/17178]
I should really listen to myself. Going directly to mci is both high quality and approx the fastest.
[deepthought pd] wget https://magiccards.info/scans/en/ema/22.jpg
--2017-11-30 11:26:20-- https://magiccards.info/scans/en/ema/22.jpg
Resolviendo magiccards.info... 162.218.229.198
Conectando con magiccards.info[162.218.229.198]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 70032 (68K) [image/jpeg]
Grabando a: "22.jpg"
22.jpg 100%[===========================================================================>] 68.39K 185KB/s en 0.4s
2017-11-30 11:26:21 (185 KB/s) - "22.jpg" guardado [70032/70032]
[deepthought pd] wget https://magiccards.info/scans/en/ema/22.jpg
--2017-11-30 11:26:22-- https://magiccards.info/scans/en/ema/22.jpg
Resolviendo magiccards.info... 162.218.229.198
Conectando con magiccards.info[162.218.229.198]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 70032 (68K) [image/jpeg]
Grabando a: "22.jpg.1"
22.jpg.1 100%[===========================================================================>] 68.39K 215KB/s en 0.3s
2017-11-30 11:26:24 (215 KB/s) - "22.jpg.1" guardado [70032/70032]
[deepthought pd] wget https://magiccards.info/scans/en/ema/22.jpg
--2017-11-30 11:26:25-- https://magiccards.info/scans/en/ema/22.jpg
Resolviendo magiccards.info... 162.218.229.198
Conectando con magiccards.info[162.218.229.198]:443... conectado.
Petici'on HTTP enviada, esperando respuesta... 200 OK
Longitud: 70032 (68K) [image/jpeg]
Grabando a: "22.jpg.2"
22.jpg.2 100%[===========================================================================>] 68.39K 386KB/s en 0.2s
2017-11-30 11:26:26 (386 KB/s) - "22.jpg.2" guardado [70032/70032]
Here's the code behind bluebones that powers MCI image url generation. We probably won't need most of this because mtgjson contains the codes/numbers we need (?)
module MtG
class MagicCardsInfo
def MagicCardsInfo.img_url(card, restrict_to_sets = nil)
# Favor recent printings over older printings to have more chance of
# correct Oracle text/creature types/etc.
card.sets.sort_by { |set| (set.released.to_i) }.each do |set|
#restrict_to_sets = [Set.Tempest, Set.Stronghold, Set.Exodus, Set.Urzas_Saga, Set.Urzas_Legacy, Set.Urzas_Destiny, Set.Mercadian_Masques, Set.Nemesis, Set.Prophecy, Set.Invasion, Set.Planeshift, Set.Apocalypse, Set.Odyssey, Set.Torment, Set.Judgment, Set.Onslaught, Set.Legions, Set.Scourge]
in_valid_sets = ! restrict_to_sets || restrict_to_sets.member?(set)
return MagicCardsInfo.img_url_from_card_and_set(card, set) if MagicCardsInfo.set_number(card, set) && in_valid_sets
end
return nil
end
def MagicCardsInfo.img_url_from_card_and_set(card, set)
code = MagicCardsInfo.set_code(set)
num = MagicCardsInfo.set_number(card, set)
letter = 'a' if card.split_card? && num.split('').last != 'a' # DGM split cards already have 'a' and older don't!
letter = 'b' if card.basic_land? && set == Set.Battle_for_Zendikar # BFZ basic lands are 'a' for normal and 'b' for full art.
# if (code == 'ddj' and num == '32' and card.split_card?)
# letter = '' # duel decks split card breaks the rules
# end
return "https://magiccards.info/scans/en/#{code}/#{num}#{letter}.jpg"
end
def MagicCardsInfo.hack(card, set)
return 'http://d1f83aa4yffcdn.cloudfront.net/' + set.code + '/' + card.name.downcase.gsub(' ', '%20') + '.jpg'
end
def MagicCardsInfo.set_number(card, set)
@@data ||= MagicCardsInfo.data
code = MagicCardsInfo.set_code(set)
if ! card.split_card?
key = card.name
else
key = card.name.split(' / ').first + ' (' + card.name.gsub(' ', '') + ')'
end
return @@data[[key, code]]
end
def MagicCardsInfo.data
sets = MagicCardsInfo.sets
r = {}
sets.each do |code|
url = "https://magiccards.info/#{code.to_s.strip}/en.html"
data = Cache.process(url, lambda { |s| MagicCardsInfo.parse_set(s) })
data.each do |code, num, name|
name = name.gsub('Æ', 'Ae').gsub('æ', 'ae')
#name = name.gsub('“', '"').gsub('”', '"') # "Ach! Hans, Run!" needs this.
r[[name, code]] = num
end
end
return r
end
def MagicCardsInfo.sets
url = 'https://magiccards.info/sitemap.html'
return Cache.process(url, lambda { |s| MagicCardsInfo.parse_sets(s) }).flatten
end
def MagicCardsInfo.parse_sets(s)
# Pretty big hack from http://stackoverflow.com/questions/2982677/ruby-1-9-invalid-byte-sequence-in-utf-8
# to get around non-UTF-8 friendly input.
s.encode!('UTF-16', 'UTF-8', :invalid => :replace, :replace => '')
s.encode!('UTF-8', 'UTF-16')
return s.scan(/href="\/(.*?)\/en.html/)
end
def MagicCardsInfo.parse_set(s)
# top and tail to avoid invalid byte sequence at start of page
return s.top_and_tail('<body', '</body').scan(/<td><a href="\/(.*?)\/en\/(.*?)\.html">(.*?)<\/a><\/td>/)
end
def find_img_url(name)
url = URI.parse('https://magiccards.info/query.php')
req = Net::HTTP::Post.new(url.path)
req.set_form_data({'textfield[]' => '1', 'textcmp[]' => '3', 'textval[]' => 'Absorb', 'textfield[]' => '2', 'textcmp[]' => '1', 'textval[]' => '', 'textfield[]' => '3', 'textcmp[]' => '1', 'textval[]' => '', 'language' => '', 'loose_types' => '1', 'subtypes' => '', 'ptcfield[]' => '', 'ptccmp[]' => '', 'ptcval[]' => '', 'ptcfield[]' => '', 'ptccmp[]' => '', 'ptcval[]' => '', 'ptcfield[]' => '', 'ptccmp[]' => '', 'ptcval[]' => '', 'ptcfield[]' => '', 'ptccmp[]' => '', 'ptcval[]' => '', 'mana' => '', 'artist' => '', 'sort_field1' => '1', 'sort_order1' => '1', 'sort_field2' => '1', 'sort_order2' => '1', 'output' => '0'}, ';')
res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req) }
case res
when Net::HTTPSuccess
puts "a"
puts res.body
when Net::HTTPRedirection
puts "b"
else
res.error!
end
end
# TODO Should this be in Set? Do these names come from anything?
# Find out what the real names are and add a translation layer only where magiccards.info is wrong
def MagicCardsInfo.set_code(set)
case set
when Set.Alpha
return "al"
when Set.Beta
return "be"
when Set.Unlimited_Edition
return "un"
when Set.Revised_Edition
return "rv"
when Set.Fallen_Empires
return "fe"
when Set.Homelands
return "hl"
when Set.Visions
return "vi"
when Set.Weatherlight
return "wl"
when Set.Exodus
return "ex"
when Set.Urzas_Legacy
return "ul"
when Set.Urzas_Destiny
return "ud"
when Set.Nemesis
return "ne"
when Set.Prophecy
return "pr"
when Set.Invasion
return "in"
when Set.Planeshift
return "ps"
when Set.Apocalypse
return "ap"
when Set.Odyssey
return "od"
when Set.Chronicles
return "ch"
when Set.Shards_of_Alara
return "ala"
when Set.Darksteel
return "ds"
when Set.Conflux
return "cfx"
when Set.Tenth_Edition
return "10e"
when Set.Morningtide
return "mt"
when Set.Lorwyn
return "lw"
when Set.Time_Spiral_Timeshifted
return "tsts"
when Set.Mirrodin
return "mi"
when Set.Onslaught
return "on"
when Set.Planar_Chaos
return "pc"
when Set.Dissension
return "di"
when Set.Ninth_Edition
return "9e"
when Set.Eighth_Edition
return "8e"
when Set.Seventh_Edition
return "7e"
when Set.Sixth_Edition
return "6e"
when Set.Fifth_Edition
return "5e"
when Set.Fourth_Edition
return "4e"
when Set.Masters_Edition
return nil
when Set.Scourge
return "sc"
when Set.Time_Spiral
return "ts"
when Set.Guildpact
return "gp"
when Set.Coldsnap
return "cs"
when Set.Urzas_Saga
return "us"
when Set.Torment
return "tr"
when Set.Legions
return "le"
when Set.Ice_Age
return "ia"
when Set.Portal_Three_Kingdoms
return "p3k"
when Set.Unhinged
return "uh"
when Set.Mirage
return "mr"
when Set.Portal
return "po"
when Set.Judgment
return "ju"
when Set.Tempest
return "tp"
when Set.The_Dark
return "dk"
when Set.Stronghold
return "sh"
when Set.Alliances
return "ai"
when Set.Legends
return "lg"
when Set.Portal_Second_Age
return "po2"
when Set.Antiquities
return "aq"
when Set.Arabian_Nights
return "an"
when Set.Starter_1999
return "st"
when Set.Starter_2000
return "st"
when Set.Duel_Decks_Jace_vs_Chandra
return "jvc"
when Set.Duel_Decks_Divine_vs_Demonic
return "dvd"
when Set.Duel_Decks_Garruk_vs_Liliana
return "gvl"
when Set.Duel_Decks_Phyrexia_vs_The_Coalition
return "pvc"
when Set.Duel_Decks_Elspeth_vs_Tezzeret
return "evt"
when Set.Duel_Decks_Knights_vs_Dragons
return "kvd"
when Set.Duel_Decks_Ajani_vs_Nicol_Bolas
return "avn"
when Set.Duel_Decks_Venser_vs_Koth
return "vvk"
when Set.Duel_Decks_Sorin_vs_Tibalt
return "svt"
when Set.Duel_Decks_Heroes_vs_Monsters
return "hvm"
when Set.Duel_Decks_Jace_vs_Vraska
return "ddm"
when Set.From_the_Vault_Dragons
return "fvd"
when Set.From_the_Vault_Exiled
return "fve"
when Set.From_the_Vault_Relics
return "fvr"
when Set.From_the_Vault_Legends
return "fvl"
when Set.Premium_Deck_Series_Slivers
return "pds"
when Set.Commanders_Arsenal
return "cma"
when Set.Portal
return "pt"
when Set.Portal_Second_Age
return p2"
when Set.Portal_Three_Kingdoms
return p3"
when Set.Unglued
return "ug"
when Set.Promos
return "xxx"
when nil
return nil
else
return set.code.downcase
end
end
end
end
We already store 'magiccardsinfo_code' against set and 'mci_number' against card_printing.
So something like:
def MagicCardsInfo.img_url(card, restrict_to_sets = nil)
# Favor recent printings over older printings to have more chance of
# correct Oracle text/creature types/etc.
card.sets.sort_by { |set| (set.released.to_i) }.each do |set|
#restrict_to_sets = [Set.Tempest, Set.Stronghold, Set.Exodus, Set.Urzas_Saga, Set.Urzas_Legacy, Set.Urzas_Destiny, Set.Mercadian_Masques, Set.Nemesis, Set.Prophecy, Set.Invasion, Set.Planeshift, Set.Apocalypse, Set.Odyssey, Set.Torment, Set.Judgment, Set.Onslaught, Set.Legions, Set.Scourge]
in_valid_sets = ! restrict_to_sets || restrict_to_sets.member?(set)
return MagicCardsInfo.img_url_from_card_and_set(card, set) if MagicCardsInfo.set_number(card, set) && in_valid_sets
end
return nil
end
def MagicCardsInfo.img_url_from_card_and_set(card, set)
code = MagicCardsInfo.set_code(set)
num = MagicCardsInfo.set_number(card, set)
letter = 'a' if card.split_card? && num.split('').last != 'a' # DGM split cards already have 'a' and older don't!
letter = 'b' if card.basic_land? && set == Set.Battle_for_Zendikar # BFZ basic lands are 'a' for normal and 'b' for full art.
# if (code == 'ddj' and num == '32' and card.split_card?)
# letter = '' # duel decks split card breaks the rules
# end
return "https://magiccards.info/scans/en/#{code}/#{num}#{letter}.jpg"
end
and then store img_url on _cache_card does the trick. Although it might be better not to loop over the printings at any stage and just pull out the right answer in base_query if we can do that.
if (card.layout == 'split' || card.layout == 'aftermath') and card.mci_number[-1] != 'a':
letter = 'a'
elif 'Basic Land' in card.type and card.most_recent_set_code == 'BFZ':
letter = 'b'
else:
letter = ''
return 'https://magiccards.info/scans/en/{code}/{num}{letter}.jpg'.format(code=c.most_recent_set_code, num=c.mci_number, letter=letter)
Oh, I'm slow. We already have this as image_fetcher.mci_image.
I will denormalize the URL to _cache_card to avoid having to query for and cycle through printings on user time.
Cards have more than one image url. Delver of Secrets the card (not the face) has two images. We have to account for that.
I think a card_image table that has all image urls we know of with some 'type' indicator that we can use as a priority makes sense?
Then you'd get the Delver of Secrets images by looping over every entry in card_image WHERE card_id = ? AND image/url_type = ?
Code that emits images becomes more complicated in this scenario but it can all be passed through a method in Card or something similar.
Do these even need to be stored or can they be derived at base_query time using set.magiccardsinfo_code and face.mci_number and GROUP_CONCAT and CONCAT, etc.
The more Scryfall improves the more I want to just live downstream of them. They are actually offering better info than mtgjson and their images are catching up with magiccards.info.
They have no way to address the image for a card via it's name only. You'd have to have a mapping of name -> image already downloaded or do an API lookup for the img_url. It is derivable from set + set number but we don't currently store that in the cards db. Not sure why not. I'm sure it's in mtgjson.
Alternatively the same process that feeds mtg.bluebones.net could just export a single file of name->img_url and we could grab it on startup and use it from then on.