Images: tokens source is broken (tokens.mtg.onl, mythicspoiler.com and others)

[JayDi85]

• [ ] https://www.mtg.onl/ updated to new version and token images download is broken now. Need to fix/upgrade or remove forever;
• [ ] http://mythicspoiler.com/ - outdated code and old sets. Need to fix/upgrade or remove forever.
• [ ] other sources see in related commits

[Malktuth]

I wrote a code in R (sorry, no python) to download the token pics. In case anyone is interested... The whole code might be a mess (I ain't no programmer)

#Load packages
if (!require(pacman)){install.packages('pacman')}
p_load(RCurl,rvest,XML,httr,dplyr,progress)

#Creates the function to download the pics
getpic = function(ed,root.path = 'E:/Stuff/MTG/TOK'){ #Change root.path to whatever
  if (!dir.exists(paste(root.path,ed,sep = '/'))) {dir.create(paste(root.path,ed,sep = '/'),recursive = T)}
  setwd(file.path(root.path,ed))
  url <- paste0("https://www.mtg.onl/tokens/",toupper(ed))

  doc = GET(url)
  doc = htmlParse(doc,encoding = 'UTF-8')
  cardnames = xpathSApply(doc, '//*[@class="tokensGroup-module--link--2XG4o"]/@href') %>% 
    gsub('^\\/token-list\\/(.*)\\/$','\\1',.) %>% 
    gsub('\\_[UWBRG]*\\_*\\d\\_\\d$','',.) %>% 
    gsub('\\_',' ',.)
  if (any(duplicated(cardnames))) 
  {
    dupes = cardnames[duplicated(cardnames)]
    for (dupe in dupes)
    {
      cardnames[cardnames %in% dupe] = paste(dupe,seq(1,sum(cardnames %in% dupe)))
    }
  }

  links = xpathSApply(doc, "//img/@srcset") %>% 
    grep(paste0(ed,'.*\\.jpg'),.,ignore.case = T,value = T) %>% 
    sapply(., function(x) {strsplit(x,',') %>% 
        unlist() %>% tail(1) %>% 
        gsub('\\n','',.) %>% 
        gsub(' \\d{3}w$','',.)})

  pb = winProgressBar('Downloading',,0,100,0)
  for (i in 1:length(links))
  {
    download.file(links[i], destfile = basename(sprintf('%s.full.jpg',cardnames[i])),mode = 'wb',quiet = T)
    info <- sprintf("%s%%", round(100*i/length(links),2))
    setWinProgressBar(pb, round(100*i/length(links),2), sprintf("Downloading (%s)", info), info)
  }
  close(pb)
}

# Get the code for each expansion
setnamesAll = NULL
for (i in c("",2:11)){
  url <- sprintf("https://www.mtg.onl/tokens/%s",i)
  doc = GET(url)
  doc = htmlParse(doc,encoding = 'UTF-8')
  setnames = xpathSApply(doc, '//*[@class="tokensGroup-module--header--JeYYP"]/h3/i/@class') %>% 
    gsub('^ss\\-(\\w+)\\sss.*','\\1',.) %>% toupper()
  setnamesAll = c(setnamesAll,setnames)  
}
setnamesAll = gsub('\\_','',setnamesAll)
dwlded = c(dir('.'),'CON') # Removes already downloaded directories a also for some reason you can create a 'CON' folder, so CONflux is out

pb <- progress_bar$new(format = "  downloading :what [:bar] :percent eta: :eta", total = length(setdiff(setnamesAll,dwlded)))
for (ed in grep(sprintf('(%s)',paste0(dwlded,collapse = '|')),setnamesAll,invert = T,value = T))
{
  getpic(ed)
  pb$tick(tokens = list(what = ed))
}

#Go up and zip all
setwd('..')
zip('TOK.zip',dir('.'))

[timhae]

I would like to implement that feature but I am a little confused regarding the current implementation. Are there two files implementing this?

• https://github.com/magefree/mage/blob/master/Mage.Client/src/main/java/org/mage/plugins/card/dl/sources/TokensMtgImageSource.java#L22
• https://github.com/magefree/mage/blob/master/Mage.Client/src/main/java/org/mage/plugins/card/dl/sources/MtgOnlTokensImageSource.java#L15
• (and maybe also this https://github.com/magefree/mage/blob/master/Mage.Client/src/main/java/org/mage/plugins/card/dl/sources/AltMtgOnlTokensImageSource.java#L15 even if it is just for non-offical tokens)

I would rewrite those into one file and make it similar to the scryfall one.

Also, should I update this file https://github.com/magefree/mage/blob/master/Mage.Client/src/main/resources/tokens-mtg-onl-list.csv programmatically? Last update was 4 years ago..

[JayDi85]

1. That's all tokens download are broken now:
   • MtgOnlTokensImageSource -- never worked, can be deleted;
   • TokensMtgImageSource and AltMtgOnlTokensImageSource are broken after site updated. I think it can be deleted at all too (after scryfall or another token source completed).
2. tokens-mtg-onl-list.csv that's internal file downloaded from onl-source, so no needs in it anymore and it can be deleted too.
3. Yes, it will be good to implement tokens download from scryfall or another source. You can add tokens download as new images source from scratch or to existed file. If you would add scryfall support and change existed source then try to make changes like that PR (#6190 -- search files with scryfall: 1, 2 -- that PR will be merged soon);

BUT what's your idea to implement it? It's not so easy. XMage uses custom tokens list and you can't download it by names -- tokens with same name can be different (e.g. different PT, abilities, etc). So you must manually link xmage's tokens with images source.

How's xmage tokens works:

• All tokens list stores at ~card-pictures-tok.txt~ tokens-database.txt -- that's tokens "database" and that's list manually edited. Each token must be linked to real token class that used in set's card (that's help to use different token images for same name):

How's to add new tokens and downloads (as example, for new set):

• Open set tokens list at scryfall;
• Take first token name and search cards with (oracle:token set:m19 oracle:token_name) -- you can find all cards that uses that token;
• Search that cards source code in xmage and look at token creating code. It can use different token classes.
• Add each token usage in tokens-database.txt. Tokens with same name but different images and clases must have different indexes (1,2,3) and classes:
• Open scryfall tokens source and add direct link to download image for same token's index;
  • Same for emblems, but it must use specific name notations in tokens and scryfall files.

[timhae]

Thanks a lot, that's very useful input! I will try and brew smething up in a pr or get back to you if I ever get stuck. I guess I will either wait until #6190 is merged or just create a branch with that fork as upstream..

[Jieiku]

How can I test which Tokens I am missing? (is there a way to generate a list of missing token images?)

In the image downloader I can see I am missing 32 card images and 238 token images....

I got most of the missing card images here using the clipboard option: https://github.com/magefree/mage/issues/9249#issuecomment-1457490644

but for tokens option 8 Copy and Paste Image URLs does not seem to apply to tokens...

I figured this out mostly: https://github.com/magefree/mage/issues/6955#issuecomment-1458924711

[JayDi85]

Web site reworked and tokens download doesn't work anymore. Also it's outdated -- miss tokens from last years releases.

But it can be used to download a proxy tokens in the future (tokens that was miss by wizards), see https://www.mtg.onl/mtg-missing-tokens/