TCG prices

[naudor]

Hi.

I tried to make the TCG example call but throw " ["", "", ""] ". You can fix, please. I could like use your API for a personal project.

Do know if you could get the TCG price of the foil version of the cards?

Thanks a lot.

[romelako]

Just wanted to add that I'm having the same issue here. Valid card names are returning ["", "", ""] for pricing. Perhaps you need to renew your "membership" with TCGPlayer for your prices? Thanks in advanced.

[Garbee]

It doesn't use the TCGPlayer API. It is scraping the site itself looking for the information. It is returning blank since the markup has changed and the software isn't updated yet to reflect these changes.

Recently though it appears as if TCGP has also added extra security to stop scrapers by detecting them and then blocking their access. So even if this gets fixed here it would also need to deal with bypassing the security on their end. Seems like a bad situation to deal with. :frowning:

[romelako]

How do you know about the scraping security? Are you not able to scrape TCGP w/ Python? What about other languages?

[bedoherty]

Sorry for my lack of availability guys, I don't check this project a whole lot between school, work, and other stuff. I'll look around at it tonight and try to push something later.

[bedoherty]

Just an update, it appears TCG players site is now secured by Distil. I'll have to figure out how realistic it is to find a work around, or if we need to give up on TCG player pricing.

[bedoherty]

Spent a couple hours messing around with it, and didn't find any easy solutions. One option would be to attempt to do this using Selenium, but AFAIK that is much more resource intensive. If anyone finds a better fix feel free to send a pull request.

[romelako]

I know for a fact you can get access to TCGPlayer's API if you email them and show them screenshots or images of a "web application" you intend to design using it. That would be the easiest fix and would be even easier than scraping a page.

The reason I couldn't get access to it was because I was going to use it for pricing cards on a storefront, which is not allowed as it would create a competing store.

[Garbee]

Tcgplayer won't approve a pure price provider though. They pride themselves in being a singular resource. It makes business sense, just sucks.

@bedoherty selenium isn't a good option either. If the api is used heavily then distill will catch onto it working and cut access. So that is a lot of work to only get little chunks at a time.

[starryknight64]

From https://help.tcgplayer.com/hc/en-us/articles/201577976-How-can-I-get-access-to-your-card-pricing-data-

Interested in adding this valuable resource to your site/app along with an additional revenue stream? Then [Send us an Email] for more information. Please be sure to tell us about the design plans for your app/site or a link to the live version. Also, let us know how you plan to use our pricing data.

While it may seem silly to even try, the worst they could do is say "No."

[naudor]

I wrote to TCG to get access to their API and they answer me that they need see the project (screenshots and maybe some demo) , this project must be a big project with a lot of traffic. I think they only give access if they think that these project will become a big deal for them.

[starryknight64]

You could use DeckBrew API to get access to the TCG Player pricing: http://deckbrew.com/api/

[naudor]

DeckBrew work's fine, but you can't get the price of the foil versions of the card El 11/11/2014 17:50, "Phillip" notifications@github.com escribió:

You could use DeckBrew API to get access to the TCG Player pricing: http://deckbrew.com/api/

— Reply to this email directly or view it on GitHub https://github.com/bedoherty/MagicTCGPriceAPI/issues/7#issuecomment-62576920 .

[Garbee]

@naudor afaik TCGPlayer does not give their foil prices out through their API in the first place. So anyone who does get it from them is scraping it directly. The api provided by this repo in the first place didn't provide Foil prices, so I don't see how that is even an issue for the problem at hand.

The bigger thing with it is, slow to get updated. Like right now it still doesn't have Commander 2014 data at all. :frowning: So the API could take a while to get updated for new sets since that is a side-project for the person doing it.

[sonicemerald]

whoops i should have read this before creating a pull request... I have added a way to parse each set at once, but have not figured a way around the single card issue. There might be a way, but it seems to need both the set and cardname... I'll try my best to work around it.

[reguengos]

none is working deckbrew is not returning values as well.

[romelako]

Yup, DeckBrew isn't returning any prices atm. It looks like we're SOL in terms of obtaining MTG prices. Does anyone have a workaround or perhaps another API we haven't heard of yet?

[bedoherty]

Heres the situation with current scraping of TCGPlayer: TCGPlayer as of the opening of this issue has protected their website with Distil's Content Protection Network. Heres what I assume is true of Distil based on what I know, but this is mostly conjecture.

1. Distils network acts as both a traffic monitoring and filtration system.
2. Distil will block your traffic if you attempt to load the page in a manner that doesn't allow Javascript
3. Distil will block your traffic if it appears to be a bot, e.g. too many page loads in a time period

If I had all the time in the world, I could possibly come up with a method for bypassing their protection, although I just find that to be unlikely in my current situation. If anyone wishes to work on or discuss bypassing their protection, feel free to use this issue thread to discuss it and I will chime in when I can.

[bedoherty]

I'll try to get around to it when I can, but if someone else wants to test something along the lines of QtWebKit it might be worth a shot. http://tudorbarbu.ninja/downloading-a-pages-content-with-python-and-webkit/

[Garbee]

The gatherer extractor software does this:

• Pulls card page from magiccards.info and parses data out
• Makes request for the TCGPlayer script on the page and parses that script response to get pricing.

Sucks since you are using more resources to do a site pull to then execute a script on it to then parse, but with caching for X time the downside can be minimized.

[sonicemerald]

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=fate+reforged (this is scraping tcg's database query)

so... what if we just get the set prices in json format and have another method to get the single cards out of the json? or is that too much work?

[reguengos]

It could work in 2 steps... 1st- get card name+ editions 2nd- get set price from which the card belong and find its price from within the suggested set..

-----Original Message----- From: "sonicemerald" notifications@github.com Sent: ‎24/‎01/‎2015 17:56 To: "bedoherty/MagicTCGPriceAPI" MagicTCGPriceAPI@noreply.github.com Cc: "reguengos" joao.pedro.grilo.soares@gmail.com Subject: Re: [MagicTCGPriceAPI] TCG prices (#7)

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=fate+reforged so... what if we just get the set prices in json format and have another method to get the single cards out of the json? or is that too much work? — Reply to this email directly or view it on GitHub.=

[reguengos]

This way it will definitely work. Tomorrow ill give it a try ;)

-----Original Message----- From: "Joao pedro" joao.pedro.grilo.soares@gmail.com Sent: ‎25/‎01/‎2015 02:46 To: "bedoherty/MagicTCGPriceAPI" reply@reply.github.com; "bedoherty/MagicTCGPriceAPI" MagicTCGPriceAPI@noreply.github.com Subject: RE: [MagicTCGPriceAPI] TCG prices (#7)

It could work in 2 steps... 1st- get card name+ editions 2nd- get set price from which the card belong and find its price from within the suggested set..

From: sonicemerald Sent: ‎24/‎01/‎2015 17:56 To: bedoherty/MagicTCGPriceAPI Cc: reguengos Subject: Re: [MagicTCGPriceAPI] TCG prices (#7)

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=fate+reforged so... what if we just get the set prices in json format and have another method to get the single cards out of the json? or is that too much work? — Reply to this email directly or view it on GitHub.=

[sonicemerald]

@reguengos feel free to use my code. I think it's on bedoherty's pull requests / my page.

[laxika]

Who has http://notional-buffer-750.appspot.com/ ? Because I found some bugs there I want to report and also thanks for the service. :)

[sonicemerald]

Its mine, most of the code base is bedoherty's. I just added the setprices. On Feb 7, 2015 12:11 AM, "Gyula Lakatos" notifications@github.com wrote:

Who has http://notional-buffer-750.appspot.com/ ? Because I found some bugs there I want to report and also thanks for the service. :)

— Reply to this email directly or view it on GitHub https://github.com/bedoherty/MagicTCGPriceAPI/issues/7#issuecomment-73355020 .

[laxika]

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Fifth+Edition http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Portal+Three+Kingdoms http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Unhinged http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Fourth+Edition http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Unglued

These sets give backe 500 Internal server error for some reasons.

And also all sets have an empty card on the end of the pricing list for example: http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Scars+of+mirrodin

{"name": "", "low": "", "med": "", "high": ""}

Thanks for your site a lot btw! I'm using it for https://github.com/laxika/Swords-of-Magic . :)

[sonicemerald]

Cool I'll look into these issues, it looks like if I just get rid of the empy set, it should resolve itself. On Feb 8, 2015 1:02 AM, "Gyula Lakatos" notifications@github.com wrote:

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Fifth+Edition

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Portal+Three+Kingdoms

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Unhinged

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Fourth+Edition

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Unglued

These sets give backe 500 Internal server error for some reasons.

And also all sets have an empty card on the end of the pricing list for example:

http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Scars+of+mirrodin

{"name": "", "low": "", "med": "", "high": ""}

Thanks for your site a lot btw! I'm using it for https://github.com/laxika/Swords-of-Magic . :)

— Reply to this email directly or view it on GitHub https://github.com/bedoherty/MagicTCGPriceAPI/issues/7#issuecomment-73402797 .

[sonicemerald]

ok, fixed the empty set, and found out that those errors are caused by cards such as Dandân, which python isn't encoding correctly.

[laxika]

Wow, nice! Thanks a lot!! I'll report more bugs if I find any...

[romelako]

Hi there. Just some bug reports. :)

The set Innistrad (http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Innistrad) is missing the following cards from the set prices: Snapcaster Mage, LIliana of the Veil. There could be more but those are the cards I cannot find prices for.

Also, the set "M15" doesn't show up on the API. I'm unsure what to call it. I've tried Magic 2015, Magic 2015 Core Set, etc. If you would please provide the proper set name for this particular set, that would be awesome.

Thanks again for all your work, bedoherty & sonicemerald!

[sonicemerald]

It's going off TCGplayers names, so Magic 2015 (M15) I believe is what you want. No clue about the innistrad stuff, I'll look into that. On Feb 15, 2015 2:38 PM, "Robert Calove" notifications@github.com wrote:

Hi there. Just some bug reports. :)

The set Innistrad ( http://notional-buffer-750.appspot.com/api/tcgplayer/setPrices.json?cardset=Innistrad) is missing the following cards from the set prices: Snapcaster Mage, LIliana of the Veil. There could be more but those are the cards I cannot find prices for.

Also, the set "M15" doesn't show up on the API. I'm unsure what to call it. I've tried Magic 2015, Magic 2015 Core Set, etc. If you would please provide the proper set name for this particular set, that would be awesome.

Thanks again for all your work, bedoherty & sonicemerald!

— Reply to this email directly or view it on GitHub https://github.com/bedoherty/MagicTCGPriceAPI/issues/7#issuecomment-74441360 .

[sonicemerald]

got Innistrad back up, i had an error in my logic, and didn't notice it. thanks for letting me know.

[mezmor]

Trivial to do with selenium:

BASE_URL = "http://shop.tcgplayer.com/magic/{set_name}/{card_name}"
tcg_url = BASE_URL.format(set_name=set_url, card_name=card_url)

driver = webdriver.Chrome('selenium/chromedriver')
driver.get(tcg_url)

price_ranges = driver.find_elements_by_class_name('priceRange')

price_result = {'normal': {}, 'foil': {}}

for index, prices in enumerate(price_ranges):
    # Normal prices at index 0
    # Foil prices at index 1
    for price_type in ['low', 'median', 'high']:
        price = prices.find_element_by_class_name(price_type).text
        price = price.split('$')[-1]  # Initial format is 'High: $50.00', here price is set to 50.00
        if index == 0:
            price_result['normal'].update({price_type: price})
        else:
            price_result['foil'].update({price_type: price})

[Garbee]

This API is built to run in Google App Engine. Last I checked is a restricted environment and you are unable to install custom software, such as Chrome, the Chrome Web Driver, and Selenium.

[mezmor]

headless ff?