ValueError: Reference build not found for set: master. Has reference been setup? If not, run build.py.

[JBrame]

Hey, just cloned this to give it a go and hit this error out the gate. Any ideas what I need to get it working?

  File "C:\{PATH}\tmp\test.py", line 3, in <module>
    recognizer = CardRecognizer(mode=Mode.SINGLE_IMAGE)
  File "C:\{PATH}\venv\lib\site-packages\card_recognizer\api\card_recognizer.py", line 41, in __init__
    ref_pkl_path = ReferenceBuild.get_set_pkl_path(set_name=set_name)
  File "C:\{PATH}\venv\lib\site-packages\card_recognizer\reference\core\build.py", line 73, in get_set_pkl_path
    raise ValueError(
ValueError: Reference build not found for set: master. Has reference been setup? If not, run build.py.

[prateekt]

Hi JBrame, I will add documentation in the next version.

Please see card_recognizer/card_recognizer/reference/core/build.py

Run python build.py [PTCGSDK_API_KEY] using your API key for Pokemon TCG SDK.

You will need to get an API key from here: https://dev.pokemontcg.io/

This will build your reference locally and allow you to use the pokemon-card-recognizer. This may take ~1/2 hour. I may choose to bundle the reference in the next release so you don't have to rebuild it.

[JBrame]

Thanks for the quick reply! I've kicked that off, will let you know if I have any other issues :)

[prateekt]

About to update the documentation and add the reference. Thanks for helping test this process!

[prateekt]

I just bundled the reference with the build on git lfs. You can try git clone again and see if the problem still persists.

[prateekt]

Also, if you do

pip install pokemon-card-recognizer

your installation should already come with the reference.

[JBrame]

Hey, some more info. Installing on a clean venv using pip. Pip is doing a lot of backtracking for dependencies which is taking a long long time:

  Using cached ocr_ops-0.0.0.1-py3-none-any.whl (13 kB)
INFO: pip is looking at multiple versions of jupyter-server-terminals to determine which version is compatible with other requirements. This could take a while.
Collecting jupyter-server-terminals
  Using cached jupyter_server_terminals-0.4.3-py3-none-any.whl (13 kB)
  Using cached jupyter_server_terminals-0.4.2-py3-none-any.whl (13 kB)
  Using cached jupyter_server_terminals-0.4.1-py3-none-any.whl (13 kB)
  Using cached jupyter_server_terminals-0.4.0-py3-none-any.whl (13 kB)
  Using cached jupyter_server_terminals-0.3.2-py3-none-any.whl (13 kB)
  Using cached jupyter_server_terminals-0.3.1-py3-none-any.whl (13 kB)
  Using cached jupyter_server_terminals-0.3.0-py3-none-any.whl (11 kB)
INFO: pip is looking at multiple versions of jupyter-server-terminals to determine which version is compatible with other requirements. This could take a while.
  Using cached jupyter_server_terminals-0.2.1.tar.gz (26 kB)
  Installing build dependencies ... canceled

I used the following option for now to try and speed things up: pip --use-deprecated=legacy-resolver install pokemon-card-recognize

It hits this error:

ERROR: No matching distribution found for torch==1.11.0 (from ocr-ops->pokemon-card-recognizer)

I installed the requirements from requirements.txt and then ran: --no-dependencies

All the dependencies except numpy installed. Turns out numpy doen't support > Python 3.9 and I was on 3.11 so downgraded and tried again. Everything install using pip install pokemon-card-recognizer and without the backtracking issues above.

Running the original script still throws the same error.

Running python build.py API_KEY now, will let you know how I get on once this finishes

[JBrame]

The build process seems to hang here, it's been like this for an hour or so

Downloading 154_hires.png
Downloading 160_hires.png
Downloading 158_hires.png
Downloading 150_hires.png
Downloading 156_hires.png
Crown Zenith: building reference...
Building master reference...
Making Eval plots...

Attempting to run the script then downloads the models, but I hit a permission error. I haven't tried running it as sudo, but why is it writing outside of the venv or current working directory?

PermissionError: [Errno 13] Permission denied: 'C:\\Users\\user\\AppData\\Local\\Temp\\tmpcdii2m0r.png'

[prateekt]

The easiest path should be:

make conda_dev

to build the conda environment that should pull the appropriate dependencies in a clean conda environment card_rec_env. The conda environment explicitly installs python=3.9.16 which is the recommended version.

conda activate card_rec_env

Then

pip install pokemon-card-recognizer

in that conda environment. Then

from card_recognizer.api.card_recognizer import CardRecognizer, Mode 
recognizer = CardRecognizer(
    mode=Mode.PULLS_VIDEO
)
pulls = recognizer.exec("/path/to/video")

Sorry, I realized I usually make the eval plots which takes much longer than it used to with more sets, but should still work. Maybe I'll take that out for users. The reference is actually technically built if you ended up at that step (it surpassed "Building master reference..."), so you should be able to use the card recognizer in your Python environment. Test again in a Python shell. There are sample images and videos in card_recognizer/api/test. You should also be able to run nosetests in the root directory cloned from github.

Thanks for so much for taking the trouble to try to install this and build the reference from scratch (which is not the intended workflow, sorry). Still, I think you actually succeeded for the most part which is great news!

What script did you hit the permission error with? It seems build.py also downloaded the data and built the models. At the step you're at, it's just building eval figs that should be stored in a created "data" directory that it creates within the confines of the root directory. It should NOT be writing data elsewhere.

Thank you so much for trying this Windows. I have only tried this on an Ubuntu linux box and M1 Mac. If this works, you'll be the first Windows user! It remains to be seen if this works well on your system GPU. The reason pytorch is such an old version is because the OCR library I ended up using (easy_ocr) seems not to like much else, but it has been the most accurate and efficient backend on NVIDIA cards. I haven't yet gotten it to work on M1. I compared multiple such libs which are wrapped in ocr_ops (https://pypi.org/project/ocr-ops/). Let me know how things go!

[JBrame]

Sorry I'm travelling at the moment so can't reply to all your comments, but thought I'd mention it's a version of the script in the readme that is throwing the permission error

from card_recognizer.api.card_recognizer import CardRecognizer, Mode 
recognizer = CardRecognizer(
    mode=Mode.SINGLE_IMAGE
)
pulls = recognizer.exec("test.jpg")

I'll be trying this on Ubuntu when I get back tomorrow because I haven't had any luck with CUDA on my machine with windows so I'll see how that goes. More than happy to keep testing this on windows for you though.

[prateekt]

I am able to reproduce your error with pip install unfortunately. The ball is in my court. I'm going to look into this, push a fix, and get back to you. Stay tuned.

[prateekt]

Repeat the suggested process for making the conda and pip install. Just tested it on both computers.

Here is the new code:

from card_recognizer.api.card_recognizer import CardRecognizer, OperatingMode 
recognizer = CardRecognizer(
    mode=OperatingMode.SINGLE_IMAGE
)
pulls = recognizer.exec("test.jpg")
print(pulls)

[JBrame]

So, some good news and some bad news...

I have everything installed and working without throwing an error! The bad news is that it incorrectly identifies a card I tested because the set isn't in the model "Champion's Path".

Is it as simple as adding the sets I'm interested to the array in build.py and re-running build.py?

[prateekt]

Yep I can rebuild the reference with the sets you want. Let me know.

[prateekt]

But yes in principle it should be that easy

[JBrame]

Well ultimately I'd like all the sets, but for now it's all the ones you have plus the few missing from SWSH.

That said, should the script not programmatically get all available sets and process them all or does that just take too long?

[prateekt]

Haven't characterized yet the full master model, but would be worth trying.

[JBrame]

I'll knock something together within build.py to grab all sets 👍

[prateekt]

I'll try to run some experiments tonight and get back to you.

[prateekt]

@JBrame The benchmarking went fairly well last night with all 150 sets currently available on Pokemon TCG SDK. A couple of sets with some issues. Going to fix that up tonight and hopefully release a version by this weekend.

Here are all the sets soon to be supported!

from pokemontcgsdk import Card, RestClient, Set sets = Set.all() [set.name for set in sets]

['Base', 'Jungle', 'Wizards Black Star Promos', 'Fossil', 'Base Set 2', 'Team Rocket', 'Gym Heroes', 'Gym Challenge', 'Neo Genesis', 'Neo Discovery', 'Southern Islands', 'Neo Revelation', 'Neo Destiny', 'Legendary Collection', 'Expedition Base Set', 'Aquapolis', 'Skyridge', 'Ruby & Sapphire', 'Sandstorm', 'Dragon', 'Nintendo Black Star Promos', 'Team Magma vs Team Aqua', 'Hidden Legends', 'FireRed & LeafGreen', 'POP Series 1', 'Team Rocket Returns', 'Deoxys', 'Emerald', 'Unseen Forces', 'POP Series 2', 'Delta Species', 'Legend Maker', 'POP Series 3', 'Holon Phantoms', 'Crystal Guardians', 'POP Series 4', 'Dragon Frontiers', 'POP Series 5', 'Power Keepers', 'Diamond & Pearl', 'DP Black Star Promos', 'Mysterious Treasures', 'POP Series 6', 'Secret Wonders', 'Great Encounters', 'POP Series 7', 'Majestic Dawn', 'Legends Awakened', 'POP Series 8', 'Stormfront', 'Platinum', 'POP Series 9', 'Rising Rivals', 'Supreme Victors', 'Arceus', 'Pokémon Rumble', 'HeartGold & SoulSilver', 'HGSS Black Star Promos', 'HS—Unleashed', 'HS—Undaunted', 'HS—Triumphant', 'Call of Legends', 'BW Black Star Promos', 'Black & White', "McDonald's Collection 2011", 'Emerging Powers', 'Noble Victories', 'Next Destinies', 'Dark Explorers', "McDonald's Collection 2012", 'Dragons Exalted', 'Dragon Vault', 'Boundaries Crossed', 'Plasma Storm', 'Plasma Freeze', 'Plasma Blast', 'XY Black Star Promos', 'Legendary Treasures', 'Kalos Starter Set', 'XY', 'Flashfire', 'Furious Fists', 'Phantom Forces', 'Primal Clash', 'Double Crisis', 'Roaring Skies', 'Ancient Origins', 'BREAKthrough', 'BREAKpoint', 'Generations', 'Fates Collide', 'Steam Siege', "McDonald's Collection 2016", 'Evolutions', 'Sun & Moon', 'SM Black Star Promos', 'Guardians Rising', 'Burning Shadows', 'Shining Legends', 'Crimson Invasion', 'Ultra Prism', 'Forbidden Light', 'Celestial Storm', 'Dragon Majesty', 'Lost Thunder', 'Team Up', 'Detective Pikachu', 'Unbroken Bonds', 'Unified Minds', 'Hidden Fates', 'Shiny Vault', "McDonald's Collection 2019", 'Cosmic Eclipse', 'SWSH Black Star Promos', 'Sword & Shield', 'Rebel Clash', 'Darkness Ablaze', "Champion's Path", 'Vivid Voltage', 'Shining Fates', 'Shiny Vault', 'Battle Styles', 'Chilling Reign', 'Evolving Skies', 'Celebrations', 'Celebrations: Classic Collection', "McDonald's Collection 2014", "McDonald's Collection 2015", "McDonald's Collection 2017", "McDonald's Collection 2018", "McDonald's Collection 2021", 'Best of Game', 'Fusion Strike', 'Pokémon Futsal Collection', 'EX Trainer Kit Latias', 'EX Trainer Kit Latios', 'EX Trainer Kit 2 Plusle', 'EX Trainer Kit 2 Minun', 'Brilliant Stars', 'Brilliant Stars Trainer Gallery', 'Astral Radiance', 'Astral Radiance Trainer Gallery', 'Pokémon GO', 'Lost Origin', 'Lost Origin Trainer Gallery', 'Silver Tempest', 'Silver Tempest Trainer Gallery', "McDonald's Collection 2022", 'Crown Zenith', 'Crown Zenith Galarian Gallery']

[JBrame]

Oh awesome! Thanks for looking into it! I'll be playing around with it again on Monday so if you have a new build by then let me know and I'll test that.

FYI, I've had a difficult time getting CUDA up and running on my PC. It's not something that I'm familiar with, so once I get it working (which I think I'm pretty much there now) I'll write an idiots guide which might be helpful for your read me 👍

[prateekt]

CUDA is optional, though highly recommended for using the pipeline. Totally understand, never had an easy time setting up CUDA. What GPU are you using on your PC?

[JBrame]

Yeah to be fair it was pretty quick on my CPU anyway but I just thought I'd try it on the GPU.

I have an RTX3060. The main issue I had was that I installed CUDA just using apt and on a fresh Ubuntu 22 it installed and cached a version higher than torch has a compiled version for. Took me a while to figure that out and how to completely remove it before figuring out how to get the right version that torch supported...

[JBrame]

FYI just created a new venv on windows, installed the latest with pip and hit the permission error again.

PermissionError: [Errno 13] Permission denied: 'C:\\Users\\admin\\AppData\\Local\\Temp\\tmpwblls7hm.png'

It looks like it's doing it before even downloading the training model.

No such issues on a clean Ubuntu venv.

[prateekt]

What step do you hit this on?

[prateekt]

@JBrame I updated the API and improved the documentation to make this easier to use. Let me know if you're still running into issues.

[JBrame]

Hey sorry, had a few hectic days so not had chance to try the updates yet. Will be getting back on this on Monday so will let you know!

[prateekt]

@JBrame Let me know if you need any help. Hopefully the API should be fairly clean now.

[prateekt]

I'm closing this issue as I believe it is resolved. Reopen if you feel otherwise or open a new ticket with latest issues.