MtG Cards from the recognizer

[paul-lrr]

As you may know, I have been working on a system for automatically recognizing MtG cards during AFK streams and showing them on the overlay. I'm still evaluating the best option for the actual recognition tech, but the process for transmitting that information to the overlay is pretty much figured out. XSplit is a little tricky to get information into, so I am using a websocket system with the publisher being a php script on the local computer and a javascript subscriber as a plugin in xsplit (much in the way we are doing the twitch sub notifier). For the sake of simplicity, I'm using PubNub to provide the websocket server as they have a simple API and libraries for pretty much any language.

During the MTG AFK streams, I often see people in the chat doing the !card command for the card on the overlay. I was thinking that, if lrrbot had the PubNub subscribe key and channel name, it could just automatically report the current card in the chat (or on a separate page). Do you think that is something people would find useful?

[mrphlip]

I think that sounds pretty useful. Personally I can usually only watch the stream live on lower quality settings, and while the card viewer helps, it's still kinda hard to read sometimes, so having the text also appear in chat is worthwhile I think.

My main concern is the stream delay, but luckily we're now working in the direction where time travel is possible... just get the message from pubnub, wait 15 seconds, then show the message in chat. Or maybe a bit less, to compensate for the time it takes for the scanner to recognise the card?

MTGJSON, which is where the bot is getting all the oracle text from, includes multiverseid, so that should be doable.

Looking at the official pubnub Python api, and it uses threading, so have to be careful with that and make sure it doesn't interact directly with asyncio... But that can be dealt with.

[AdmiralMemo]

I'm taking stats (from different streams to get a wider base to go by) on how long to delay the message:

http://strawpoll.me/6783956/r - GPLP 2/10/2016

Currently a 10 to 12 second delay seems ideal for most of Chat without being way too early or way too late, but we'll see.

[d3fr0st]

What about making a page for it on the website similar to subscribers notifier. That way we'd keep an history for anyone on a longer than expected.

[mrphlip]

A webpage is also a good option... could even show the card art for the latest card there, like what's on the stream, as well as the history of recently-scanned cards. We'd have to keep linking it in the chat (probably have a text command for the link), but that'd probably still be less spammy than people using !card all the time.

I'd also be open to the cards appearing both in chat and on a webpage for history, but maybe that's overkill?

On another aside: today I caught up to the TTC where the card recogniser is first used (I'm behind on my podcasts, OK? I'm working on it), and I had the thought you'd probably want to make sure that's a separate pubnub channel if that's using the same tech (iirc you use xsplit for recording the vidcasts?) so that every time you do a crack-a-pack, the contents don't get automatically spilled into the chat... (The TTC episode is about the OGW leak. How ironic.)

[d3fr0st]

We could do both without it being overkill... Make a simplified chat message like card name, set, card type with a link to a webpage that has a full card description. Plus that way the bot doesn't spam the chat when cards have a ton of words on them.

Also yeah TTC would need to not be spoiled

[paul-lrr]

The TTC point is a good one. I also usually do pretty extensive testing before a stream to make sure the camera is lined up and so forth, which we wouldn't want to send to LRRbot. A single "app" in PubNub can have any number of channels, so making a separate TTC and/or testing channel wouldn't be a problem. For safety (and because it seems likely I could forget to change the channel), it might actually be better to keep the function off most of the time and only subscribe to the channel when a MtG AFK stream starts and a mod gives a command (eg. !cardscan on).

[AdmiralMemo]

http://strawpoll.me/6783957/r - LRRMtG 2/11/2016

Responses a little more varied this time, and skewing a little towards more delay. Still, 12 to 15 seconds seems pretty good.

[AdmiralMemo]

For the final one...

http://strawpoll.me/6783958/r - W&P 2/12/2016

So, depending on how Twitch is feeling, judging by these 3 polls, 10 to 15 seconds is ideal, and I'd personally go for 12 to 13.

[mrphlip]

OK, created a first draft of the system. The pubnub code is based off what was shown in that tech demo stream from a couple weeks back... all it's missing is the subscribe key. Though I understand the system's gone through some revisions since then (including replacing the OCR tool), so I wouldn't be surprised if there's some changes needed there.

Basically, it subscribes to xsplit_image on pubnub and waits for anything that looks like a Gatherer card image URL, pulls out the multiverseid and gets the card details from there. The whole thing can be turned on and off by a mod command in the chat (it defaults to off, someone'd need to turn it on at the start of an AFK stream and then turn it back off again after).

Right now it's just outputting the card details in the chat, but a webpage could be added at some point that works like /notifications. I'm just too lazy to build it myself right now.

From my testing, it looks like the pubnub system has a second or two delay, so I've set the delay to 10 seconds as a first approximation. We can always tweak it later if necessary.

@andreasots: Can we get a pip3 install pubnub on the server?

[mrphlip]

(Incidentally, the pubnub API docs are... lackluster. They look pretty slick, seem to have a full API reference, but then it's got a bunch of typos in things like parameter names, and then the docs don't actually contain the word "thread" anywhere, which seems like some important details to leave out...)

[paul-lrr]

That's a shame about the API docs. The PHP and JS ones seemed to work for me, but I didn't delve too deeply. What do you folks think about the idea of the function being off most of the time and LRRBot only actually subbing when there is a live stream and a mod tells it to? Also, while not super confidential since it doesn't allow publishing, I would rather the sub key was not publicly available. Is there a good way to send the key such that it won't be publicly viewable in this github project?

[d3fr0st]

Turning the feature on by mods seems like a good way to approach this.

We do have a couple of excluded files which only mrphlip, qrpth, and I have access to, for auth keys and such.

[andreasots]

http://strawpoll.me/6783956/r - GPLP 2/10/2016

Expected value: 13.264705882352938 Standard deviation: 4.685661211816985

http://strawpoll.me/6783957/r - LRRMtG 2/11/2016

Expected value: 14.190972222222218 Standard deviation: 6.149138902315447

http://strawpoll.me/6783958/r - W&P 2/12/2016

Expected value: 13.323943661971834 Standard deviation: 5.194637848955638

!cardscan on, subscribing on demand

If XSplit widgets are basically HTML documents, couldn't it send an HTTP request on page load to LRRbot to subscribe to Pubnub and on beforeunload or unload to unsubscribe? !cardscan should be in addition to this.

Can we get a pip3 install pubnub on the server?

Done.

Is there a good way to send the key such that it won't be publicly viewable in this github project?

E-mail or LRR forum private message.

[mrphlip]

...couldn't it send an HTTP request on page load to LRRbot...

They're using the same widget for their dry-run tests, though, and probably also using the same widget for TTC, so this wouldn't really solve anything... since the point is to make it so that only cards scanned for real live streams are shown in the chat...

[paul-lrr]

something that I hadn't really thought about is that the system is currently designed to work with cards from a variety of sources, not just the recognizer. When the recognizer finds a card, what is actually being sent through pubnub to the overlay is the URL of the card image (from gatherer). This allows me manually override the image if the card recognizer is having problems and also allows us to show spoilers, tokens, or other cards that are not in the gatherer database. It is easy to extract the multiverse ID from the gatherer image URL, but would it be better if I just sent lrrbot the multiverse ID by itself?

[andreasots]

It is easy to extract the multiverse ID from the gatherer image URL, but would it be better if I just sent lrrbot the multiverse ID by itself?

I don't have strong feelings either way. I guess it would be nice.

cards not on Gatherer

I remember seeing a full-art card during the Highlander tournament and I think those aren't on Gatherer and thus have no multiverse ID. LRRbot could look those up by name if the recogniser even detects those.

[paul-lrr]

Sorry for the delayed response, I've been working on other aspects of the overlay. As you may have heard, we are doing an AFK draft on Friday, so it would be cool if this feature could be running by then. Would there be a good time today or tomorrow to do a test of the system?

[andreasots]

Before Heather's stream today or before James tomorrow would work for me.

[paul-lrr]

let's do it around 11am PST (an hour before Heather's stream)

[andreasots]

Tech test complete. Seems to be working.